(For reference, here is the scene)
There might not be a bug
Screenwriter and director Francis Ford Coppola candidly admits that he never decided whether there was a bug to begin with.
In his screenplay (available for download; pdf) (starting on page 155), Harry's search is described in detail, but makes no hint as to where the bug might actually be. It merely reads:
He has not found the tap, if one exists.
In the DVD commentary (available for streaming or download; mp3) (starting at 1:49:06), Coppola says:
"The other most asked question to me ... is where was the bug that bugged Harry at the end? Of course many people have their theories. One notion was that it was in the plastic Madonna that he had in his apartment but he thinks of that himself ... indeed it does not turn out to be there.
"... The other possibility ... and the theory that I always imagined might be the case, was that it was in the little saxophone strap, that little clasp in the saxophone that was hanging around his neck during all the time that he was there ... but that was never confirmed or disproved.
"... And then again of course there is always the possibility that there was no microphone, there was no bug, that the microphone really was more in Harry's degenerated state, in his personal madness brought on by this story and what had happened.
"I know it's very difficult for you to accept from me, but the truth of the matter is I don't know where the microphone bug is."
Difficult to accept indeed!
How finding a bug would change the scene
40 years later The Conversation remains a spectacularly relevant character study and is prescient in its depiction of the slow burning psychological strain of life under surveillance. Its final scene is iconic, and its strength is its ambiguity.
To me, the point isn't whether Harry finds what he thinks he wants to find, but his psychological desperation in searching. That there might be a bug is the straw that breaks the camel's back. We witness a composed man finally give in to deep rooted frustration and paranoia. By the time he resorts to tearing up the floors it's a moot point whether the bug is still in the apartment. Finding it will not assuage his guilt or lessen his complicity in the acts that led up to this; his violated sanctuary is no less violated; discovery is not resolution, only confirmation, the redundant inconsequential kind obsessively pursued by the powerless. Harry is profoundly psychologically burdened, and it culminates in him destroying his home.
In this sense, I think not finding the bug is far from a plot inconsistency. Finding it would weaken the scene. Harry literally and symbolically "comes up empty" which amplifies everything else and ends on a poignant character note. If the bug exists, so be it, but if he finds it, the ambiguity is removed, Harry is vindicated at least in that moment, and the significance of his desperation is diminished.