My understanding is that by episode S03E02 "Sick" of The Walking Dead, Rick and his team had cleaned out the whole prison and hardly one walker was alive in the vicinity. The prison was locked on the outside too. How did the walkers then enter the prison on hearing the alarm bell in S03E04 "Killer Within"? Was some part of the prison still infested with walkers?
Rick's group only clear a single cell block, the one they lived in, and the outside area where they keep their vehicles and later grow crops.
They specifically did not clear the rest of the prison: at one point they discuss it, and decide not to for the following reasons:
It is risky. There are lots of narrow, dimly-lit corridors. They would have to move slowly, and risk being cut off from escape and overwhelmed.
There is not much need to. The doors are secure, being a prison: it would take a live person with specialized tools to break through a door. Walkers cannot use tools or do much of anything other than make noise, bite, and scratch. So "doing nothing" is still an effective defense against walkers.
Finally, the biggest reasons:
The walkers in other cell blocks are a defensive mechanism. If another group of living humans breaks into the prison, they may be in another cell block that is not cleared. That puts them at risk of dying due to the reasons in point 1 above, before they can reach Rick's group.
Rick's group only needs active defense for a much narrower portion of the prison than "all of it" which makes their job easier. Fewer guards patrolling at any given time means more opportunities to stay well-rested, grow food, and perform any other tasks required to survive.
Now, as to what exactly happened in S03E04: the group is split, with most everyone in the courtyard. Walkers appear behind them. Carl turns around and sees them, then they all start fighting. T-Dog makes a comment that "the gate is open" implying that one of the gates to the uncleared area was left open, allowing walkers into C block. The group splits up in the chaos and appear to enter another cell block that is not cleared. When the alarm goes off, this would "wake up" any walkers that were not already actively chasing the protagonists, and they would then respond to other noises and seek out the living.
They only cleared one block. They were helping the prisoners to clear another one but the deal felt apart and the rest of the prison was left infested.
You can clearly see there were more zombies in the prison towards the end of the episode when Rick chased one of the prisoners to a exercise yard full with zombies and that was just a small part of prison.