I'm aware a lot of the comments discuss this already, but this really does appear to be a case of bad program design. Pick any movie discussing website online and you can see people echoing your views:
IMDB:
...if any government department in any country ran anything remotely
as incompetently as these guys do in "The Following" then we are
seriously, seriously doomed.
And then they escape because the FBI or the Police were looking the
other way at the critical moment.
...Joe's escape from jail was the final straw. The premise to be moved
to another jail is bizarre and if that's not bad enough then prison
officials, FBI agents and Federal marshals all manage not to be able
to put one prisoner in the back of a transport truck and move him. But
he then gets magically in the trunk of his lawyer's car and escapes.
We all have different tastes in shows but I find it baffling that
anyone could find this show even remotely watchable because it's so
insulting to the viewers.
From Digital Spy, who note the lack of a prison log of any sort:
...There's Ryan's slipper snake ex Molly (Jennifer Ferrin) - who's now
one of Joe's followers. A self-styled 'Angel of Death', Molly has it
in for Hardy and first met Carroll - you guessed it - in prison.
Seriously, Feds, check that visitor log.
And finally, a rather entertaining rant over at Time Entertainment. Some of the choice quotes from this article:
The good guys are apparently idiots.
At first, I was okay with this. I could accept that law-enforcement
might be fallible; otherwise, the show would be an hour of Kevin
Bacon’s former FBI agent Ryan Hardy talking to James Purefoy’s
creepy-yet-charming psychopath Joe Carroll, explaining that there was
no way that an imprisoned serial killer would’ve been able to use the
Internet to set up thousands of websites to gain acolytes without
everyone in the prison knowing about it (and logging the IP addresses
of those visiting the sites).
...The problem with The Following is that these moments of idiocy are not
few and far between. In fact, as Vulture pointed out earlier this
week, every single episode of the show has asked us to believe that
the FBI agents hunting the serial killing cult have made at least one
fatal, and often ridiculous, error in judgment or procedure that
prevents them from catching the bad guys...
So effectively, the answer to your question appears to be because the FBI, by the decision of the show's creators, are complete idiots - something many people online have noticed and been irritated by.