It's the people, not the role.
Jim Caviezel only plays one type of guy: dour, serious. Look at his performances in Count of Monte Cristo or Person of Interest. In those works, as well as in The Passion of the Christ, he plays the roles as a very fatalistic, driven, and detached person. Even on talk shows he is the same kind of person.
He's also extremely open and passionate about his personal religious beliefs (as a devout Christian), which would probably make him uncomfortable to have around in a lot of Hollywood scenes... "oh here comes Jim, he's going to preach to us. He doesn't drink. He's a goody two shoes", etc... That, and Jesus movies aren't particularly good sellers in America. They tend to make people feel bad about their own shortcomings, and reminds them of the 1 to 2 hour religious sessions many of them already experience every single week at church. That's the worry that Gibson was getting at, I think.
Take Jeffrey Hunter in King of Kings for another example, for some reason he declined the role of Captain Kirk on Star Trek, despite acting as Kirk in the pilot. We may have never known William Shatner as anything other than a bit part actor if Jeffrey Hunter had stayed on as Captain Kirk. After that Hunter kind of faded away.
Others, however, who portrayed Jesus went on to have great careers... people like Max von Sydow or Christian Bale have quite the range and huge amounts of charisma. I mean, Christian Bale is both Jesus Christ and Batman.
It's also not unique to Jesus flicks. As others have already indicated, folks like Leonard Nimoy as Spock or Mark Hamill as Luke Skywalker were instantly and forever typecast/shoehorned into that role... they're so unique and memorable that real people literally can't imagine such actors as their real selves, only as the characters they portrayed.
Many heavily typecast actors don't appear to be suffering from the same fate, because they were lucky to be typecast as "action" actors... think Liam Neeson, Bruce Willis, Jason Statham, Michelle Rodriguez, etc. and to a lesser extent people like Dwayne Johnson. You just can't picture them in a role where they aren't an action hero (even Qui-Gon or Action movies are very enjoyable experiences to repeat for audiences, because the contents of the film are novel and outlandish... so they make big bucks. Likewise, these typecast actors never seem to be out of a role.