When Javert arrests the man he thinks is Valjean why doesn't he look for Valjean's tattoo on the prisoner? After all, it is how Valjean showed he was the real man so Javert must have known about it and it is not as if the prisoner is going to refuse to show his chest as he was facing a nasty prison sentence.
I think this is a question that is best answered by the original book. Not having read it, however, I can't imagine that Javert didn't check for the number. I wouldn't be surprised if the faux Valjean had some numbers, maybe close numbers, and maybe a number that was hard to distinguish.
A better explanation might be that there was a scar where the number would have been (or maybe some numbers missing/hard to read), which lead Javert to believe that he tried to remove it. Evidence of this theory would be the inmates Javert called in to testify that the accused was indeed Valjean. Why would he need to prove he was Valjean if the number was easily recognizable? This also explains why Valjean had to stand and testify in court to convince his inmates that he is the real Valjean. Showing his number was the last straw to prove his identity since it wasn't fully working.