This is covered in the episode, Holmes mentions it to the cabbie and the cabbie replies something like:
Four games and four wins, that's not random
If you want an excellent, explained example of this in action - watch The Princess Bride:
The man in black's intelligence test. He provides two vials, one of which is poisoned.
Vizzini explains his chain of thought thus expressing how he feels that the placement of the poisons is not random but is a psychological trick. In the same way the taxi driver feels that there is psychology in the way he plays his 'game' and that it is not random - whether either if them are right or wrong is not the point, the point is that they feel there is more to it than random chance
Note: the twist in the Princess Bride's version about immunities is not relevant to the Holmes's version. I only included this as it explained the chain of thought as to which glass should be drunk from.