The actor certainly is, in terms of average male height... though there is a lot of joking about how short actors tend to be.
Alfonso Ribeiro, the actor who plays him is only 5'6" (1.68m). To compare, Will Smith is 6'2" (1.88m) and James Avery is 6'5" (1.96m).
To compare, the average height of males in the US is 5'9.5" (1.77m) (2007-2010).
Other adult male characters' heights:
Now, he is taller than his 5'4" (1.63m) tall little sister, Ashley (Tatyana Ali), but not his "mom", Vivian (Janet Huber), and older sister, Hillary (Karyn Parsons), who are both 5'7" (1.70m). That being said, we generally split the genders when talking about height anyways, which is why I have separated them out.
Now, all of this is really out-of-universe evidence... because an actor's actual height and the height of the character they play aren't always the same... I don't have any evidence that the height jokes were part of the casting process for the show - it's possible that Carlton's height jokes came about because of Alfonso's short stature.