The exact specifics are never mentioned as far as I'm aware. But based on what we do know, it's not surprising.
First off, Kimber generally deals with wealthy people, who are require less coercion compared to working class people - they tend to only bet money they can stand to lose and will generally shy away from any conflict (or things that damage their reputation). This is different from the Shelby bookie ring, which consist of working class betters who are much more liable to not pay up - thus requiring the Shelbys to have a much greater physical presence to enforce betters pay up. There's also the general tendency for the upper classes to be less rough cut anyway (including the willingness for "casual" physical violence).
Additionally, having rough goons work for you is going to negatively impact the opinion of the wealthy people who you deal with. No upper class gentleman (or woman) is going to want to deal with a rough cut working class individual. So Kimber's hriing policy will have more focus on classy presentation instead of rough muscle. And up until Tommy gets involved, Kimber didn't really seem to need the extra muscle anyway.
Secondly, Kimber is an idiot who is riding the coattails of his name. He may be the face of the operation but his accountant is the real brains. The point I'm making is: the Kimber empire is being run by an accountant, not a fighter nor a criminal. In several scenes, Kimber is shown to be hotheaded and fairly easily fooled (the scene where Tommy interrupts Kimber and Grace's interaction just goes to show how simple it is to spin any story to Kimber and have him believe it), and his accountant is diplomatic but mostly a dealmaker, not an enforcer.
All of this leads to the conclusion that Kimber's empire is established but crumbling due to lousy management and simply not having had any conflicts recently.
Thirdly, and most importantly, Tommy is actively undermining Kimber. Up until that point, Kimber had no issues, but Tommy made sure that Kimber would be targeted. It puts Kimber in a spot of needing help, and Tommy just so happens to have exactly what Kimber needs: muscle. This opens the door to Tommy, who uses it and then turns on Kimber to usurp his position.