In one of the first episodes introducing the character of the Governor, we see him and his men murder a team of soldiers heavily armed, to rid them of their weapons.
If he did not had killed them, but had instead incorporated them to its little community, with their equipment, training, and experience as a unified team, the soldiers might have been strong enough to take over the control of the organization of the Governor's community.
This is something that the Governor could not bear : the power he has over the community is all he wants.
That's how I started to think about this character after watching the movie "The Postman" (that I actually liked), and comparing him to the bad guy, "General Bethlehem".
The General shares some aspects with the Governor : he explains he was a kind of loser before the apocalypse, but he has power in the new order of the world and therefore enjoys it a lot.