Malcolm is called Sir by multiple people through out the series, including Zoe and Kaylee and even Rayne (sarcasticly). He's also called Sarge, Sergeant, Cap and Captain. All titles of authority, both earned and given.
As you mentioned, this is more of a TV thing than real life. In real life, the don't call me Sir thing varies by person, and situation. Drill Sergeants on a peace time base require the cadets to call then Sir, (yes sir).
Firefly is also set a few hundred years into the future, in a neo- wild west. The terms and slang and speech has clearly changed, as noted by the inclusion of Mandarin as a lingua franca.
During the battle, the whole rebellion really, Zoe's use of formalities (Sir) is also done to keep the lower ranks inspired, reinforcing the chain of command, and the need for strict adhesion to the rules. A loose troop falls apart easier than a strict one, if current military doctrine is accurate.
Oh. And if Wash is to be believed:
Don't forget to call him "sir". He likes that.
Indicating he genuinely likes the title, or Zoe likes to screw with him cause he doesn't.