It's primarily a mix of English and Mandarin Chinese with allowances for language drift and substitutions for TV purposes.
From this answer on Science Fiction & Fantasy
The United States and China, the two great superpowers of the Earth, gradually grow together and form the Anglo-Sino Alliance (though their empires remain separate), rather than killing each other as originally predicted. In a somewhat ironic reversal, the American Empire annexes England.
[Firefly website and "A Brief History of the Universe, circa 2507 A.D." In an interview on the site, series creator Joss Whedon first described some of this background to account for the mix of cultures in the Alliance, and the pre-production memo gives further details. This history is implicit in the design of the Alliance flags seen in "The Train Job" and "Bushwhacked," a combination of the present-day flags of the two countries in question.]
You can read more on languanges used at the Firefly Wikia
For instance:
Human beings have happily fouled the gift of language with whatever inventive, vindictive, and insulting expressions they can imagine. While the traditional English swear words have survived intact, a few additional crude cuss words have been added to the common man's vocabulary.
The basics include Gorram ("Run! It's the gorram law!"), Ruttin' ("It's gettin' too ruttin' hot in here."), and Humped ("He's got a gun on us. We're humped!") Cursing in Chinese is considered more imaginative and expressive, and most everyone does it - at least when his mother has left the room.
