Sadly, there's no explicit explanation in canon.
What Reavers do
The Reavers are shown to cooperate from the first time they are observed in-universe:
0.1% of the population of Miranda [...] acted conversely to the Pax gas agent, causing them to go insane.
The other 99.9% die by complete inactivity. All non-automated responses from Miranda cease. Alliance scientists come to investigate. The last member of this scientific exploration records the message that Serenity's crew finds on Miranda.
The message ends with a pack of Reavers busting through the door and killing the last scientist. This was a coordinated effort. The Reavers were out for this kill. They knew someone was in there. They expended effort to break in. They risked their lives in a fight.
As for
mental status to be able to pilot starships
The Reavers' scenes closest to a cooperative effort in working on ships are:
- A booby trap on the convoy with the "survivor". This trap was set up off-screen. However this establishes that Reavers don't only use but also manipulate technology.
- When Serenity sneaks to Miranda: Two ships pull a third apart. One could argue that Reavers scavenge ships for parts.
How to become a Reaver
It is also shown that normal people may become Reavers far from Miranda and possibly without Pax: The crew of Serenity loots a powered down convoy and finds a lone survivor who repeatedly stutters
No...
Mercy
Brought up for inspection by an Alliance cruiser, the crew is only allowed to go on, because Mal helps hunting down the "survivor", who turned into a Reaver.
Mal has something like this to say about him:
[The Reavers killed everyone onboard and] made him watch. [...] He took the only course left to him: Becoming one of them.
If Mal is right, being a Reaver is a mental state of aggression with no other option than following the group pressure of violence against weak and fleeing targets.
What Reavers don't do
Mal seems to understand the Reavers, as he repeatedly makes correct predictions about their behavior, e.g. Mal's call not to try and run when they crossed path with a slowly floating Reaver ship as to not provoke them to a hunt.
Reaver cooperation seems limited. Their ships are deranged and regularly with defunct core containment. They don't exactly care for each other or their own well-being.
Reavers don't stop. They take no cover and they don't hesitate. You can see this from the bank heist in the movie. At its end River remarks
He wouldn't stop.
This is kind of ironic, because in the end they find out that Reavers are the inhabitants of Miranda that didn't stop.
My hypothesis
Reavers extend their influence and attack other planets, because they are driven by aggression. However Reavers aren't insane as in "not able to think straight". I'd characterize them as utterly violent psychopaths. They also don't seem to be egoistic but always work towards the common goal of violence.
Maybe they are trying to get back at the Alliance without knowing whom to target specifically. Alternatively they may be trying to turn normal people into Reavers.
Notice River's pleas to the dead people on Miranda to move. Maybe aggression (which is what Pax was supposed to suppress) was the only way to keep people going and they couldn't stop afterwards.
