When Roddenberry created the original series, he roughly based the uniform colors on 19th Century naval uniforms, which used various combinations of red, gold, and blue. In the original pilot, Captain Christopher Pike, along with the rest of the bridge crew, was given a gold shirt. Spock, however, was given a blue shirt to emphasize his role as Science Officer. Regular background crew were given the red uniforms. Over time, this developed further and the whole of StarFleet was officially categorized by three "departments":
- Command Division (gold) - command/bridge crew, anyone on the command track, administrators, VIP (honorary ranks), etc.
- Science Division (blue) - scientists, computer specialists, research staff, medical staff
- Operations Division (red) - engineers, maintenance crew, security, anyone involved with the day-to-day running of the ship or station
This division/color match-up took a while to cement itself. For instance, Kirk can initially be seen in an avocado-green tunic, although it often appears to be a gold-like color on screen. This was originally to make him stand out against other gold-wearing crew. Due to the cameras & film of the time, the colors appeared differently on-screen than they did in person. Below, Kirk (in the green) appears gold, whereas the gold uniforms appear to be a beige color:
You'll also note that here - in the first aired episode - Scotty is wearing gold instead of his traditional red, and Sulu is wearing blue instead of his normal gold. Obviously, at this point, they were still working out who wore what color.
When making the transition to film, the division colors were dropped altogether due to the feeling that the bright colors wouldn't work well on the large screen. Instead, the first movie introduced white & gray duty uniforms. This decision proved unpopular with fans, however. With the second film, officers began wearing maroon jackets instead.
Years later when starting production on "Star Trek: The Next Generation", the division colors were back, but the red and gold were swapped as you noted. This occurred for two reasons:
- Patrick Stewart & Jonathan Frakes, the highest-ranking actors, both looked better in red than they did gold.
- Brent Spiner, on the other hand, who played the android Data, looked horrible in red since his gold skin-paint clashed with the bright red color.
Both problems were fixed by swapping the colors of the Command and Operations Divisions. And afterwards, in later shows, those are the combinations that stuck.