The Star Trek Star Fleet is modeled after the modern navy. Seldomly does a person in the military retain the same rank and duty station more than a few years before they are promoted or transferred to a position with higher responsibilities (making them promotable). This also opens up a person’s current position for it to be filled by the next promotable person. The expectation is that all service members progress in their duties and become promotable. If a person does not earn/warrant a promotion after a certain amount of time, that indicates lack of progression. Eventually, if more time is spent in the same position, the military will seek to separate them from service as unpromotable.
Warrant officer positions are a little different than other officer and enlisted positions. The warrant officer is a valuable subject matter expert. They may stay in the same position as long as the military and the person come to an “agreement” as such. This still does not rule out frequent transfers to the same position elsewhere. Just not as often in most cases.
Based on observation alone, the Star Fleet Table of Organization and Equipment (TOE) precludes having an officer above the rank of Captain command the Enterprise. Under a Modified Table of Organization and Equipment (MTOE), a higher ranking person can be added to the crew under specific or special circumstances, as the situation dictates. This higher ranking person would command the captain of the ship due to their outranking the captain.
Therefore, Pike’s departure from the Enterprise was inevitable and expected. No explanation would be necessary. Spock’s retention as science officer is less clear. He might have been classified as the preeminent subject matter expert and exempted from the normal line of progression even though he was not a warrant officer. Or, his lack of people skills might have limited others’ perception of his leadership skills until they were proven under (and attested by) Kirk, his superior.
Coincidentally, the US military has toyed with the idea of exempting pilots from the progression structure in order to make the position more appealing to the best candidates. Instead of frequent duty position changes and an eventual promotion out of the cockpit to fly a desk, pilots will be retained in their positions for a longer period of time. That way, their family lives are more stable. And, they have the opportunity to do the thing they love most, flying, while still receiving promotions and pay increases. It will be a voluntary exemption. Pilots that chose to continue to climb the military political ladder may still choose to do so.