Director is a much bigger role
A film's director is "chief artist" for the entire project
All the other creative roles we audiences pay attention to are under the command of the director, who is ultimately responsible for everything: tone/mood, pacing, characterization, framing/mise-en-scene... I honestly don't even have a complete-ish list of the myriad creative dimensions that exist within film. This is enshrined by auteur theory (which is not the only way to approach film appreciation).
The thing about creative work is that it's not merely more effort: for every one of these dimensions, you have to generate ideas and/or cultivate the creative energies of others, curating all of it in service of some grand vision that doesn't exist until you and your assistants give birth to it. There are plenty of people who wake up energized every morning but are not prepared to create ex nihilo.
Yes, it is entirely possible to make a movie in which all these dimensions are not meticulously aligned. Those are called "mediocre" -- when anybody bothers to call them anything at all.
A film's director is significant for disgusting business reasons
Audiences care who directs a film: some directors draw people to the box office, others repel them. So the people who fund movies as a form of financial investment also care who directs a film: they always want guaranteed returns, so they always want to pick a director who audiences love. Other creatives like the screenwriter often care, because they have no choice but to trust that this person will do something good with their input instead of fumbling it horrifically. Actors care too: some directors are great with actors, and others are famously terrible, and every starring role is a pivotal moment in an actor's career. The film can be received well, or poorly, or be a non-entity (which is sometimes just as bad as poorly).
It's worth noting that none of these people is hoping to make a mediocre movie. They are hoping to find a director who will marshal every dimension, every trick in movie-making, to craft a powerful film that audiences will pay (repeatedly) to see.
So setting aside the actual directing duties that a director has, there's all this extra (and frankly stupid) pressure on that one role to keep all these powerful, selfish stakeholders placated.
A person can avoid all this insanity by taking just one step down, where they can focus much more on the craft and art.
"Life is what happens while you're busy making other plans." It seems quite plausible to me that plenty of second-unit directors make "temporary" plans to not reach for the brass ring ASAP, and the next thing they know, they've got gray hair, aren't so hungry, and a lot of life's doors have closed.
