There's a lot missing with natural light and there's a lot that can't be done if you rely on natural light and most of it relates to being in control of your set.
Natural light is great... if you want natural light. It has a warm glow that feels really nice on film. It can't be beat... if that's the look you want.
Natural light isn't always what a person wants, though.
Sometimes you have a grueling scene that's going to take time to get through... meaning hours... guess what that light's going to do... That big ball of light in the sky (the sun) is going to move. And if the scene is supposed to take place in 15 minutes but it takes you six hours to shoot it... your light won't match when you go to edit the scene. It sounds minor but it can be very noticeable.
With a light kit, you control the "sun". It doesn't move. You can shoot for 12 hours straight and never move the "source" and your light will be perfect and constant.
Malick loves to shoot in real locations and, being an auteur, choosing to self fund most of his projects and producing them relatively inexpensively... he has that option. He shoots a lot outside, in actual houses... places with windows and available light to use and he shoots simply - he catches a lot "in the moment" without necessarily retaking shots... he wants it to feel real. If you're shooting this way (particularly outside), you don't need lights... or you can't use lights because you're limited by the sun (unless it's night).
Many movies (and particularly TV shows) are shot on sound stages. They're giant, dark spaces designed to keep the sun out so that everything can be controlled. Natural light isn't an option... and, often, they don't want it to be because they want that control. They want to be able to shoot night for day or day for night... if they can control the light, they don't have to shoot overnight when they need a scene that happens at night, they just cut down the lights outside the "windows" and shoot at noon. It's practical. Night shoots are expensive and people hate them.
Another thing is that it's easier to get good quality capture when things are "over lit". If you've never been on a set before, you'll be surprised by how bright it often is. You have more options in your lens's aperture settings, which allows you to choose from a wider range of focal depths. If you don't get enough light from natural light (as in shooting inside), you're limited in your aperture choices.
Light also sets moods. Sometimes you want that grainy green look of horrid florescent light... Sometimes you want a shaft of light to fall across something that's otherwise in shadow. Sometimes you want to shoot at night, which requires that you light the area somewhat. You (as a human) may be able to see well by moonlight but camera recording media (whether film or digital) is not great in low-light situations... it needs more light to record images, so lights are necessary.
You seem to be under the impression that nothing happens while gaffers are at work setting up lights.
If they're doing the initial lighting of the set, it's usually first thing in the morning and everyone else is doing other stuff. Camera is setting up their equipment, sound is getting ready, cast is in makeup and wardrobe, the director is reviewing the script, talking with the cast, the producers... everyone is busy.
If they have to adjust the lights in the middle of the shooting day, the time is often used for talent to rehearse lines or blocking, to discuss the scene and the emotion in it with the director or with each other... or to have their makeup and wardrobe touched up. The camera will often also be using that time to set up their frame, any camera movement etc.
This time is valuable and used by many different departments, not only the lighting team.
Also, if multiple sets are being used in a day, the gaffers may rig one set while everyone else is at breakfast and leave one or two people on call to "fix" stuff... and then send the rest of the team to light another space so that it's ready when they move on to that scene (assuming they have sufficient gear).
Yes, there are certainly times where you're waiting on the gaffer to fix something but it's not usually hours or even tens of minutes... unless it's a major change, you're only waiting a couple of minutes.
As to how much time you spend waiting for a gaffer vs how much time you spend waiting for natural light... well, if you're on a sound stage, you'll be waiting for that natural light for years... until the building falls apart and the light comes through the cracks in the walls.
Any sort of interior set requires lighting unless you're in a magical house with windows everywhere.
So, let's look at your points:
- I overestimate the relative cost/time of lighting a set/location
As I've tried to explain above, you assume that this time requires everyone else on set to wait for the lighting crew to do their work. This is not the case. Keep in mind, too, that many studios own their own lighting kits for anything shot on the lots, so paying for them on a per-film basis is not really an issue. They may have to buy them to have on hand and then pay for maintenance and storage but they don't have to buy or rent new lights for every film unless the DP wants something really fancy and new... which will probably be used on future projects, so the cost gets shared.
If they're on location and have to rent, yes things can get very expensive but it really is worth the cost.
- I underestimate the value that others place on the visual aesthetic.
It's amusing to me that someone who appreciates Malick's work doesn't care about aesthetic. His films are all about aesthetic. The entire job of a cinematographer is to make a film look unique, special, and (usually) beautiful. Without them, everything would look like daytime TV or home movies. Trust me, you care more about aesthetic than you think you do... it's just that good cinematographers do what they do in a way that you don't recognize the work that went into it.
- Lighting practices are vestiges of a time when inferior film technology necessitated massive lighting setups, and inertia has kept them around.
No. In fact, digital media is (at this point) worse with light capture than film stock is or was. I discuss this at length here. If anything, shooting digital requires more work on lighting, not less. Also, with the addition of LED as a lighting option in the last 10 years or so, the choices for lighting are much more varied now than they used to be. Lights are getting smaller, lighter weight, and don't put off as much heat, so they're more energy efficient and less expensive to run.
- Studio contracts require the filling of potentially unnecessary positions.
These positions aren't unnecessary. Even if you shoot everything you possibly can in natural or available light, that light still has to be shaped by someone. Grips still have to flag off overly bright sunlight. Someone has to pick which color temperature of bulb to put in the fixtures on set and shade off any unwanted lighting. Someone has to be there when you want to shoot at night to use actual lights to give structure to the set.
Movie studios are cheap... and independent filmmakers (who I've worked with a lot) don't have money to waste... they know that if the lighting is good, they'll get a better product. If it was more cost effective to not spend a huge chunk of budget on lighting and crew, they'd do it... it's very much worth it to spend money on lighting and gaffers.