TLDR:
There are plenty of films where the "bad guy" wins.
Ultimately, the reason why the bad guy wins can come down to a number of reasons, including to be more realistic, to set up a later "good" ending in another film or because it's unexpected, to name a few reasons.
Long Answer (note: there will be spoilers for some films below):
There are really a few different things to look at here. For example:
- Films where a villain "wins" by physically or literally defeating a good guy.
- Films where a villain "wins" by psychologically defeating a good guy
- What exactly is a good guy versus a bad guy.
- What exactly does winning mean?
Now, the first film that comes to my mind when I think of villains winning is Seven. In it, two police officers hunt down a criminal who is murdering people based on the seven deadly sins. The film ends with the "bad guy" manipulated one of the good guys into killed him, thus fulfilling the "final" deadly sin.
Is this a win? I would consider it so. The good guy's morality was damaged and the bad guy got what he wanted - even if it meant his death. Therefore, I'd consider this under Option 2 above - psychologically defeating the good guy.
Another famous example is No Country for Old Men, where the "good guy" comes across a lot of money at the start of the film, and spends the film avoiding a hitman who is after him for it. He doesn't succeed. He is killed towards the end of the film. We see the hitman get involved in a car crash. He could have died. Instead, he recovers and walks away.
So the good guy is dead, the bad guy survives and walks away. A very clear example of Option 1.
Similarly, The Usual Suspects would fall down this road. But what about The Empire Strikes Back? Or Harry Potter and the Half Blood Prince?. Or The Hunger Games: Catching Fire. The "bad guys" certainly won at the end of those films - but do those count, since they were part of an overall series where the good guys won?
When looking at Option 3, I mention the difficulty in defining good and bad. In years gone by, black and white hat symbolism could make this a simple task. But in modern media, it's increasingly difficult. Take a show like Game of Thrones - how does one determine who is "good" and who is "bad"? Everyone will have their own subjective opinion.
And what about horror films? Quite a few of the Saw films end with the bad guy winning. Many other horror films do. Do we count these?
Similarly, what about Option 4? What is winning. Does Amy win at the end of Gone Girl, a film about a woman who intentionally disappears to heap suspicion on her husband, but returns when it backfires (having murdered another person) and forced him into a situation where he must stay with her. Did they both lose? What about Nightcrawler, where a stringer who films live footage of crimes sets up a scene to kill his partner, so he can film it. He was the protagonist of the film, but that didn't make the creepy ending to this film any more comfortable - but it wasn't supposed to be comfortable.
There is a lot of subjectivity around all of this.
Therefore, I'd propose to set aside Points 3 and 4 for now. If we just focus on obvious examples of victory, through defeating the good guy or psychologically breaking him, there are a number of fascinating reasons why a film might let a bad guy win:
Unexpected
It is unexpected for bad guys to win. We got to the films, and watch TV, to be entertained. Often this involves rooting for the good guys, as many of us identify as good ourselves. Of course, these guys might be rogues (like Captain Jack Sparrow, Han Solo), but their characters are written in such a way as to be interesting, relateable and fun.
When the tables are turned and the good guy loses, it suddenly becomes unexpected and adds shock value.
Cliffhangers
Take a show like Mortal Kombat: Conquest. Its first series ended with almost all the good guys dying. The intention was a second series would have resolved this (supposedly by making the first series ending a dream). However, the show was cancelled. So for all of time, there is a depressing ending!
There are slightly different examples like I mentioned above, where films that are part of an overall story arc can have cliffhanger endings where the "bad guys" win - although this is often to set the good guys up for an even bigger win later on.
Realism
Films like No Country for Old Men displayed "bad guys" winning because it was a genuinely realistic finale that was intended to both shock and horrify. And it worked, brilliantly.
Historically accurate
Imagine an infamous criminal from modern or historical times who got rich, lived happy and survived until old age. He has technically "won" (and quite possibly defeated many good guys along the way). However, as this is a film that's historically accurate this "victory" will be shown, even if people watching the film find it distasteful.
So how do audiences react?
This is very difficulty to objectively judge. Are we basing this on critic reviews? On box office revenue? For films like The Usual Suspects and Seven, they reacted brilliantly. The endings are adored. For others, they are not.
Conclusion:
Ultimately, there are plenty of films where the bad guys win.
However, the biggest problem I believe for studios in deciding whether the "bad guy" should win, is understanding why he is wanted to win. If it is just "for fun", they're much less likely to do it. If it's for a twist ending, for shock value, to reflect the brutality of a situation, or to set up an ending for a later, more positive film - then it's much more likely to be acceptable.
Film studios do care about audience opinion. If they feel the audience will appreciate the ending or if it fits with the theme of the story, then they'll proceed with it.