The reason why musicals are less popular now, or more prolific back in the first half of the century is pretty long, but hopefully engaging and interesting. It certainly was to me when I studied it. There are tons of academic books written about the downfall of musicals, but here's the short(er) version:
Musicals (along with Westerns) were very much a staple of the now defunct Star System that American studio's used to participate in. As such, if you could make a Musical Western, you were laughing all the way to the bank.
As you have correctly pointed out, these days Musicals (and Westerns, for that matter) are only produced as Prestige Pictures, and there are reasons for that...
The 1920's saw a technological leap forward that would change cinema forever: Namely, the introduction of sound. The Jazz Singer featured the first sequence of synchronized sound in a widely distributed feature film, and was met with immediate success. From its very inception, the use of sound in cinema was linked with music, and more specifically yet singing, so the connection to musical theater was recognized and explored immediately.
The Star System was already in place, and as such singing and dancing were added to actors feature lists to make them all rounders. Early successes of the pre-sound cinema, like Florence Lawrence, the 'Biograph Girl' fell to the wayside in favor of more musical fare. Appropriately, the Biograph Girl's story was turned into a musical itself, and both The Artist and Singing in the Rain borrow elements from her story. The studios enjoyed great success from the stars they promoted into musicals, and there was global appreciation for them for many years.
It wasn't until the late 1950's that things within the Star System started to fall apart, and it was just as much the studio's fault than it was that the audiences didn't want to see musicals anymore.
You see, for a long period The Studio System enjoyed Vertical Integration, meaning that they owned and controlled their entire production, distribution and exhibition processes themselves. This level of control may seem beneficial, but it whittled movie production into only 5 competing studios, 'The Big Five': MGM, Warner Brothers, 20th Century Fox, Paramount Pictures, and RKO. (There was also 'The Little Three', but as their nomenclature would indicate, they weren't as big!)
The result of this set up meant that Studios were only really competing with a small market, and as they had all more or less decided to keep out of each others pockets, very little even there. This meant that there was no competitive market to decline or bargain for movies, so the only thing that could influence what sort of film was being made was a studio executive in his office, who were hardly representative of the masses!
It wasn't until 1948 that the Supreme Court ruled that this system should be broken up, and it was itself only delayed because of the focus on the war. This meant that the studios' control was finally broken up, but it had very little real effect to the Big Five who had used their time wisely to consolidate their assets into a firm grip on the industry.
However, for the time being, the studios continued to prosper. Different Genre's came in and out of popularity, and so not being a genre itself, but more of a method of application Musicals survived by simply adopting the genre a la' mode and wearing it like a mask for a while.
WWII nostalgia films become popular? War Musicals.
Warner Bros. have a spate of successful Gangster movies? Gangster Musicals.
It didn't matter whether or not people actually wanted to see these movies, there was no choice in the matter and as a result of that, they enjoyed the illusion of popularity, not to mention being genuinely popular anyway as a means of family entertainment in a very conservative America.
The real killing blow to musicals came with the same blow that threw the entire industry into crisis: advent of Home Television. It was a slow death however, over a number of years. For the most part, the Industry didn't even realize it was dying. People had been provided with Choice for the first time, and for an industry who's unofficial motif was "You'll get what you're given", this was bad news...The motto slowly became its epitaph. Some Studio's began to experiment with Television, and set up sister studio's in the rival industry, but even this did little to change their programming output.
The rise of Suburbia also struck its blow, as people were expected to take long journeys into cities to see movies, and the Television was already sat in their living room. It would have to be something pretty impressive to budge them, so in a way they were voting with their feet.
Still, the studio's remained stubborn in their refusal to adapt. Whatsmore, the Hays Code was still very much in place, not only prohibiting certain types of film but encouraging Studios to stick to what they knew already, stifling any creative experimentation.
When this all finally came to a head in the late 60's, the industry was in crisis. It didn't know, or more accurately didn't care, what their audiences wanted anymore.
The only solution to this was tantamount to almost total replacement.
Crew and Cast from leading men to lighting operators, Directors to set designers were systematically replaced for younger, fresher and more in touch counterparts. This became known as The New Hollywood, and produced what many people consider the greatest movies of 20th century's second half.
This came at its price...
The Directors had to fight for years, and continue fighting, to wrestle creative control from the old guard. They heavily resented the former ringleaders of the industry, and poured scorn on their pitiable output of incessant Musicals.
Furthermore, many of these directors looked up to the writing of Cahiers du Cinema, and the European cinematic giants who wrote for it: Bresson, Goddard, Truffaut to name a few. These were figures that hated Musical Cinema, not only for what they considered to be unoriginal copies of stage productions, but for the American Imperialistic intent these films harbored as they washed over Europe. This hatred found its way stateside and embedded itself into the New Hollywood.
The Hays code was finally lifted in 1969, giving these film makers a freedom no one had experienced since the mid 1920's. And what did they do with this freedom? They ran as far away from Musical cinema as possible.
The only studio to continue exploring Musicals as a prolific statement is Disney, for obvious animation reasons. It is for this reason that they are able to credibly produce programmes like Glee and movies like High School Musical: They have a fair claim to owning the Modern Musical Mastership, and they pretty much kept it breathing for the last part of the 20th century.
So, until the late 60's, audiences were forcibly saturated with Musicals, and came to collectively loathe them, even if it took a bunch of pretty spiteful up and comers to point this out to them.
These up and comers ended up taking on their own downfall, however, proving history repeats itself. But that is a story for another time!