To be honest with you, the best answer to this question is the adage that beauty is in the eye of the beholder.
There are a plethora of reasons why actors looked so drastically different back then, some of them social, some of them technical, but to assert that they were any less handsome is a misnomer.
Popular culture of the 1950's (and also but a little less the 1960's) circulated different ideals of what was attractive to an audience than we do today.
The 1950's was an interesting time in Hollywood, as the 'Leading man' persona was being jeopardized by the collapse of the star system, yet also given ballast by the post-war victory movies that were churned out at an unimaginable pace (compared to the rate of production today).
These movies privileged the idea of a leading man being strong, rugged and masculine. It's possible to represent this conservative view of masculinity with a younger male, but there was very little precedent for younger male leads at the time, it was simply felt that Audiences would respond better to more established stars.
There was also fierce patriotism to contend with, not only because of McCarthyism and the Cold War, but because of a strong development of American national identity following WWII.
Also, many of the older stars had embedded themselves into not only the public consciousness but the machinations of the studio industry.
The Studio system that was prevalent in the 1950's and 60's used contracted stars, who would be used for a predefined number of pictures. These contracts HAD to be honored, so as such there was little room for new talent.
This situation became pretty dire (as it extended to production crew and directors alike), and caused what was tantamount to crisis in the late 1960's. The only way for studio's to re-engage with their audiences was to allow New Creative Talent to take over from the stalwarts of the previous generation.
This transition became known as New Hollywood, and is what gave birth to the American Auteurs (Coppola, Kubrick, Peckinpah etc..). you'll notice that after around 1968 (ish), films start to become not only more graphic and controversial, but also took greater risks in using unknown, younger actors. Dustin Hoffman and Jack Nicholson got their big breaks during this period, to name but a few.

Also, the 1950's was (for the most part) before the Teen Explosion: the introduction of 'The Teenager' as a marketable entity, with its own personality and traits. James Dean was at the threshold of this movement, and for most of the 1950's was really the only younger megastar.
This list of The most Handsome actors of the 1950's has James Dean as the only actor under the age of 30, and he died in 55'.
To Summarize, the 1950's idea of a leading male was an older, wiser, stronger representation of masculinity than we have today, and as such they picked their stars to match.