The initial motivation was to compete with television, after that it was more like an arms race
The original aspect ratio for film was based on an arbitrary choice in the very early days of cinema based on the standard 35mm film used. The resulted in several decades where 1.33 (4:3) was the standard modified (almost imperceptibly) to the academy ratio of 1.37 to accommodate optical recording of sound on part of the film.
Things stayed with this until television became popular. This worried the cinema industry as TV was taking their audience. So they decided to fight back by doing something TV could not do: offer "bigger" images in a different aspect ratio. The first way to do this was Cinerama (first used in 1952) which produced a ratio of 2.59 by using three conventional cameras both to film and to project. This was successful but expensive. The fact that it was a success spurred an explosion of innovation driven by the desire for "bigger" pictures (or wider ratios than TV) but constrained by the cost of the various systems. This was effectively an arms race of new technology constrained by cost.
Paramount first responded (in 1953) by releasing movies in 1.66 ratio by cropping academy shot film and projecting on a bigger screen (but this compromised image quality to get a wider picture). In 1953 an old invention, the anamorphic lens, was used to shoot wide screen on conventional film. This was CinemaScope which had a ratio of 2.35. This was followed by other innovations like VistaVsion in 1954 which gave a 1.85 ratio.
Innovation didn't stop there. in 1954 some studios started to use 70mm film to give ratios of 2.2 and even 2.76 with anamorphic lenses (as used for the MGM 65 process in Ben Hur).
Since then innovation has continued in an attempt to balance the cost with the desire to have non-TV ratios.
When TV started to catch up in the 1980s with new video standards a new compromise ratio was developed for video production that would minimise the empty space on screens when movie ratios were shown on TV (and eventually DVD and higher definition broadcasts such as HDTV). This was a ratio of 1.78 (16:9) and this is what most small screens and digital video formats such as 4K are now based on.
The proliferation of formats was initially driven by the need to do better than TV but became an arms race of new formats driven by the need to trade off the technology and production cost of new formats with the benefits of non-TV ratios.
Most of the facts here are derived from this excellent video by FilmMakerIQ.com which is well worth watching for more detail.