This can change depending on the scale of the production, the star's profile and the nature of any preferential deals or professional networking that can occur with a specific agent/agency.
You'll notice in all films/TV there is typically a credit for 'Casting', and this is the person in charge of talent acquisition.
A casting agent can be hired on the basis that they have a track record of acquiring a certain level of Star (someone can be chosen if they've consistently landed A-List actors for a role), and it's the casting agents job to 'sell the film' to a prospective star, and to oversee the hiring process. They obviously work closely with the productions producer and a director; if one has been appointed before a star is cast, which isn't as common as you might expect in a Hollywood Studio system, where films are often vehicles for a specific star.
There is no set protocol of whether an actor has to audition, and whether this is a formal process or simply sending in 'a tape' of them reciting part of the script.
It stands to reason that the bigger the star, the less likely they'll have to audition. With someone like Robert De Niro or Christopher Walken, they'll have such a pedigree that the studio knows exactly what it's getting, and will often write the role specifically for that actor.
Contrastingly, if someone is being cast against type (say Arnold Schwarzenegger in a RomCom), odds are they'll still have to audition for the role.