You are right that the Asgardians are more like a highly advanced alien race with superhuman abilities. They do have a lifespan, but it is much longer than that of humans. As Odin says in Thor: The Dark World, "We are not gods! We're born, we live, we die, just as humans do." Loki adds, "Give or take five thousand years."
Based on the movies and the official MCU timeline book, we can estimate the ages of some of the main Asgardian characters:
Hello there! Thor tells Surtur, "I thought my father killed you half a
million years ago." Now, we know Asgardians live a long time, but do
they live that long? It's hard to say how long Odin has been around,
'cause sometimes he falls into the Odinsleep.
- Marvel Studios’ The Marvel Cinematic Universe: An Official Timeline
So, to answer your question, Thor, Loki, Odin, and the Asgardians are very old in Earth years, but not immortal. They can die of old age, or injury, just like humans. This is confirmed by the Time Variance Authority (TVA) (introduced in the Loki TV series), the organization that monitors and maintains the main MCU timeline, also known as the “Sacred Timeline”, as their AI mascot, Miss Minutes, explains. From the Official Timeline:
Hey there! How long do Asgardians live? That's a great question! As Odin the Allfather will one day explain, they exist for a very long time but aren't immortal. He says they are born and die just like regular folks, but his son Loki adds: "Give or take 5,000 years." That's a whole lotta wiggle room!