In The Empire Strikes Back, when Luke is leaving for Bespin, Yoda implores him to remember his failure at the cave. Obviously, this is a reference to the cave that was strong in the Dark Side and where Luke faced the Vader illusion. However, what was Luke's failure? What should he have done in the cave?
Luke's failure in the cave was his fear.
- He was afraid of what he might find so he took his light saber with him to fight what he might encounter - despite Yoda having told him that he would only face what he took with him.
- In his fear, he "killed" the Darth Vader apparition. It wasn't a reasoned response to a threat, it was a reflex reaction to a fright.
- The dead Vader apparition had Luke's face, revealing Luke's deeper fear that he may fall to the dark side.
Luke's failure was that he was still ruled by fear - and that can be manipulated to make him fall to the dark side.
There was nothing specific Luke should have done differently. To not fail the cave encounter, he had to be free of fear. That could lead to many results in the cave - but we don't know what could have gone differently because he was afraid.
After thinking, rewatching the scene, and considering the comments, I've heavily edited my answer.
Luke's improper actions:
- Yoda told him he would not need his weapons, but he took them inside anyway
- When the cave showed him Vader (which Luke must have in some way brought into the cave, since all the cave contains is "what you take with you"), instead of being calm and at peace to know the dark from light, and instead of using the force for knowledge and defense, he drew and activated his lightsaber
- After the Vader figure attacks, Luke does not back away, flee, force jump out of the cave, or otherwise try to avoid killing the Vader figure - he meets aggression with aggression
It doesn't seem unreasonable to me to see the cave as a kind of spiritual or force "mirror". It shows you something about yourself. So we can interpret Yoda's use of the word "failure" to mean that Luke did some thing(s) wrong regarding the cave (bullets above), and/or that what the cave reflected back to Luke shows that Luke is not yet ready and not yet a Jedi.
I think focusing on the word "failure" and trying to pick it apart might be missing the point: The cave showed, one way or another, where Luke is on the path to being a Jedi, and it wasn't great.
I feel Luke's failure is his impatience.
Yoda: If you end your training now, if you choose the quick and easy path, as Vader did, you will become an agent of evil.
We see it when Luke obstinately takes his weapons into the cave, against Yoda's suggestion:
Yoda: Your weapons... you will not need them.
What he takes into the cave with him is actually his Skywalker heritage; it alludes to how Luke is the son of Vader:
Luke: What's in there?
Yoda: Only what you take with you.
We see a premonition of what would happen were Luke to simply destroy Vader with his lightsaber. In particular, we see Luke's lifeless face on Vader's apparition, which highlights the consequences of killing one's own father: it's akin to killing oneself. Luke doesn't understand the premonition because he is not yet ready.
Nevertheless, with his friends suffering, Luke impatiently seeks a kind of "quick fix" (defeating Vader solves everything). It is in this context when Yoda says:
Yoda: Remember your failure at the cave!
As Luke matures, in Return of the Jedi, we see that Luke has overcome his impatience:
Obi-Wan Kenobi: You're no longer so reckless now, Luke. You are strong and patient. And now, you must face Darth Vader again!
He even discards his lightsaber vs. both Vader and the Emperor.