In Inception, it was told that the totem's concept was invented by Mal.
Then why didn't Mal use her spinning top to find out if she is in a dream or not before jumping off the window?
Is it the case that she didn't remember that she has a totem?
Because you can't battle a notion once it's successfully inserted into your subconscious.
Cobb: An idea is like a virus. Resilient. Highly contagious. The smallest seed of an idea can grow. It can grow to define or destroy you.
Nothing could convince Mal. Presumably, Cobb had better ways than a spinning top to prove to Mal that this is reality and that the world outside and her family and friends are all real. But she wouldn't believe any of it, thinking it's all fake. This is why Cobb is wary of inception in the first place.
EDIT: As onewho correctly points out, Mal also has every reason not to believe her totem anymore because it's what Cobb used for her inception when he spinned it inside her safe. So the byproduct of the concept 'your world is not real' might be 'and your totem has been tampered with'.
Ultimately, this was the problem with Mal's inception. It was predicated on manipulation, which led only to doubt and mistrust. Compare it with Fischer's successful inception which ended positively because he eventually filled the safe with his own concepts and its content wasn't manually rigged by others.
A totem was an object with some behaviour which was considered to be a secret known only to its owner. Totems could be used therefore to determine whether someone was in someone else's dream because the dreamer would not know what the secret behaviour was, but they could not reliably be used to determine whether someone was dreaming at all, because if the person was the dreamer (or in limbo I assume), the dream version of the totem would obey their own knowledge of how the real world one would behave.
That said, I agree with Walt that the bigger issue was that once she was convinced, she was convinced.
A totem is an item, in the real world, with a secret, unique property. It's purpose is that, when somebody targets you for extraction and tries to counterfeit that object in the process of counterfeiting the rest of the world, you will notice they have missed that detail and you are able to know that the world is not real. This idea was invented by Mol.
The first time this concept is introduced in the movie is when Saito rubs his cheek on the rug and instantly knows that, despite appearances, he is not really lying on his carpet in his apartment.
As far as the audience knows, the first time Cobb ever actually touches Mol's totum it is really just a projection in the dream realm.
Mol's totem is in the shape of a top which the audience is led to believe, through Cobb, is perfectly average in reality but will spin forever in a dream. This totem both:
One logical conclusion of all this is: If somebody else believes, incorrectly, that they know the special property of your totem and they try to use it, they are in danger of fooling themselves into a trap. The same way building familiar places can make you forget that you are in a dream.
This tells me that Cobb is actually incorrect about the special property of the totem he took from Mol, and it topples, not because Cobb is in the real world, but because he believes he is in the real world and has misunderstood how the totem works.
Considering all of this, Mol's totem was still reliable to her and it allowed her to accurately know that they were both still trapped in Cobb's dream.
I agree with the answer Walt gave. However, I also think it doesn't matter to Mal whether she's awake or dreaming.
The idea implanted by inception is stuck in her psyche: if she dies, then they can be together forever (or something like that). It was pushed into her mind while she was dreaming, and it was still in her mind when she was awake. So the totem doesn't matter because it doesn't matter to her whether she's awake or asleep.
It's a tragic example of how powerful inception is.
Here is the screenshot from the movie Inception. This shot is just after Cobb enters the hotel room and Mal already ready to jump down outside window. It says she has used her totem, and it was lying toppled (lying on its side, immobile), meaning she was present in real world.
From this, it is clear that the idea incepted in her mind was so strongly incepted that she even could not believe her own totem.
Mal in Limbo trusts her totem. She has been spinning it for ~50 years, and it has been giving her the truth: by spinning forever, it confirms that she is still in a dream. This is what she expects.
Cobb is desperate to leave Limbo, but Mal wants to stay. So Cobb sabotages her totem.
We see that Cobb secretly finds it and studies its behavior so he knows what she expects it to do. We do not see what happens next, and he doesn't tell us precisely what he does, but there is only one possibility: he creates a duplicate top that behaves like a normal top, and leaves that in her safe instead of her special top.
Sidebar: Mal's totem is categorically absurd, because a person in the real world cannot create a real top that will have a specific impossible behavior in the dream world.
Ariadne drills a hole into a chess piece to make it topple predictably. What do you suppose Mal could do to her top in the real world that would cause it to reliably spin forever in dreams?
Arthur's totem, the loaded die, does not have this problem: real people can create real dice in the real world that are weighted but which look like real dice.
Arthur's die is a workable totem concept because it is simultaneously consistent with physical reality and epistemology. Mal's totem fails on the first count.
So, imagine what happens the next time Mal decides to use her totem: she spins it, and it behaves like a normal top instead of a magical one that spins forever.
A reasonable person presented with that evidence will not merely conclude that her reality has suddenly changed. No, she will start to doubt the basic idea that totems can be trusted, because -- from her perspective -- a totem which has been reliable for 50 years has suddenly started yielding a different result even though she hasn't exited the dream or gone into a new dream. (Another thought she might have, if she believed in inception, is that Limbo is not the bottom of all shared dreaming. I think it's reasonable for her to have discounted this possibility.)
Mal's conclusion is that totems are categorically unreliable. That is the belief that follows her into the real world, and it's why she can no longer be persuaded that the real world is real: she now distrusts all totems generally because she has seen what she believed is her own totem give her inconsistent answers. She won't trust her own totem, she wouldn't trust Cobb's totem, and you can't trust any totem that was created anywhere other than the real world -- which she can no longer ever be sure she has returned to.
A related misconception is that letting somebody else touch your totem breaks it. That's not what the movie tells us.
If somebody touches your totem, it does not magically "contaminate" the totem. It just means there is now a person who knows its secret, and that means they could create a dream that has a fake version of your totem that mimics the secret special behavior you expect, making you believe a dream is reality. In principle, if you really trust somebody, you could let them hold your totem, and that would not by itself invalidate the totem's operation.
What Cobb did to Mal isn't even "inception", properly so-called. He breaks her tool, and then her own perfectly good powers of reasoning are unable to fit a new theory to the facts because the facts she has truly are inconsistent -- because her testing apparatus has been sabotaged. A similar thing happened to the British Navy when they were trying to figure out the cure for scurvy, and they weren't being "incepted."
Sorry to necro this question, but I think the other answers miss the truth.