In Inception, it was told that the totem's concept was invented by Mal.
Then why didn't Mal use her spinning top to find out if she is in a dream or not before jumping off the window?
Is it the case that she didn't remember that she has a totem?
Because you can't battle a notion once it's successfully inserted into your subconscious.
Cobb: An idea is like a virus. Resilient. Highly contagious. The smallest seed of an idea can grow. It can grow to define or destroy you.
Nothing could convince Mal. Presumably, Cobb had better ways than a spinning top to prove to Mal that this is reality and that the world outside and her family and friends are all real. But she wouldn't believe any of it, thinking it's all fake. This is why Cobb is wary of inception in the first place.
EDIT: As onewho correctly points out, Mal also has every reason not to believe her totem anymore because it's what Cobb used for her inception when he spinned it inside her safe. So the byproduct of the concept 'your world is not real' might be 'and your totem has been tampered with'.
Ultimately, this was the problem with Mal's inception. It was predicated on manipulation, which led only to doubt and mistrust. Compare it with Fischer's successful inception which ended positively because he eventually filled the safe with his own concepts and its content wasn't manually rigged by others.
A totem was an object with some behaviour which was considered to be a secret known only to its owner. Totems could be used therefore to determine whether someone was in someone else's dream because the dreamer would not know what the secret behaviour was, but they could not reliably be used to determine whether someone was dreaming at all, because if the person was the dreamer (or in limbo I assume), the dream version of the totem would obey their own knowledge of how the real world one would behave.
That said, I agree with Walt that the bigger issue was that once she was convinced, she was convinced.
A totem is an item, in the real world, with a secret, unique property. It's purpose is that, when somebody targets you for extraction and tries to counterfeit that object in the process of counterfeiting the rest of the world, you will notice they have missed that detail and you are able to know that the world is not real. This idea was invented by Mol.
The first time this concept is introduced in the movie is when Saito rubs his cheek on the rug and instantly knows that, despite appearances, he is not really lying on his carpet in his apartment.
As far as the audience knows, the first time Cobb ever actually touches Mol's totum it is really just a projection in the dream realm.
Mol's totem is in the shape of a top which the audience is led to believe, through Cobb, is perfectly average in reality but will spin forever in a dream. This totem both:
One logical conclusion of all this is: If somebody else believes, incorrectly, that they know the special property of your totem and they try to use it, they are in danger of fooling themselves into a trap. The same way building familiar places can make you forget that you are in a dream.
This tells me that Cobb is actually incorrect about the special property of the totem he took from Mol, and it topples, not because Cobb is in the real world, but because he believes he is in the real world and has misunderstood how the totem works.
Considering all of this, Mol's totem was still reliable to her and it allowed her to accurately know that they were both still trapped in Cobb's dream.
I agree with the answer Walt gave. However, I also think it doesn't matter to Mal whether she's awake or dreaming.
The idea implanted by inception is stuck in her psyche: if she dies, then they can be together forever (or something like that). It was pushed into her mind while she was dreaming, and it was still in her mind when she was awake. So the totem doesn't matter because it doesn't matter to her whether she's awake or asleep.
It's a tragic example of how powerful inception is.