I believe that destructive effect is only for the Power Stone. All the others are held without a problem multiple times as you pointed out in many MCU films. But when someone as strong as Thanos removes the Power Stone from the Gauntlet to power punch Captain Marvel, it shows him struggling and in pain from using it like that, before quickly placing it back.
Instances of Infinity Stones being held in their raw form include:
The Mind stone
Is studied throughout Age of Ultron (but not handled directly) and finally is inside Vision his whole life.
The Space stone
Is broken from the Tesseract and handled by Thanos in Infinity War.
The Soul stone
Is held by Thanos and Hawkeye after being acquired on Vormir, both time held in hand without issue.
The Time stone
Is held by Thanos in Infinity War and by Hulk in Endgame.
The Reality stone
Never seen being held in raw form
The Power stone
Held by the Collector's assistant, Carina, who was violently ripped apart by it in Guardians of the Galaxy later it's held with difficulties by Peter Quill and The Guardians and again with issue by Thanos in Endgame