In Avengers: Endgame, in a scene with Bruce Banner and Sorceress Supreme, we learn that the infinity stones create the flow of time and that if any of the stones are removed then they will create an alternate "doomed" timeline. We also learn from Infinity War that the stones were created from the big bang and each control an essential aspect of existence.
So if the stones each control an aspect of existence and can't be removed from a timeline without dire consequences, then how can they be destroyed without the same consequences?