Red Matter - The rare mineral decalithium is processed into red matter that creates a black hole when it comes in contact with nuclear matter.
It was presented with a lot of artistic license which made it red when in actuality it wouldn't be and also the matter being "destroyed" (we know it can't be, but Vulcan went poof) yet the spacecrafts didn't, just transported across time/reality. But the fictional inaccuracies aside, its simply an unstable substance that produces massive energy creating a chain reaction that ultimately leads to a black hole. Black holes are theorized to bend spacetime and in this it can act as both massive energy that converts matter into antimatter (Vulcan just becomes antimatter feeding the black hole) and the magnetic wash it creates can explain the manipulation of time and dimension (although again no proof other than science fiction assumptions to date).
But considering some faint elements that are rooted in actual science of quantum chromodynamics. The principle of color charge used to explain quark confinement (basically why protons/neutrons can bind together forming lager groupings) has three aspects of primary colors RGB, though has nothing to do with actual color. Although unlikely this Red Matter was a Gluon, it could still be explained away as possibly some exotic form of Quark matter. This would explains its "destructive" capability (again I use that loosely as we know energy cannot be destroyed just transformed) and that amount of massive energy I suppose could facilitate a shift in spacetime.
Hope that helps.