To me it seems like the other answers to this question so far are wrong; at least based on what the film is telling us about the physics of what's going on.
If one tries to take the approach that the objects are manufactured with "defects", a simple watching of the film makes it clear that doesn't seem to hold true at all, some examples being:
This doesn't explain how the affected objects, such as the cracked glass or car mirror, tend to only get these "defects" right before the point in time when the inverted object impacts them.
Something like the building at Stalsk-12 clearly can't have been "manufactured with a defect" of having been blown to pieces.
It obviously doesn't work for humans, such as Neil, who can't be "manufactured" with the defect of a hole in his head and lying inside a bunker.
To understand what's actually happening, one must consider what Neil says in the scene where you get the most explaining of the physics (even though it's not much), which is when he and Protagonist are inverted with Kat on the boat as they are headed back to the airport to reinvert themselves there. In this scene Neil describes why Protagonist survived:
But because the environment's entropy flows in our direction we dominate. They're always swimming upstream. It's what saved your life—the inverted explosion was pushing against the environment.
This is what Protagonist humorously refers to as "pissing in the wind".
In other words, wherever you have inverted objects attempting to impact regular objects, their effects will tend to struggle against the greatly dominant entropic direction against them, which will essentially work to erase their impact. In fact, not only will their impact be erased, but the entire object itself will with time be whisked out of existence completely, presumably along with an equal portion of regular matter, and taking more time to do so. I'll get to some more about that later.
Think about it in terms of the aforementioned examples of the glass and the car mirror from the inverted perspective: we see that from the regular perspective, the glass and mirror seem fine, until they slowly start cracking as the event of impact approaches, at which point they're suddenly fine once more; this means that from the inverted perspective, the inverted bullet hits the glass and the car hits the mirror respectively, both cracking them, but with time the entropy of regular objects work against and erase their impact until they disappear completely. The exact same is true for the building at Stalsk-12, except it's destroyed in regular time as well immediately after the inverted impact (which would be like breaking the glass/car mirror right after it "fixes itself" in regular time upon encountering the inverted object that broke it); what isn't shown is what happens to the building before being destroyed in inverted time, i.e. before the events of the battle in regular time, which would be the building, just like the glass and car mirror, going from its intact and standing state and breaking apart out of nowhere to leave it in the state it's in after the inverted destruction.
To try to illustrate these three examples more clearly, here's a breakdown of each from the perspective of regular time:
Glass:
- Gets manufactured fully intact.
- Slowly starts to crack out of nowhere, and the inverted bullets also appear out of nowhere over a longer period of time.
- The bullets fly back into the inverted gun through the holes in the glass, instantly fixing them as they do so.
- Is once again fully intact, just as it was manufactured.
Car mirror:
- Gets manufactured fully intact.
- Slowly starts to crack out of nowhere.
- The inverted car hits it and it is instantly fixed.
- Is once again fully intact, just as it was manufactured.
Building at Stalsk-12:
- Gets manufactured fully intact.
- Slowly starts to crumble and fall apart out of nowhere.
- Gets hit with inverted rocket launcher blast, instantly fixing it.
- For a brief moment is once again fully intact, just as it was manufactured.
- Gets hit with a regular rocket launcher blast and is destroyed.
Hopefully this makes it clear how those things are supposed to work.
Perhaps the hardest thing to swallow about this explanation is what I briefly mentioned above, i.e. the entire "disappears into nowhere" and "appears out of nowhere" part, but it really does seem to be how it works. To me it seems implied that inverted objects interacting with regular objects will seem to shift each other into a direction of time that's not parallel to either, thus seemingly disappearing completely from our view (but perhaps recoverable through an even more advanced future understanding of physics), or in the other direction appearing out of nowhere. There are at least two prominent points that come to mind to suggest this is indeed the case:
Wheeler warns about not touching your inverted self, as it would lead to "annihilation"; this is of course specifically about yourself, but in the same scene is also when we're introduced to the point of needing inverted oxygen to breathe as well, strongly implying that trying to inhale air that's inverted from your perspective could be very dangerous. In any case, the phrasing "annihilation" seems to allude to the type of "annihilation" that occurs when matter meets antimatter.
The turnstiles clearly make it seem like objects either appear out of nowhere in one direction of time or disappear into nowhere in the other; i.e. when you use a turnstile to invert something, you will see it move into the turnstile on the other side as well, and if you remain outside the turnstile, both will have disappeared into nothing, whereas if you're standing outside a turnstile as someone is inverting from the other direction of time, they will appear out of nothing. This is the most telling about how the objects are not really appearing or disappearing, but "simply" changing temporal direction, seeming to us to disappear due to how we continue in our own temporal direction.
To me it seems like this is clearly how the physics are supposed to work in the film, both as explained and as shown in the various events.