It's not specific to horror movies, but the event or action that kicks off the plot of a movie is typically called it's inciting incident.
This is a literary term that applies to almost anything with a plot. Usually there is a short portion of the movie, the "setup" or "backstory", then some seemingly minor event that triggers all of the players to start interacting is the "inciting incident".
What you're asking about is something slightly more nuanced, though, in that the inciting incident doesn't just get the plot going, but that it sets up the main characters to deserve their eventual fate, even if it's not being punished directly for their initial acts. As @Tetsujin mentions in his comment, I don't think there's a specific term for this kind of event, but you could describe it as "karma" (or "poetic justice") stemming from the inciting incident.
The movie Cabin in the Woods, which is a "genre-saavy" horror movie with a lot of meta references in it, uses the same word you did, "transgression", to describe the action that the victims took that "allows" the system to punish them, so that seems as good a word as any:
They have to make the choice of their own free will. Otherwise, system doesn't work. [...] They have to choose what happens in the cellar. yeah, we rig the game as much as we have to but in the end, if they don't transgress they can't be punished.