Is there a category for films like 'Shutter Island' where the story is turned upside down at the end of the movie, and then you have a "WTF" feeling?
I'd classify them as films with a twist ending, and Shutter Island in particular to have what's called an "unreliable narrator". See this wikipedia page for more classifications of plot twists with a few examples of each.
I won't compile a list here--Google should be able to--but keep in mind that watching a movie with the knowledge that a plot twist is coming might diminish the experience for you :)
This film, and the novel it's based on by Dennis Lehane would be classified as a Psychological Thriller with a touch of psychological horror.
It's in line with films like, Psycho, Vertigo, The Shining, Gone Girl, Hannibal Lecter Franchise, Memento, The Prestige, or The Sixth Sense. All of these films rely on a character driven story that then has major plot twists, suspense, and mystery with the identity of characters or the nature of their reality being questioned.
The film takes place in 1954 in a remote island off the coast of Massachusetts where is located Ashecliffe hospital for mentally disturbed and violent criminals. Two detectives, Teddy Daniels (Leonardo DiCaprio) and his new partner Chuck Aule (Mark Ruffalo) are summoned to the island to investigate the disappearance of a patient named Rachel Solando (Emily Mortimer) who is considered to be very dangerous. This will start a chain of events in which nothing is as it seems at first glance. Doctors, medical staff and guards are not willing to cooperate, which only deepens suspicions of Sheriff Daniels whom will further investigation lead to the terrifying discoveries. In an effort to solve the mysteries of the hospital, Teddy through flashbacks returns to his personal tragedy, the death of beloved wife Dolores (Michelle Williams) that occurred during a fire. With each new day, Teddy is getting more secure that maybe would never leave this island…
Shutter Island requires complete attention and makes that viewers be maximally involved in the story to along with Teddy reveal the mystery of the island. This brilliant psychological thriller is directed by Martin Scorsese and screenplay is based on same name novel by Dennis Lehane. https://www.psychologicalthrillers.net/movie/shutter-island/
This movie is classified as neo-noir genre by wikipedia. Neo-noir movies are defined as
Movies in which characters were often conflicted antiheroes, trapped in a difficult situation and making choices out of desperation or nihilistic moral systems.
This clearly suites the movie's plot.
Now spoiler alert (in reverse order)
Edward "Teddy" Daniels (aka: Andrew Laeddis) was incarcerated in the mental asylum aka Shutter island for murdering his maniac depressive wife-this fact is disclosed only at the end of the movie. Going back in the movie to the beginning,we find the leading protagonist as a impeccable U.S. Marshal Edward"Teddy" Daniels who was appointed to investigate the disappearance of the patient Rachel Solando. Over the course of investigation,Teddy finds out that the mental asylum members(Dr.Cawley Et al.) has tricked and trapped him to be the 67th patient for carrying out their fearsome lobotomizing experiments.And from here, Teddy becomes very much confused and paranoid(so do the viewers) and does many chaotic activities out of sheer desperation like breaking into the restricted Ward C, igniting Dr.Cawley's car etc., All these bizzare events were only resolved at the end like a whodunnit play where the real culprit's identity is disclosed at the end. All these elements in the plot makes it very aptly to call the movie as a Neo-noir movie .