TLDR:
- Conan Stevens left to star in The Hobbit.
- Ian Whyte moved on to other roles within the show and the rumours are he was only ever an emergency replacement for Stevens, not the long term solution. Another potential reason was the desire for a more physically imposing character, given Clegane's heightened importance in Season 4.
Answer:
Conan Stevens was originally cast in Season 1 (but had very little screen time). He then took on the role as Bolg in The Hobbit trilogy:
With the amount of time his filming entailed, he left the role and was replaced by Ian Whyte.
Now, I'm trying to find something close to "official confirmation" as to why Whyte departed, but the rumours appear to be that he was never intended to be a "long term" replacement for Stevens. He was only a very last minute role-filler. He didn't appear often in Season 2 and the entire character didn't appear in Season 3.
This explanation, although unconfirmed, fits with the roles he was given on the show in Season 3 onwards, when he began to play first Dongo and then Wun Weg Wun Dar Wun:
This change left the role needing to be recast a second time. Stevens by this time was actually interested in returning to the role. He discussed this in an interview:
MJ Snow – As you know, the rumors are flying regarding whether or not
you’ll reprise your role as Ser Gregor Clegane in Season 4 of Game of
Thrones, however in a recent blog post, you mentioned that you’d not
been contacted by HBO about returning to the series. Is there still
hope? Would you still be open to donning the armor again, if the
opportunity should arise?
Conan Stevens – I’m not sure where Game of Thrones production are at
the moment but I am pretty sure if there had been a plan of that sort
that I would have been informed quite some time ago, so the answer
would be, that barring some unforeseen occurrence, I won’t be back for
Season 04.
As to me being open… let’s just say the upcoming scene, if it is
indeed portrayed in the TV series, was the scene that I was most
looking forward to when I originally read the books. It is the best
acting scene for Gregor in the books to date, the most dialogue, and a
strong chance to show who Gregor really is, and by that stage Gregor
really has a reputation to live up to. As an actor it would be a huge
challenge living up to the books portrayal of scene and the well read
fans expectations.
As I mentioned in the recent article on my website, I already had an
idea how I would go about doing this scene so it’ll be interesting to
see how another actor with a different background approaches it.
But by this stage it seems the creators had already settled on Hafthor Julius Bjornsson to take on the role.
Another possible reason explaining the departure of Whyte was the upcoming elevated role of Clegane in Season 4 and the desire to make him as awe-inspiring as possible for his fight with Oberyn.
After all, would you rather fight THIS:
OR THIS?!
Edit:
Final point - in relation to the IMDB trivia part of your question, I think you may have misread it. The only mention of height is in comparing him with Kristian Nairn, the actor who plays Hodor. At 6 foot 9, Björnsson is indeed shorter than Nairn, who is 6ft 10.5. I think you are indeed correct. Using the wiki:
- Stevens: 7' 1/4
- Whyte: 7' 1
- Bjornsson: 6' 9
So he is indeed the shortest!