Actors are usually hired if they fit the role they are supposed to play.
In case of GOT, some of the characters can be visualized with existing actors. As you can notice, none of the star or lead characters (Jon Snow, Arya Stark, Daenerys etc.) are known faces.
Senior actors are roped in because they either fit the role or lend some good acting in the scenes. Sometimes, when the character is important, mature actors are often cast as they have a better knowledge to face the challenges required in the role.
In case of GOT, the number of characters and roles is phenomenal. The costs alone for more known actors would have increased the budget several times. Also, the actors that play the character are definitely good in their roles.
Having new faces in the series also makes it more immersing for the fans as they can visualize the characters with the actors. Most fans might not recognize the previous works these actors have done, and hence recognize them only by their GOT persona.
Hence, the producers and directors many a times go for newer faces and fresher talent.