There are few reasons why there are so few giants shown on-screen. (There's more than one -- at least three, but still not many):
- Giants don't like people. Ygritte calls them "shy", but it seems more that they avoid human contact. The fact that Mance managed to get any of them at all to come help him was a pretty big deal.
- They're nearly extinct. I don't know if there was anything specifically mentioned in the show, but in the novels the total population of Giants is in the low hundreds at most. So seeing one at all is very rare. (Some smart people in the show do claim to believe the giants are extinct, but they said the same thing about dragons and Children of the Forest, so...)
Of the three giants Mance recruited, two of them died in the assault on Castle Black, so the one that you keep seeing everywhere is just the only one left alive after the wildlings move through the Wall.