This question has been raised by fans before and has been addressed by George R. R. Martin, author of the A Song of Ice and Fire novel series that the show is based on.
Is it possible to circumvent the Wall to the West with an army?
No, the mountains and a deep river gorge make the terrain impassable for all but small groups of raiders.
—George R. R. Martin, "Many Questions", So Spake Martin, DECEMBER 28, 1998
The Night King could conceivably march down to the Gorge with his army and trickle down small groups of wights, but this would be a waste of soldiers, since it would make it very easy to deal with a bottlenecked army that approaches in scattered, small groups at a time.
According to A Storm of Swords, Brandon the Builder had the Wall built wherever was possible, so this gap in the west is probably due to the impassable terrain making it practically unfeasible to build there, and it was probably decided that an army would have trouble getting past.
Brandon the Builder had laid his huge foundation blocks along the heights wherever feasible, and hereabouts the hills rose wild and rugged.
He had once heard his uncle Benjen say that the Wall was a sword east of Castle Black, but a snake to the west. It was true.
—George R. R. Martin, A Storm of Swords, Chapter 30, Jon
