There was a question about Night Kings' purpose and one of the answers there contained information you might find helpful. It said something like:
The White Walkers were created centuries ago before the long night by the Children of the Forest as a weapon to fight back the first men as can be seen in the TV show.
After the White Walkers were created both men and Children of the Forest had to fight them. They both made an alliance and were able to defeat the White Walkers (the Azor Ahai legend is from this period of time). After this, they built the wall and for some time the Children of the Forest even provided the Night Watch with Dragon Glass in the case of the Walkers came back.
From this it was pointed that the magic that created the White Walkers didn't work as planned so they had to unite and fight them.