It's a consequence of the mutations he underwent during the additional experiments performed upon him as part of his training/conversion into a Witcher:
I underwent the usual mutation there, through the Trial of Grasses, and then hormones, herbs, viral infections. And then through them all again. And again, to the bitter end. Apparently, I took the changes unusually well; I was only ill briefly. I was considered to be an exceptionally resilient brat…and was chosen for more complicated experiments as a result. They were worse. Much worse. But, as you see, I survived. The only one to live out of all those chosen for further trials. My hair's been white ever since. Total loss of pigmentation. A side effect, as they say. A trifle.
This loss of pigmentation is presumably the reason why his skin is also pale.
The reason he is the only Witcher like this is that no other child has survived these additional experiments.