Mance Rayder in intending to move the entire Wilding population south to avoid the oncoming winter, which is presumed by many to be likely to be long, and accompanied by the reappearance of 'the others'. He has an army estimated as 100,000 strong.
The remaining other forts along the wall are deserted yes, but its also presumed that the tunnels through the wall at these points has been filled with stone & ice. So yes, Mance could get people to climb over the wall here (as indeed we saw last season), but he cannot move vast quantities of people over the wall, which would ultimately have to include children and the elderly of his population. Even getting 100,000 over the wall seems unlikely - we must presume he only sends his most capable raiders on that route.
For Mance's invasion of the north to work, he either needs to tunnel through the wall or capture one of the occupied forts. He knows that the force of Crows is historically light in numbers, even if he doesn't know their true weakness - and that his army will also have to face the army of the Warden of the North at least, if not the Seven Kingdoms. Attacking and getting his army past Castle Black is the key first step in his invasion.