Some of them knew him personally
Some of the men certainly interacted with Walter and Jesse. This included the manager of the laundry who saw Walter and Jesse enter and leave the basement regularly, and who could have identified them.
This is the manager who was persuaded by Steve Gomez to allow a search of the premises when it was suspected by Hank of being the location of the meth lab, and nothing was found.
Here he is talking to Steve Gomez

Here he is in prison talking to Mike, who was pretending to be a paralegal accompanying his lawyer:

Why kill all the men?
Walt is managing risks here. Yes, he could kill just those that he directly interacted with. However:
Walt can't know for sure who could identify him or not. All the DEA need are a name, a face or some identifying characteristics of Walt. Mike used to observe and tail Walt frequently because he didn't trust him. He probably used his crew to do some of this. All it might need is a comment that Mike used to regularly take cellphone calls from someone Mike called 'Walter'.
Even if someone cannot directly identify him, who knows what kinds of evidentiary trails can be started from them. If the DEA are trying to build a case against Walt in the future, who knows what kinds of evidence would start from clues these men give away when cutting deals with the government.
From Walt's point of view, better to avoid the risk entirely. From his perspective he has to protect himself and his family, and it's only money he has to spend.
Finally, it becomes clear that Heisenberg's reputation is enhanced by the act of killing all these men in a simultaneous action like this. It's not clear whether Walt predicted this, but perhaps he also considered how it might make him feared.