Two points. The first is that "cutting off your balls" is a standing order, for any threat, not just Tyler/Jack.
You know the drill. You said if anyone ever tries to interfere with Project Mayhem, even you, we got to get his balls.
They do this to a police commissioner earlier in the movie as well.
The second is that Tyler is an expert at reading people. He knows how people will react, what their limit is. He manipulates, breaks down and brainwashes/indoctrinates people the entire movie, Jack included. That coupled with him BEING Jack and KNOWNING that He and Jack are the same person, he (thinks) he knows exactly how far Jack is willing to go and what will stop him. He knows that Jack will not go that far, or that he can escape it and will back down. He's obviously wrong at the end, but his argument with Jack in the skyscraper near the end shows this is what he believes. That Jack's a coward.
Hey, you created me! I didn't create some loser alter-ego to make myself feel better. Take some responsibility!