In-universe, this is the sexual equivalent of the "You do NOT want to know" trope. The implication is that he's been doing something unspeakably kinky and weird to Marla (something which caused her to scream in ecstasy) and no further explanation is given. You're basically expected to leave it to your own dirty imagination to work it out.
Notably the incident doesn't appear in either the source book or the shooting script.
Out of universe, this interview with Director David Fincher strongly implies that they (he and Pitt) chose to put the glove into the scene specifically to weird-out the studio executives who'd stipulated that Pitt had to be shirtless twice in the film. Again, no specific sexual act was intimated:
“One of the studio’s issues with the material was how we were going to
handle the sex between Marla and Tyler,” says Fincher, whose trade-off
for the studio shelling out on Pitt’s dentistry [they paid for the
removable cap that ensured Tyler could have a chipped tooth] was that
the star would sometimes take his shirt off. He did it twice. One of
which was when he opens the door after sex with Marla—wearing a rubber
glove. “We had that take starred and sent over to Fox under the guise
of, ‘Look! Look how good he looks, he has his shirt off!’” laughs
Fincher. “I’ve learned that one way to control people is to give them
other things to worry about. If you’re worried that somebody is too
fearful you can either try to empower them or you can give them so
much to fear that they just don’t want to be around you. Either way
you’ve sort of neutralized them!”