A.bakker has a great answer, but misses an important point. The Dread Pirate Roberts is a pirate, and one with a very specific reputation.
Fezzik and Inigo are obvious underlings and just following orders, yet they do show honor.
Vizzini, on the other hand, takes pride in being a snake. He has no problems showing his bad side (as that might be the only side he has, but we don't know enough about him to really know this). Also, when Roberts meets Vizzini, he has a knife to Buttercup. He also treats her roughly even after they agree to a match of wits, which also shows that he has no honor and that he really only cares about himself. This is also evident by Vizzini leaving his men to try to slow down Roberts. I also seem to remember some snide remark about the other two losing, when Roberts arrives, showing his lack of empathy. (It's been too long since I watched the movie or read the book.)
Then there's the fact that Vizzini has the kind of general braggadocios attitude that makes even the most gentle of people want to punch him. With all of this evidence, it doesn't take much for someone who is known as a ruthless pirate to think, "The world would be better without this guy. And he's bumping up against my ego, anyway."
And what better way to show a guy like Vizzini up than by setting him up for something he simply can't win. As for making it lethal, there's nothing to say an SOB like Vizzini won't go back on his word and still go after Buttercup again. Killing Vizzini just guarantees no more problems.
Edit
After watching the movie again last night, Vizzini is the one that suggests it's lethal, as others have mentioned. This is after he brags that Roberts can't match his intellect, so of course Roberts thinks differently and suggests a match of wits, trying to get Vizzini's ego involved in his own downfall.