There are some theories that Hyman Roth told the would-be assassins to say that as part of a larger plan to turn Pentangeli against the Corleones, hence why Tom Hagen says that "Roth played this one beautifully", but this raises more questions about how much of what went down (the random cop walking by, and the shootout that follows) could have been planned in advance.
One thing we do know is that Danny Aiello simply ad libbed the line and Coppola kept it in the film.
Personally, I think he did it to taunt Pentangeli by making him think he'd been duped and that Hyman Roth was able to turn a failure into an almost success, but that's just me.