There was no spell. It was actually the venom of Swooping Evil which only erases the bad memories. Since Credence's attack created bad memories for everyone especially for no-maj, that venom wiped it out from everyone's mind affected by that rain.
There is no such thing that it'd only affect muggles. It can affect wizards as well as pointed out in this answer and this one. Queenie uses an umbrella to come out in the rain and the rest stay under the roof of that subway.
The reason why it didn't work on Jacob Kowalski, because the venom doesn't really obliviate everything except bad memories and he didn't have any as he said in Fantastic Beasts: the Crimes of Grindelwald.