I think you can watch this as a stand alone movie. Apart from a few references to the Harry Potter world in the movie (name drops of Albus, Hogwarts - total of around 10 small references through out the entire movie) etc., rest of it is completely new. Its set in the 1920's era. Before Harry Potter started.
So no, you don't have to watch the Harry Potter movies to understand this one.
It's a new experience even for HP fans (you would know the beasts they show in the movie if you read Fantastic Beasts library book before)
Spoilers below as requested.
Hogwarts keeps coming up - also a my-school-is-obviously-the-greater chat comes up with the main protagonists ---
Gellert Grindelwald comes up. You can even see the back of his head right in the start ---
Dumbledore's name comes up. One character asks Newt why Dumbledore favors Newt.---
Nifflers, Bowtruckles, Occamys, Diricrawls, Demiguise etc., beasts from 'fantastic beasts and where to find them' book from Hogwarts Library appear in the movie ---
Seeker, Chaser reference comes up..
and few more references which I can't exactly recollect.