It's practically a complete work of fiction. While some details are derived from certain fact - IP Man did teach wing chun, but not in a school. He was a police officer for a while - precisely what and how of his day to day life would be something akin to a game where you tell a simple story and pass it to each new generation. By the time even a few generations pass, you have very little remnants of the truth left.
And in a tale like IP Man, as we have seen in these 4 installments, it sort of has to be. Truth be told, the life of a martial arts instructor is not very interesting. Back then, they were revered as master scholars, like how a PhD bearing professor might be today. I'm sure on the odd occasion you had challenges, but for the most part that didn't and does not happen today. Not because of laws or what, but because generally speaking people don't like being punched and kicked and even if you know what you're doing, walking into a martial arts studio like they do in these movies looking to fight the whole school would be an act of suicide.
I don't have references for you. But you can bet the IP Man series is about as factual as the Hatfields and McCoys is on Netflix. Yup, there was a feud. Probably everything else in between was made up to be entertaining.
Unnecessary extra info: My family is from Canton, China back when it was called Canton. We've all been through the phases where we thought Bruce Lee was cool, martial arts, history, this and that. I recall reading about Bruce's life as a kid and his success in Hollywood. He wasn't even particularly great. He was just famous. Also, he was fast and that certainly helped his accolades, but in the grand scheme of things Bruce Lee was more marketable than he was talented. IP Man was just a martial arts instructor in his pastime and his family origins were one of fair wealth compared to most people. But if you want to make a few solidly kick ass movies, you kind of have to make up a few things to make history sound a lot less boring.