I saw this when I was very young, probably around 1983, and I haven't come anywhere near figuring out what it was. I saw it on TV, possibly on a premium movie channel like The Movie Channel or HBO. I'm almost sure it was a short film, and it revolved around a man finding a broken or fallen apart little wooden shack that looked basically like an outhouse on what I think was a beach.
There is one of those weighted punching clowns sitting in the middle of the fallen down walls. He puts the little building back together and then a woman comes out of it, and she runs off. This repeats a few times, with different women coming out of the little building dressed in totally different costumes and then running off, possibly with other men that come by. Finally the man enters the little building, and all the women return and reenter the little shack as well. Then the four walls fall apart and there is a new punching clown that looks like the man who had been watching the whole time.
I had originally thought the man was played by Woody Allen, but I've scoured his filmography and couldn't find anything that matched this memory. But my guess now is that the man was a nebbish, and Woody Allen was something I came up with later on.