It was probably done with the rear projection technique.
I could not find any materials discussing this effect in Modern Times. Most of the easily-available material on effects for this movie cover the roller skating scene. That said, this effect was most likely done using the rear projection technique, in which a projector behind the screen projects the image through it to create the impression of a video screen image. This technique was most often used for the background of driving scenes, but it is also perfectly suited for this application.
At the time this film was made (1936), this technique would avoid the problems you raise in your question. According to the wikipedia discussion of the technique's history, by the 1930s, they had developed cameras and projectors that could be linked to ensure identical frame rates. They also had better film and more powerful bulbs that made rear projections look brighter and more defined. By projecting from the rear--with the projector likely inside an enclosed compartment behind the videophone screen--they would also avoid illuminating anything in the foreground with the projector.
Again, I cannot find a source confirming this was done with rear projection, but the technique (used widely in 1933's King Kong) was available in 1936 and seems to be the best explanation for the clarity of the videophone images in Modern Times (shown in the first few seconds of this clip).