It's only difficult to not look into the camera for you because you're not used to being around them. If someone's taking a photo or video of you, you're "trained" to look at the camera because it's generally a portrait-style shot.
Actors are trained to not look at the camera. Looking at the camera is called "spiking the camera" and it means that you've just ruined a shot and you have to start over... congrats, you just cost the company money. As the TV Tropes article points out, editors don't always catch these, so you may see them in films and TV:
A specific form of Breaking the Fourth Wall accomplished merely by looking directly at the camera. Usually done by accident, by amateur actors who happen to notice a camera is pointed at them. The trouble here is that the editor didn't catch it, allowing the character to make eye contact with the audience. Typically a film or live TV trope, especially in dramas.
Trust me, even for new extras, you've had "don't look at the camera" so drummed into your head from the second you set foot on set, it's running non-stop through your mind that it's the only thing you're thinking about.
On set, directors will give actors a specific eye-line for where to look. Usually they're looking at another actor... the person they're doing the scene with. This is beneficial for performance reasons. If the person they're acting opposite isn't available due to scheduling or they're in makeup, or something else, they often have someone else there to deliver lines and act as eye line.
Sometimes they're looking at a thing off in the distance... building, animal, ocean... whatever. Sometimes they're looking at a piece of colorful tape that's stuck to part of the camera or other piece of equipment (this is generally used when the talent is almost looking at camera)... And, in CG films, they're probably looking at a colorful tennis ball on a stick being controlled by someone off camera.

And, finally... and probably most importantly... if you're good at acting, you're so involved in what you're doing that everything other than the person/people you're acting opposite simply fade away into the background. If you're doing it right, the cameras, director, lights... everything else... it just goes away and it's just you and that other person having a conversation or fighting a monster or watching TV...