I know that not all science fiction is scary and not all horror has unrealistic monsters or super powerful psycho killers, but at what point do these genres cross and what criteria separates a scifi film from a horror film?
Gorchestopher H did a decent job of explaining how they differ. I think how they differ is fairly obvious. If you want a good example of them combined, I would recommend the movie Event Horizon.
Star Trek is sci-fi because it is science fiction, but it is not scary.
The Ring is horror because it is scary, but it does not contain sci-fi.
Event Horizon contains aspects of both, so I would call it a sci-fi horror.
The answer lies not so much in the content as in the intended impact on the audience. Horror is intended specifically to instill fear. Science fiction has always been about stimulating the imagination of the audience, using science-oriented speculation to explore the ramifications of ideas, usually in a way that provides some commentary on society. Fantasy has been similar to science fiction, but uses magic-oriented speculation instead (with a heavy emphasis on good versus evil). The book publishing business got this wrong decades ago when they intermingled these three, probably because they didn't want so many categories of books.