I know that not all science fiction is scary and not all horror has unrealistic monsters or super powerful psycho killers, but at what point do these genres cross and what criteria separates a scifi film from a horror film?
Gorchestopher H did a decent job of explaining how they differ. I think how they differ is fairly obvious. If you want a good example of them combined, I would recommend the movie Event Horizon.
Star Trek is sci-fi because it is science fiction, but it is not scary.
The Ring is horror because it is scary, but it does not contain sci-fi.
Event Horizon contains aspects of both, so I would call it a sci-fi horror.