So I understand Rudy wanted Dexter to feel the need to connect with him on their mutual urge to killing people in the altercation happening in the final episode of Dexter Season 1. But since Dexter was hesitant towards this, Rudy could have very well threatened Dexter by letting Debra know the true nature of Dexter instead of going one-on-one with him?
I don't have proof to back my theory, Since Rudy took the time to get to know Dexter more closely, precisely from his childhood days he probably learnt that harry had told him to keep it a secret which he respected.
OR(the better theory)
It might also be because she is a cop and his little brother might be put on death row if they figured out that he was a serial killer and rudy wanted to spend time with dex not get a means to his end
Rudy’s goal is to make Dexter join him to “reunite the family”. It’s the nature of this endeavour that it only works if Dexter voluntarily embraces this: If Dexter were forced into this by means of threat, there would be nothing keeping him from faking it and backstabbing Rudy on the first occasion. You cannot just force somebody to join the dark side. Also, if forcing were an option for Rudy, there would be no need to go through his elaborate plan to “awaken” Dexter and he could just have threatened him from the very beginning.
Of course, at some point Rudy may have accepted that his plan had failed and thus he would resort to saving his skin, but the earliest point from this is when Dexter prevents him from stabbing Debra and initiates the fight. This fight is so short and intense that there is no real room for making a threat. And this would only makes sense if Rudy noticed that Debra was awake – which is not certain.