Holt was better detective than Wuntch. He did realize that Wuntch was thinking she'll have a chance with him.
That's why he had to drink to build up courage to come out to her because he knew she ignored any "hints". And by hints I mean Holt talking about his sexuality. Which, if you know Holt, was saying "I couldn't be happier with my hay partner" with a stone face. And you know that even Jake can't read that man. And Jake is so good detective he made Holt "dad proud".
So, Holt was openly gay. Holt realized that Wuntch disregard that information for her own benefit ("Boooooneeeeee" to quote Rosa). Holt coming out wasn't "coming out" per se. He told Wuntch he won't have sex with AND made her realize she was a bad detective by not realising it on her own. And to me it also add their enmity. Holt hate that detective that is so bad is so high in the ranks. Wuntch hate Holt because he's the reminder of how bad she is.