In his early scenes with her there's clear banter between them, she's initially nervous and overly-familiar, but he likes it:
David: Hello, Natalie.
Natalie: Hello, David. I mean, sir. Shit, I can't believe I've just said that. And now I've gone and said 'shit'. Twice. I'm so sorry, sir.
David: You could've said 'fuck' and
we'd have been in real trouble.
Natalie: Thank you, sir. I had a premonition I was gonna fuck up on my first day. Oh, piss it!
[David smiles at her as she leaves]
After failing to engage her in conversation the first time they're alone together, he not-so-subtly tries to learn some more about her later:
David: Natalie. Erm, l'm starting to feel... uncomfortable about us working so closely every day and me knowing so little about you, it seems elitist and wrong.
They talk about themselves and get to know each other a bit better before he asks his question:
David: Erm, and you live with your husband? Boyfriend? Three illegitimate but charming children?
Natalie: No. I've just split up with my boyfriend, so I'm back with my mum and dad for a while. He said I was getting fat.
David: I beg your pardon?
Natalie: He said no one'd fancy a girl with thighs the size of tree trunks. Not a nice guy, actually, in the end.
David: No. You know, erm... being Prime Minister, I could just have him murdered.
Natalie: Thank you, sir. I'll think about it.
David: Do. The SAS are absolutely charming. Ruthless, trained killers are just a phone call away.
[Natalie leaves]
David: Oh, God. Did you have this kind of problem?
[Turning to portrait of Margaret Thatcher]
David: Yeah, course you did, you saucy minx.
So he's clearly very into her. Then the next time he goes to see the US President, he opens the door and what does he see...

President: Er, Natalie, [smirking] I hope to see much more of you as our countries work toward a better future.
Natalie: Thank you, sir.
Then before either the President or Natalie have time to explain, David makes the speech and asks his secretary to have her transferred out of Downing Street. So really it was just a misunderstanding, David thought she was a willing participant rather than just the sexually harassed secretary as he later realised.