Tyrion has sacrificed a lot, including loyalty to his family, to be Daenerys' advisor and hand. He believes in her and loves her as a queen and agrees with her vision of the new world she wants to build.
He, along with Varys, has tasked himself primarily with keeping her potential madness in check (indeed, this is something Daenerys explicitly tasked Varys with) - this is highlighted in many episodes but is made explicit in a scene with Cersei in "The Dragon and the Wolf" (Game of Thrones, S07E07):
CERSEI: Eventually, you want everyone to bend the knee to her.
TYRION: Yes.
CERSEI: Why?
TYRION: (PAUSE) Because I think she will make the world a better place.
CERSEI: You said she'd destroy King's Landing
TYRION: She knows herself. She chose an advisor who would check her worst impulses instead of feeding them
—"The Dragon and the Wolf" (Game of Thrones, S07E07)
Tyrion strongly believes in Daenerys' "better place", but also has expressed deep concern (sometimes even hints of regret) about the things she does (when he watched Drogon burn the Lannister troops, or when he walked through the field of ash, or when he tried to stop Daenerys from burning the Tarlys).
He already recognised this problem with the Jon-Daenerys romance at Dragonstone, in "Beyond the Wall" (Game of Thrones, S07E06), when he saw Daenerys risk everything (her own and her dragons' life, along with any hope for a new world) at the first mention of Jon Snow being in danger.
Seeing Jon Snow entering her quarters (and correctly inferring that their romantic/sexual tension had come to fruition) struck him with deep concern because now he knows his own influence to keep Daenerys' worst impulses in check has been compromised, and moreover Jon will be able to influence her, or have her risk her life again whenever he is next in danger.
Out-of-universe it is supposed to echo the concern going through the audience's mind, as we (watching this scene narrated by Bran) are experiencing the dramatic irony of this sex scene, and are having our own reservations about an incestual sex scene between an aunt and nephew, and are having our own thoughts and concerns about the political and romantic strife this will cause later on, when they find out. Tyrion's expression of concern serves narratively, but also out-of-universe to echo our own concern.
Update: From the mouth of the actor who plays Tyrion, Peter Dinklage
In Game of Thrones: Cast Commentary on A Union of Fire and Ice (HBO), Peter Dinklage, the actor who plays Tyrion, comments on this very scene and gives some insight into why he reacts the way he does:
It's dangerous for everybody involved. I'm sure it's good for both of them in the moment, but ...
You don't even get the relief of how beautiful it could be or should be, it's ..."No! It's not good!". But it should be, "but it's not!".
It's Game of Thrones - there's a long history of romance not ending well on this show.
—Peter Dinklage, Game of Thrones: Cast Commentary on A Union of Fire and Ice (HBO)