I'll try to answer this question, although spoilers will be rife.
Who does Varys serve? Remember back to his conversation with Ned Stark in Season 1:
Eddard Stark: "Tell me something, Varys. Who do you truly serve?"
Varys: "The realm, my lord. Someone must."
Varys may or may not be lying. We don't know. Up until now though, we know that he is playing a game, like Littlefinger and all the others. We also know that he knows much more about Daenerys than he lets on.
Arya Stark, again in Season 1, overheard him talking underneath the Red Keep to Illyrio Mopatis, the man who was looking after Dany and Viserys. At the time, their conversation is confusing, but as the series' continues it becomes more and more obvious what they were discussing. From the wiki:
...Illyrio pays a secret visit to King's Landing where
he discusses their mutual plans with his ally, Varys, in the secret
passages under the Red Keep. Once Khal Drogo's son is born, the
Dothraki can move to invade Westeros and Viserys can retake the Iron
Throne. However, Westeros must be in chaos and civil war before this
happens. When Varys warns that a civil war, launched by a conflict
between House Stark and House Lannister, is imminent, Illyrio becomes
concerned that things are moving too fast, but Varys suggests
motivating the Dothraki to move faster might be a viable strategy.
They talk about how new Hand of the King, Eddard Stark, now has the
same genealogy book that his predecessor read, and has met Robert's
bastard son Gendry: the same evidence which led Jon Arryn to discover
the truth about Queen Cersei's children. They discuss the possibility
that the same fate may befall Lord Stark as did his predecessor.
This is incredibly important and was revealed very early on in the series. Whilst Robert wants Dany killed and Varys agrees quietly, he has no intention of doing this. Spoilers:
He is revealed in the books to actually warn Jorah Mormont that this is the King's intention and to be careful. The poisoner in Season 1 acted on his own accord, not on Varys'. See this question for more information.
Now, in the books all of this isn't revealed until Book 5 - the series is very premature in showing us all of this, revealing it all by midway through Season 1.
However, since it does reveal it all, we know that Varys' has plans for Daenerys and is in league with Ilyrio to protect her (for what end and reason, we don't know beyond his supposed desire to serve the realm).
Now, all of this is spoilerific and describes books 4 and 5 (and thus likely Season 5):
In the books, Cersei effectively rules as Tommen is too young. She is paranoid, incompetent and surrounds herself with allies who give her poor advice. Jaime grows distant from her. In the books, there is a Lannister, Tyrek, who disappears during the riot of King's Landing shown in Season 2. Jaime begins to suspect Varys had a part in this. There is also some speculation that Varys had some part in a puppet show that begins to be shown in the capital, discussing how a kingdom is ruled by arrogant, greedy lions. A stag stands up to them, but he is eaten. Finally, a dragon hatches and burns all the lions. Cersei is outraged. She puts to death the creators of the show, takes half the wealth from anyone wealthy who watched it and takes an eye from those who don't have wealth.
Following these minor mentions of Varys, in Book 5:
Kevan Lannister effectively takes control of the kingdom after Cersei is ruined. However, Varys enters Pycelle's chambers and kills him, before finally using his little birds to have Kevan Lannister summoned and killing him. He basically tells him this is because he is doing too good a job reversing all of Cersei's incompetent and moronic actions. Her actions are sowing rife and discontent between the Lannister, the Tyrells and leaving the realm open for the taking. There is another heir to the throne (besides Dany - more spoilers!) and he intends to make the realm ripe for the taking.
So after ALL THAT, including many spoilers:
His leaving the realm is consistent with his overall plan. When he sees Tyrion leaving, he knows it is a good time to leave to. His overall plan is truly underway and he knows Cersei, now mad with power and rage, will continue to ruin the realm. With Stannis far away, Renly gone, the Starks gone and the Tyrells and Lannisters fighting, it's the perfect time for his long-in-waiting plan to finally be put into full swing.