Drogon may not always be and effective and precise tool of destruction. He is not a spear which hits the target precisely, but shoots jets of fire, dragon fire that they have no control over.
Daenerys's state of mind:
"but her plan seems to be working perfectly well until she herself decided to willingly kill the innocent bystanders?" this is what they were afraid of. All of her counsel were always in awe/reverence of their queen until recently when they feel/decide shes no longer fit to be the queen
Their concerns come from a place of insecurity. They've seen their queen spiralling into instability, losing her kids, the dragons, her close aides Missandei and Jorah, almost half her army. Taking absolutely rash decisions. Had Daenerys been in a stable state of mind like in season 7, they might just have been more assured of Dany's decisions.
Daenerys's troubled and angered state of mind did very little to assure that Dany might just be reasonable or controlled in her actions. And riding a dragon over red keep, they were not assured was a good idea, keeping all of this in mind.
They have been skeptical of the queen and her actions since this season began. Doubting her every move, and calculating her every action.
In earlier seasons it used to be reverence
S8E4 TYRION AND SANSA
TYRION: But why provoke her? How is that in the best interest of
your family, - or the North, or--? -
SANSA: You're afraid of her.
TYRION: Every good ruler needs to inspire a bit of fear.
S8E4 VARYS AND TYRION
VARYS: Do you think she wants to share the throne? She does not
like to have her authority questioned.
TYRION: Something she has in
common with every monarch who ever lived.
VARYS: I worry about her
state of mind.
TYRION: We are advisors to the queen. Worrying about
her state of mind is our job.

And when they actually say it's a bad idea :
S8E4 VARYS AND DAENEYRYS
Your Grace, I promised you I’d look you in the eye and speak directly
if I ever thought you were making a mistake. This is a mistake," he
says of her plan to Dracarys King's Landing. "Do not destroy the city
you came to save. Do not become what you have struggled to defeat."
DAENERYS: Do you believe we're here for a reason, Lord Varys? I'm here
to free the world from tyrants. That is my destiny and I will serve
it, no matter the cost.
But perhaps it's good the people see that Daenerys Stormborn made
every effort to avoid bloodshed, and Cersei Lannister refused. They
should know whom to blame when the sky falls down upon them.
Taking Chances:
Dany knows, we know that the dragon can be as precise as it can, but to Tyrion, Varys and the others/her counsel, Drogon is a dragon which is being ridden by a very angry and unreasonable queen at the moment.
A jet of fire ejected, could spread like a wildfire and destroy more than intended or the entire red keep and its people.
That jet of fire could destroy the palace, and the rubble could landslide down into the red keep and the people sheltering in the red keep.
This is the chance that Tyrion and Varys don't want to take while trying to establish a new queen (a queen they vouch for) in King's Landing. Minimum/No Casualties, makes a great new entry of the queen and her aides into the kingdom. They don't want to take a chance with the dragon and its fire. Varys and Tyrion being on the highest of Dany's ranks, know how important it is for their queen to not hurt civilians while taking the throne from Cersei. They want to avoid mass killing intended or unintended.