Congratulations! It looks like you spotted a clue towards a plot twist originally planned to be revealed in Season 3, that was ultimately dropped.
Pycelle was originally imagined by the showrunners (possibly by author GRRM too, it's unconfirmed as far as I know) as a meek careerist toady, someone who maintains his comfortable high-status position by being useful and loyal to those with power and by avoiding letting himself be seen as a threat.
Part of that, as they saw it while writing at least the first 3 seasons, was that he deliberately acted more physically feeble than he really was, in a conscious attempt to appear harmless. They filmed a (later deleted) scene as part of filming Season 3, where Tywin calls Pycelle out on his act.
As soon as he realises he's been found out, Pycelle stands up straight, drops the stammer from his voice, and admits (rather awkwardly...) that his mannerisms were a phony ploy. It's an extreme transformation - he looks like a different man, suddenly almost as healthy and sturdy as Barristan.
This scene never made the cut. I don't think they explicitly state why, but given that in Season 6...
...he doesn't particularly stand up straight and fight even when being murdered to death by children.
Maybe he lasts a little longer than you might expect for someone so seemingly frail, but he's quickly overwhelmed.
...it appears they changed their minds, or at least decided to tone down the contrast between how weak and frail he acts compared to how robust he really is.
So it appears they'd anticipated this "big reveal" throughout the writing and filming of the first three seasons, peppering in various clues (some YouTube commenters also mention scenes in Season 2 after he's challenged by Tyrion, where he struggles vigorously, or briefly drops the stammer). Then, after filming this scene and reviewing it, for whatever reason, they changed their minds and dropped it.
I'm not aware of any interviews where the writers discuss why this was dropped. Maybe they didn't like how the transformation looked compared to how they'd envisioned it; or maybe they felt that, compared to all the major "this changes everything!" moments in Season 3, revealing the character we all knew as a slimy, duplicitous toady to actually be a phony slimy, duplicitous toady wasn't quite the "wow" moment they'd hoped it would be.