Since answers started to show up I've decided to summarize my comments into one.
In the books Renly's age is known, he is 21. Based on the fact that the actor Gethin Anthony was 27 when the series begun we can assume that in the series he's older (in his late twenties).
Regarding the manhood questioning part, it seems that indeed there was some.
We can hear it directly from Robert in the only scene he had with Renly:
Robert Baratheon: You ever f*ck a Riverlands girl?
Renly Baratheon: Once. I think.
Robert Baratheon: You think? I think you'd remember. Back in our day,
you weren't a real man until you'd f*cked one girl from each of the
Seven Kingdoms and the Riverlands.
(source)
Renly also complains about it to Loras in this scene:
My brother thinks that anyone who hasn't been to war isn't a man. He
treats me as if I'm a spoiled child.
(source)
Unlike Robert, Renly (in the series) doesn't like fighting, doesn't take part in the tournaments and sees war as a tragedy. He is homosexual, which is a kind of an open secret and Robert surely knew about it.
When Robert was questioning his manhood he was referring to the above facts, not his young age.