As we all know, going near a black hole (Gargantua in this case) slows down time, according to the theory of relativity.
So when Cooper comes out of the black hole, after spending a fair bit of time there, shouldn't he be much older than Brand who hasn't gone inside Gargantua?
There is indeed a little age difference between the two, but not a significant one.
First of all, let's look if there was an age difference between Cooper and Amelia in the movie. The movie itself highly suggests that there wasn't much of a significant age difference between the two. When Cooper, Amelia and the two robots make their gravitational slingshot around Gargantua and right before Coop and TARS throw themselves into it, he says "That little trip cost us 51 years", to which Amelia jokingly answers "you sound good for someone approaching 120". So at this point, where both's times didn't yet deviate, Cooper is already pretty near to his supposed 124 (as measured by earth standards) at the end, as also detailed in the answers to this related question. So the age difference he does acquire by his trip into Gargantua is not much, only a few years maybe.
And this also fits to the depiction of Amelia Brand's whereabouts at the end when Murph says something like "Brand...She's setting up her camp...Perhaps she currently prepares herself for the long sleep", if we take into account that it took Brand some time, probably at least a few months if not years, to reach Edmunds' planet, and then more time to set up the camp and send a signal home (as discussed in this related question).
Now to the question if there should be a larger age difference between the two -- and thus also between Cooper and everyone else, especially Murph -- due to his stay inside Gargantua. You seem to assume that he resided inside the black hole and its gravitational influence for the whole of his communication with Murph. But as the answers to this question and this question discuss, Cooper hasn't been inside the black hole all the time. He plunged into Gargantua only to be effectively saved from there by the "Bulk Beings" shortly after, who put him into the Tesseract and transported him out of the black hole by employing an additional spatial dimension. So he was only subject to Gargantua's huge time dilation for a short moment. But I'm also too less of a physicist to say if that bit of time dilation he experienced should still have been larger.
But to gain some further insights on the matter, we can take a look into the book The Science of Interstellar by physicist Kip Thorne, who advised and accompannied the movie from its earliest conception onward. As already explained in this answer, Cooper was basically trapped between two different singularities, an "outflying" one from stuff falling in much earlier that him and scattered back to him and an "infalling" one from stuff falling in long after him.
When I explained the two singularities to Chris, he immediately knew which one should hit the Ranger. The outflying singularity. Why? Because Chris had already adopted, for Interstellar, a variant of the laws of physics that prevents physical objects from ever traveling backward in time. The infalling simgularity is produced by stuff that falls into Gargantua long after Cooper falls in [...] If Cooper is hit by the singularity and survives, the universe's far future will be in his past. He will be so far in our future that, even with the help of bulk beings, he won't be able to return to the solar system until billions of years after he left, if ever. That would prevent him from ever reuniting with his daughter Murph. So Chris firmly chose Cooper to be hit by the outflying singularity, not the infalling one...
Chris's choice, though, presents a bit of a problem for my scientist's interpretation of the movie. [...] If the Ranger falls directly into Gargantua from the critical orbit, then its infall is slow enough that the infalling singularity will catch up to it and hit it. For the Ranger to hit the outflying singularity instead, as Chris wants, the Ranger must nearly outrun the infalling singularity, which is descending at the speed of light. The Ranger can do so, if it is given a large, inward kick. How? The usual: by a slingshot around a suitable intermediate-mass black hole soon after leaving the Endurance.
At the singularity's edge the tesseract awaits Cooper -- placed there presumably, by bulk beings.
So practically as shown in the movie Cooper should indeed have just fallen into the black hole and become part of the infalling singularity and thus lived through millions upto billions of earth years in those short moments. But if we assume that he plunged into Gargantua with a significant speed boost, he basically outruns the infalling singularity (at least for a sufficient time), reaches the outflying singularity in a few moments and is then right after that saved from the black hole and its time-dilating influence by the Tesseract, presumably fast enough to not cause his time to deviate by more than a few years from Brand's.
The problem is the closer he gets to the even horizon, the greater the relativistic effects. If the first planet has 7 years to 1 hour at a stable distance, and the slingshot is 51 years to a few minutes, then at the horizon the effects are likely to be Eons to the few moments. Fraction of seconds to Coop and TARS will be thousands and millions of years back on Earth. There is no escaping it.
Unless of course, They, in their 5th dimensional existence and powers/technology, account for relativity. Given their ability to transverse time like a physical dimension, they would just need to drop Coop and TARS anywhere close to relative time, which they clearly did. They would only need to ensure that the time frame Coop was dropped into would NOT negatively affect the time stream to prevent a paradox. Note that TARS seems to have experienced more relative time than Coop. He's old, scorched, batteries dead, while coop only experienced hours at most. This tells me he was dropped into time earlier than Coop was by maybe years, or more likely, he experienced more time in the black hole than coop did, relatively speaking.
Between the time of traveling through the worm hole and to the time of entering black hole cooper spend about 4 or 5 years.
Inside the Black Hole the time dilation is infinite. SO the time Cooper spend in the black hole is Negligible. Then he travels through the bulk (Hyperspace) and meets Amelia Brand (she was traveling through the bulk too.). SO he has only aged for 4 or 5 years since traveling through the worm hole with Amelia. So he looks the same.
