It's a little slippery, but...
Both Cooper and Brand lost time during the slingshot maneuver, putting them both X number of years forward, to where Murph was an old woman. Then when Cooper entered the tesseract, he was free to maneuver through time and see Murph both when she was young and when she was on the verge of solving the gravity problem. Then, perhaps because it was the last point at which Cooper was outside of the tesseract, he got spat out maybe a year or two after the slingshot maneuver ended. This could be treated as an intelligent choice by the tesseract's controller, as the colony ship was on the verge of entering the wormhole, and Brand was on the verge of establishing the colony.
EDIT: The conceit I'm working on is that the "entrance" to the Tesseract encompassed most of the mass of the black hole, meaning that the amount of time between jettisoning Brand and entering the tesseract didn't cause an immense or insurmountable amount of time loss.