Alfred told Bruce of his hope that he had when Bruce was gone for seven years. There's this specific cafe he went to once a year when he took vacation. He would go there and look around and hope that he would see Bruce there. Bruce would be happy and with a woman, maybe even with his own family. They would see each other and then go on with their separate lives. Why wouldn't Alfred the man that raised him and thought of him as a son want to be more than just a head nod greeting in a cafe, instead of somehow becoming a part of his life?
Alfred: [to Bruce] I had this fantasy, that I would look across the tables and I'd see you there, with a wife and maybe a couple of kids. You wouldn't say anything to me, nor me to you. But we'd both know that you'd made it, that you were happy.
IMO, what Alfred meant was that he did not expect Bruce to acknowledge the past. He just wanted him to be a happy family man with a new life, even if it meant Alfred wouldn't be a part of it. The statement had a sacrificial tone to it.
If we think from Alfred's view, he always wanted Bruce to live a peaceful life. So if he gets that chance anytime in the future, he must not intervene into Bruce's life as it might make Bruce remember about him being the Batman (Alfred was the one who was there from the beginning, when Bruce became the Batman). So if he leaves Bruce, it might help him forget about the accidents of his life (including the death of his father). Remember, Alfred expressed he had never wanted that Bruce returned to Gotham as it was "full of pain and tragedy" for him. So he might considered himself as an emblem of it to Bruce. In this way, maybe Alfred was trying to help Bruce in making a normal life with a fresh start.