A major plot point in early seasons of Game Of Thrones was the idea that when a member of the Baratheon family had a child, that child always had thick black or dark brown hair, regardless of the hair colour of the other parent.
So the comment would make extra sense if you, your brother, at least one parent and any children, nieces or nephews have dark hair, while your other parent, wife and/or other parents of nieces or nephews do not.
It might also just be the resemblance - especially if one of you is large and wild and the other svelt and fashionable, while both dark haired and bearded. Here are two typical-looking Baratheons, Robert and Renly:
There's one other possibility worth mentioning (or, warning about...) - if you look like the young man above at the rear (Renly), and your brother looks like this stern Baratheon below... watch your back, and beware of shadows.
The night is dark and full of terrors.