I don't recall this being explicitly addressed in the show, beyond the point you already mentioned about "men seeing what they want to see." However, I suspect that Moira had control over how she appeared, and she chose to appear as the old lady version so that the wife would accept her working there.
Also, keep in mind that the older version of Moira is not "real." When Moira was killed, she was her younger self - she never aged to be the old woman version in life, so that image of her as an old woman is either partly or entirely made up. At best it's what Moira thinks she would've looked like as an old lady.
Go back and watch the scene where she ingratiates herself into the house with the wife by rambling about how she's always worked there and "comes with the house." Imagine how that scene would've likely played out if she had appeared to the wife in her younger form. Especially given that Ben was a known adulterer and the whole point of moving into this house was to "work on their marriage."
She never would've gotten in the door. So she made up this old woman appearance to placate the wife and get into the house, so that she could play the role she needed to play.
When Ben sees her as the old lady, that was likely Moira playing mind games with him. IIRC, by that stage he was beginning to question what was happening there, and maybe even beginning to question his own sanity, so she was taking an opportunity to confuse him even further.
