The following is admittedly much speculation on my part, but I think Ethan's motivation at this point was not entirely clear and this is the way (or the possible ways) I interpreted this scene.
First of all he didn't really completely exorcise the demon from her, but merely supressed it again temporarily, the same way he supresses the werewolf that's inside of him. And in fact Vanessa still has that demon inside of her after this event, just less prominent and again hidden deep inside of her as before. This is also the reason why she distances herself from Dorian Gray, since it was the passionate sex with him that allowed the demon to fully break through and this is likely to happen again if she further keeps meeting with him.
So how did he suddenly know how to do this? In fact Ethan is quite similar to Vanessa in the way that he carries a dark force inside of him, too. It is hinted previously throughout the series with various little incidents that Ethan is himself a werewolf and at the end of the season's finale it becomes as evident as possible. This is also why it wasn't alluded to previously that Ethan has such knowledge, together with Ethan's overall background it was deliberately kept a mystery to the others and the audience. This scene was thus on the one hand made as yet another surprising allusion to his mysterious ways when you were actually expecting the only possible outcome of Ethan holding the possessed and death-wishing Vanessa at gunpoint. And when one sees Ethan's nature as a werewolf as a kind of possession of his body/soul by an external force, it is quite similar to Vanessa's possession. And that amulett Ethan used to "exorcise" Vanessa was most likely used by him exactly for protecting himself to some degree against his own animalistic side.
So why didn't he consider to do this before? There are multiple possibilities, I think:
- On the one hand he might not have wanted the others to know about his dark secret and that he is acquainted with all this stuff.
- On the other hand it might very well be that with this ritual he somehow transferred the amulett's protection from himself to her, sacrificing his own protection against his inner wolf for her well-being.
- Or he didn't consider this a viable alternative, since it merely supresses the demon without really freeing her from it, still leaving her a disrupted soul. He might not have wanted such a further existence for her, seeing his own torn identity.
And before this incident there was still some hope for her, I think. But it was at that moment (and after the priest refused to do an exorcism) that the only alternative to salvage her was death. He might even have considered this alternative for himself before, but was either not strong enough to really follow it through or chose to embrace the werewolf as part of his own identity. But when holding Vanessa at gunpoint he was just unable to kill her and chose an alternative that he might not have been entirely sure to be the better solution on the long run.