You seem to think that once Nix discovered humanity's reaction to the idea of their demise (being that they amplified their "efforts" towards it), he should have tried to shut it down, like Frank did. But that's not what Nix intends, as clarified in that scene. When he says that line about shutting it down, that was just an act before Frank got onto his game. He never intended to shut it down.
He knows that the signal causes humanity to further their demise. But he blames the humans, not the monitor. In his opinion the signal only amplifies the humans' inherent urge for chaos and destruction. In contrast to Frank and Casey he doesn't believe in humanity.
We can even go further than that. His reaction to what the monitor caused was as much a self-fulfilling prophecy as the monitor's signal itself. I would postulate that Nix never had much respect for the rest of humanity to begin with and when he showed them the visions from the monitor it only confirmed his prejudice against them when they didn't understand its message. You could even say the signal served as an excuse for Nix to finally drop all concerns about humanity, since, well, they bring it upon themselves anyway, right? It's not like he didn't "try" to help them and show them the errors in their way, right?
So once he learned, in his opinion, that humanity is unsalvageable, there was no point for him to shut the signal down. If they want to destroy themselves, then just let them do it and get it over with. He stopped caring about the humans and the monitor "confirmed" his actions. For Nix the signal basically just gave him the power and the authority to make humanity fulfill what he thought to be its destiny anyway. He long lost his hope for humanity, if there ever was some is another question, though.
