In Season 2, Episode 3, a project called Tin-Man was activated by a rogue named Raymond Gordon, the main purpose of which was to steal a suitcase that belonged to the Observers.
During his interrogation, he said:
We don't know who they are, but I can tell you what they want, they want to exterminate us.
In fact, he was right, and I think he was the first one to notice the Observers and understand their goal.
So the question is, how did he know all of that?