Bran knows something..
"I imagine you've seen things most men wouldn't believe, to go through all of that and make your way home again only to find such chaos in the world, I can hardly imagine - "
"Chaos is a ladder"
This is a shock to Littlefinger because Littlefinger himself had said in season 3, something Bran would have no way of knowing until he became the Three-Eyed Raven. In episode six of the the third season, when Littlefinger was serving Joffrey , he said this to Lord Varys:
"The realm. Do you know what the realm is? It's the thousand blades of Aegon's enemies, a story we agree to tell each other over and over, until we forget that it's a lie."
"But what do we have left, once we abandon the lie? Chaos? A gaping pit waiting to swallow us all."
"Chaos isn't a pit, Chaos is a ladder. Many who try to climb it fail and never get to try again. The fall breaks them. And some, are given a chance to climb. They refuse, they cling to the realm or the gods or love. Illusions. Only the ladder is real. The climb is all there is."
Additional point mentioned by @Flater in a comment
It seems relevant to mention that Petyr's quotes of "Chaos is a ladder" and "This dagger is Tyrion's" are separate events that took place in different places. Bran asked for confirmation whether Petyr knows whose dagger it is. But instead of stating that he knows what Petyr said to Cat about the dagger, he chooses a more subtle reference. To Bran's mind, he wanted to convey to Petyr that Bran is aware of everything (not just the dagger). Whether Petyr understands Bran's magical omniscience (compared to Bran having heard the "chaos is a ladder" quote from Varys) is unclear.