This is because Jesse is a lot smarter than the viewers give him credit for.
Initially, Jesse only helped Walt because of Walt's speech about Gus poisoning Brock with the ricin in order to manipulate him into not only giving permission to kill Walt, but to do it himself.
Jesse, realizing these pieces fit believes him and helps Walt get the information he needs to finish Gus off. However at the end of Face Off it is revealed that Brock was actually poisoned by a flower called Lily of the Valley, later revealed to be in Walt's backyard, thus implicating Walt in the poisoning.
It has yet to be revealed how exactly Brock was poisoned, but nevertheless the facts remain that Walt orchestrated the ricin cigarette to be lifted from Jesse in order to help prove this idea, along with planting the fake in the Roomba to convince him that he had nothing to do with the poisoning in the first place.
Skip ahead to Confessions Jesse pieces together that Huell must have lifted the ricin cigarette from him just like his weed this time. From here he connects that Saul must have had something to do with the poisoning considering this is his perceived second time having something lifted by Huell.
He then goes to Saul's office, literally beats the truth out of him that it was all orchestrated by Walt. This sends Jesse into a fit of rage, ready to burn down the White residence at the end of the episode.
The whole connection goes back to Jesse realizing that the only person that wanted Gus dead at that point, was Walt. There is only one person he knew of that was that vindictive and manipulative enough to accomplish this task and instead of just asking for Jesse's help in eliminating him, he went to the trouble of not only poisoning a child, something very against Jesse's ideals, but a child that Jesse was close to to begin with.
While Walt may not be the one directly responsible for Brock's poisoning, Jesse believes that no one else could have orchestrated such an act whether or not it was actually Kuby, or Huell, or Saul, or a random meth head that did it, it all points back to Walt's planning and idea.