TL;DR : Catelyn slapped Roose because she realized that they were about to be betrayed, and that Roose was a part of it. She knew this because he was wearing chainmail at the wedding.
One of the most sacred traditions in Westeros is that of "Guest Right", which is a sacred law protecting both hosts AND guests from hostility while staying at another's castle or home. Under this law, even the most hated of enemies dare not attack each other. The law works equally for host and guest, and is based on the honor system - anyone who breaks the right is almost guaranteed to be executed themselves by other Houses, since no one can trust a House that would break the right.
Catelyn was already suspicious of the invitation, given the circumstances. Her son Robb had blatantly disregarded his earlier pact with Walder Frey, and yet Frey still extended an invitation to attend his daughter's wedding. She reminded Robb of this very fact in episode S03E07 "The Bear and the Maiden Fair", yet Robb accepted since he believed that Frey would never break the law of Guest Right, especially over one "little" broken oath.
In the scene you describe, three things happen that tell Catelyn what's about to happen:
- The band starts playing a song that is practically the Lannister anthem
- Walder Frey's men start closing and locking the doors of the banquet hall
- Catelyn notices both of these things, and then sees that Roose Bolton is wearing chainmail - something he wouldn't do at a wedding unless he was expecting combat
At that moment, Catelyn realizes that they are about to be betrayed, and it's obvious that Roose Bolton is part of it. With little else she can do, she slaps Bolton out of anger and then tries to warn Robb. Of course, it's all too late by that point.
Note: Also, it should be noted that - even as crazy as he was - Walder Frey could only be convinced to break Guest Right once he had the assurances of Tywin Lannister that nothing bad would happen to he or his family afterwards. He and Roose Bolton discuss this after the massacre. Of course, though, as we see later in S06E10... Tywin Lannister didn't count on
a vengeful Faceless-trained Arya Stark returning from the dead.