There was no interaction between Chris and Joe save for when Chris went to get Joe to share his Brandy, which Joe did, in the deleted cut. What would make Chris hate Joe and call him Ugly during the interrogation scene? It was funny as hell but came out of character. Chris really had no reason to hate Joe prior to finding out he poisoned the coffee.
I think an important part of the story is the setting (i.e. post-Civil War America). Chris Mannix is the new sheriff. People were tired of fighting, Mannix was from the losing side and may still harbor some resentment to the country as a whole. Today, we might talk about PTSD after such a period of unrest.
As a man of law, it is normal to be suspicious of strangers or drifters as this was still around the Wild West times and Joe Gage presented himself as a mysterious cowboy.
This is a different type of movie but in Rambo, the sheriff also has no reason to hate Sylvester Stallone other than the fact that he is a stranger but the same principle applies.
Unlike Rambo, Sheriff Mannix' suspicions were actually spot on as it turns out.