Butch Pooch (James Remar) doesn't take his hat off the entire film. Especially in the setting Tarantino places the film in, the character must have a reason; there's nothing absurdist about the film; it still deals within the bounds of reality.
So when Jamie Foxx's character says "You don't wear a hat in the house white man, even I know that." it's not like Butch doesn't know that, right?
He's wearing it in order to be imposing, isn't he? I mean, why wouldn't Candie make him take it off if not because it helped him do what M. Candie hired him to do?