We're not supposed to know, and that's the beauty of it.
You have to know everything, don't you? (Kramer to Jerry, The Parking Space)
There's no explanation provided in the DVD extra dedicated to this episode, but the episode's writers (Tom Gammill & Max Pross) say it was based on a similar incident from Jerry's life, where a woman refused to taste his pie and it drove him mad. They thought it was a funny thing to be angry about.
And Jerry's frustration is exactly what's funny. The show could easily supply a mundane explanation, but chooses not to, not only so we could share his frustration, but also because it's beside the point. Instead, through Jerry's incident with Poppie and George's botched job interview, it cleverly shows us that there could easily be an explanation for Audrey's behavior, but Jerry will never get that because he's incapable of putting himself in someone else's shoes, and blindly prefers to scrutinize others.
In fact, Jerry's so blind to this, that it's easy to come up with a simple solution to this enigma just by drawing on his own experiences. Like, maybe Audrey saw someone sneeze (a la the pasta primavera) or cough (a la the babka) on his slice, was embarrassed to tell him but later decided to risk it anyway. Maybe (again, like the babka) it had a small hair or bug on it. Or, even more plausibly, maybe she just didn't want a bite out of someone's half eaten pie - note that she refuses right after Jerry double-dips his fork into it. A germophobe like Jerry should've understood her reluctance and drop the whole thing.
Jerry, however, is more prone to obsess over a mystery until it drives him insane and often costs him a relationship. After all, this isn't the first or last time this formula was used. He relentlessly tried to discover why Newman was turned off by his date, why everyone thinks his great girlfriend is a loser and why the woman that he's seeing keeps wearing the same dress - and these mysteries were never solved in those episodes either, because a big part of the series (right up to the very finale) is about the pitfalls of neurotic obsessions, where trying to know the social unknown never ends well.
Jerry: Well, I think we've proven who the psycho is.
Elaine: (sarcastically) We certainly have.