In Shame, Sissy tells Brandon
We're not bad people. We just come from a bad place.
I assumed that this bad place may have meant they were abused as children by their parents, which led to Brandon's sex addiction and overall behavior, and Sissy's not wanting to return home ever again.
Is there any asserted basis for my assumption? Do we know what bad place they came from?