The simple answer is that we all saw it in the show, therefore it is 100% possible for dragon fire to easily destroy walls in the reality that exists within the show.
If you insist on having a real-world physics explanation for how a magical creature in a fictional land can destroy walls, I would suggest that the primary mechanism was the conversion of water to steam.
As a port city, King's Landing would tend to have more moisture in the air and a large amount of water trapped inside and between the brickwork and stonework of the city's walls and larger buildings. The high heat of dragon fire would instantly convert that water to steam, developing a significant amount of sudden pressure that would blast the masonry apart.