In the movie The Rock Mason said he timed the flame bursts to work his way through, however, how was he able to do this without being caught?
I highly doubt that he observed the flames just once, he probably visited the boiler discretely to determine that there is a pattern, and lucky that the flames followed this pattern rather than burst randomly.
When the prison was functional, I doubt they would leave the door unguarded. Logically, they would've had a human guard or a camera.
This IMDb says:
When Mason is with the troops in the basement of Alcatraz, he states that he studied the timing of the boiler that he must roll through. The question has often been asked why he wouldn't then just be able to open the door during his original escape. However, when Alcatraz was functional, the door may well have been guarded or locked from the other side.
The door that's supposedly guarded is also next to the boiler with all the flames. The guard would literally have to be asleep to miss that Mason has gone into the boiler. If you can make it into the boiler, why not steal a key and open the door rather than risk being killed by fire?
I found this post that somewhat answers why he didn't go through the door the first time but it doesn't answer how he observed and timed the flames without being caught, or why he didn't steal a key, to prevent the alarm from going off, to escape.