I'll preface this with the fact that I have not read the books, and if I am wrong on any of this feel free to correct me. Possible answers from what I gathered from the episode:
A very small amount of time has passed between when Bran was touched and when the walkers arrived. Hodor and Meera are seen preparing their things to leave when the walkers arrive, so they probably assumed they had more time, and wanted to get as much done at the tree as possible before leaving.
Bran's green sight into the past can be sustained after leaving the tree, but can only be started by touching the tree. Every sight into the past Bran has prior to episode 5 begins with him holding the tree and ends with him being taken away by the Raven, and letting go of the tree. I think due to the fact that they knew the Night King was coming, the Raven was able to help Bran sustain his sight farther than usual using his last bit of power.
The cave is still the safest place for them to be. Although I am not entirely sure how the mark that the Night King placed on Bran works, he may be able to find Bran anywhere. Because the cave still has resistance to the resurrected zombie type walkers it is safer than being out in the wilderness without the protection of the cave and the Children of the Forest
Now if the mark on Bran CAN be used to find him anywhere, it seems that leaving the cave will do little to save them. Because the Night King can then simply walk out of the cave and follow Bran, killing him shortly after the final scene of episode 5. For this reason, and the fact that the Raven says, "He knows you're here," and not something like "He can track you now," my third answer is most likely (and hopefully) wrong.