If Tangled (2010) used live-action, what would have been the problem?
Would the effect be different if Tangled was acted by real actors/actresses?
How do film-makers decide which story will be produced as an animated film?
Animated films are just another form of storytelling, and I wouldn't say that executives or writers or anyone looks at a script that's been written and decides if they want to do it via live action with live actors, or animation of some kind. I believe that sort of thing is decided well in advance.
However, animation can allow for certain things that live action can find harder to achieve, such as suspension of disbelief.
A good example of this is the horse Maximus. His actions and mannerisms would have felt completely odd and out of place had they decided to go with live action. They would have needed a CG horse, which imposes some restrictions on how they can make the character move and act without seeming too unreal. However, with a fully animated and highly stylized film, they can make the horse do things that would seem really odd to see a real horse do: act as a pointing dog, express emotions with his face, just to name a few.
People are more willing to suspend their disbelief for animated films, especially ones that look more "cartoony", than they are for live action.
Tangled earned $559 million word wide from box office sales, not including merchandising profits. It cost $260 million to make. So doubling your investment isn't bad.
Red Riding Hood (2011) was a live action story based upon a fairy tale. It cost much less $42 million, but only did $89 million. That would be counted as a success, except to a major studio $47 million in profit wouldn't cover their laundry bill.
Animation is a stylized format that has a broader appeal then traditional film. So more people will watch it.
To answer specifically to your case of Tangled, Disney has a long tradition of blockbuster animated musical fairytales, and it's a business model that works.
Their most recent live-action exception, Enchanted (2007) did pretty well at the box office. It didn't perform as well as animated movies aimed at the same audience though, and was beat out by Shrek the Third and Ratatouille (and also Alvin and the Chipmunks, which kind of throws a wrench in my theory, but I'm forging ahead anyway). Tangled pulled in over 200 million more than Enchanted, which seems to show that animated movies, particularly animated princess movies, do better than live-action ones.
Merchandising is also always a major concern for Disney, and it is harder to make palatable merchandising out of live-action images. Dolls, t-shirts, etc., especially modeled for kids, look a lot better with images that were meant to be 2D. (that is opinion-based, not fact)
What is fact, though, is that live-action films cause major licensing problems for Disney.
This requires a detour into the history of the Disney Princess merchandise line. Since its introduction "Sales at Disney Consumer Products rose from $300 million in 2001 to $3 billion in 2006. Today there are over 25,000 products based on the franchise". In recent years, the line-up has been exploding to provide more products and revenue, and it's gone from 8 characters to 13 since 2009. That's a princess a year, a big jump from the earlier rate of a little less than a princess a decade (starting from 1937 with the production of Snow White). This shows a desire from Disney to increase revenue by introducing more Princess characters.
Live-action princesses are harder to include in this merchandising boom though. From an article on the phenomenon in the Wall Street Journal, Disney abandoned the plan to add Giselle to the princess line-up "when it realized securing the life-long rights to the image of Amy Adams, who plays Giselle, was harder than an ugly step-sister squeezing into Cinderella's glass shoe."
So, in sum, Disney chooses to animate fairy tale movies because it lets them make more money, both in the box office and over the long term.