According to the movie script and Wikipedia article, the team was
ordered to execute a counter-insurgent mission to capture Shah.
However, they encountered communication problems.
Upon arriving at their designated location, the SEALs are
accidentally discovered by an elderly shepherd and two teenage goat
herders. Knowing that if they release them, the herders will likely
alert Taliban to their presence, the team is split about whether to
execute the herders or not.
Then, the most important thing was to find a spot where they can communicate with their headquarter and safely return as their mission couldn't continue because they were discovered by three Afghans.
The goats sound like they are howling. The vibe on the mountain is becoming deathly serious.
If Taliban soldiers or sympathizers had heard the goat howling, they would have noticed something was wrong. Therefore, the best decision was to move away from them as soon as possible. They couldn't take the prisoners up to the extraction point as it would take more time and goats were howling.
They had only three options; (1) kill them and run away, (2) let them go and run away and (3) tie them (in the movie, they worry about wolves eating them if they tie them up) and run away. Since goats were howling, No. (1) and (3) options would not have made such a big difference. They had to choose No. 2 and got into trouble.
The script shows Luttrell saying:
Luttrell: I don’t know. I don’t like it. We kill them. Yea ok. We kill them. Bury them. They get found. Then what...
Axe: Then what?
Luttrell: I’m just saying. It ain’t gonna be private. Gonna be out there for the
whole fucking world. CNN -- SEALs kill goat farmers. I don’t want that
legacy, bro. I’m not killing goat farmers. Not feeling that.
If we kill these kids, it’s international news. CNN doesn’t care about Rules of Engagement. Seals kill kids. That’s the story. Forever.