Back to the future plays fast and loose with time travel rules.
Usually time travel is expected to play out using one of those three models:
- Stable timeline (Ex: You go to the past to kill Hitler as a baby, replace him with a random orphan baby to hide your tracks, come back to the present to see that the "evil" Hitler you always knew was actually the orphan, and it's your fault he got a family to raise him)
- Alternate timelines (Ex: You go to the past to kill Hitler as a baby, replace him with a random orphan baby to hide your tracks, and now there are two timelines existing at the same time. One has Evil Hitler, the holocaust, etc, while the other had completely different events)
- Unstable timeline (Ex: You go to the past to kill Hitler as a baby, replace him with a random orphan baby to hide your tracks, come back to a completely changed future. Since your grandfather fought in WWII, which happened a lot differently now, maybe he died or maybe he didn't meet your grandmother. Either way, you were never born, so never went back in time, and so a paradox destroys the universe)
Back to the future mainly uses the unstable timeline model. Marty seems to be able to change events very easily, with great risks to himself and everything else of course. On the other hand, some events seems to follow the stable timeline model, like Marty inspiring a "Chuck" by playing Johnny B Goode, giving the idea to become mayor to the employee at the bar, etc.
Considering that some events use a model, while others use another, there are two explanations for Doc's decision that night.
If we assume that Doc has successfully created a stable time loop when he read Marty's letter (he was obviously trying to, since he wore a hidden bulletproof jacket instead of just doing his demonstration an hour earlier or in another location), then he decided to fight the libyans the way he did in order to succesfully send Marty to the past, because he knew that already happened.
If we assume that both mall parking fights were different, and doc actually died the first time, then Doc's decision the first time can be explained by his own words. I watched the film in another language, so I have to translate aproximately what he said in English.
The libyans. I thought I had shaken them off my trail, but they found me.
This indicates that Doc certainly didn't plan for them to show up that day, and is surprised they did. Doc being the overly prepared guy he is, having this pistol with him was probably for generic self-defense. So a bunch of terrorist are after him, with much larger weapons than anything he is prepared to fight, and he knows they want him dead. Combining this information to the stress that he must have been experiencing from his first(?) successful use of the machine (He's dancing, so he's glad it worked), and he's probably scared of Marty getting caught in the crossfire.
That's a huge amount of stress to put on an old guy like him. He probably just didn't think about running away at the time. Furthemore, as we can see with Marty in all three movies, it seems easy to forget that time travel can solve most problems when Under considerable stress. Marty didn't even realize he was in the past before the next morning.
So I just assume that Doc reacted irationally from an extremely stressful and dangerous situation. In fight-or-flight mode, he decided to fight since he had a weapon. This was probably a bad choice, but every movie character makes bad decisions.