In the book The Godfather I read that Vito Corleone became mafia because the society couldn't protect, that we have to act to protect ourselves.
But somehow I didn't get it. Why did he become mafia, why a cold blooded murderer? Why does he run an illegal business? He could have been successful in a legitimate business like Genco Oil. Why?
You answered your own question; "because society can't protect". You should probably take a look at the Wiki page for Vito Corleone, as it does a good job summarizing his entry into organized crime.
In short, Vito is somewhat thrust into the role originally due to the actions of Don Fanucci. Vito must rid the neighborhood of Fanucci, and in doing so he becomes well-respected in his neighborhood. This leads to additional income from offering protection, which is much more than he could make by just being an oil importer.
If you remember, before Vito Corleone, it was Don Fanucci who was ruling the neighbourhood. I think he jumped from a small burglar gang with his friends to organized crime when he didn't like how Fanucci exercized his power and took him out.
You can see in this video how Fanucci was too heavily into extortion and was responsible for Vito being fired. This might have induced personal resentment from Vito.
When he was in charge of the neighbourhood, he was more protective of the other people, hence more respected.
I suppose when you ask "Why cold-blooded murderer?", you are talking about Don Fanucci. This question explains it well. As Don Fanucci was not liked by the people, as he was putting too much pressure on them, and Vito supposed he was not as powerful as he seemed, he took his chance to take him out. He had probably in mind that he would be more careful to his neighbours.
It is the opposite. The Genco Olive Oil Company was created after he started illegal businesses to cover illegal incomes. You can see here, when Roberto the landlord comes to visit Don Vito to apologize and return the money, Vito's office being decorated as Genco Import Co. meaning that he was already established as the local godfather:
The Godfather wikia page says:
The Company was a front for the family's criminal activities.
Vito had no intention of becoming Mafiosi, he saw what it did to his family in Sicily. But then Fanucci has him fired so that Fanucci's nephew could have Vito's job. With Vito out of work he comes into contact with Clemenza who introduces him to crime. Vito sees how much money can be made in relative safety from prosecution and gets deeper and deeper involved. When Fanucci demands money or he'll send the cops after Vito and his friends, Vito realizes that Fanucci is a fraud, no real Mafiosi would threaten anyone with the police, it was then that Vito decided Fanucci needed to die. After killing Fanucci the entire neighborhood changed towards Vito and began treating him as an important person. Even Clemenza and Tessio changed towards him showing both fear and respect. Only Genco treated him the same which is why Genco became consigliere.
Vito became a Mafia crime lord because he had the skills and the talent for it, just as Michael did later.
You need to go back to Corleone. Vito family is murdered by a mafia Don because his father opposed him. He witness his mother begging the Don to spare his life and then murdered because she wanted to protect him.
In America he see similar pattern. Don Fanucci is larger danger then anything else. He don't protect his own people, he don't help them. And he fear that one day Fanucci could go after him and after his family. That's why Vito after killing him go to his kids and say
Michael, your father loves you very much
He then take the role of mafia Don as a protector and helper (the Signor Roberto scene).
An accumulation of things. We see at each point he resists, seems content to work in a grocery store (until don Fanucci's nephew displaces him), is by chance given a gun to hold by the young Clemenza and is worried plainly about how his family is endangered.
In the book, he kills Fanucci because Fanucci is demanding 700 dollars worth of what Vito helped steal (which, because of Don F. he was forced into) and he reasons, if Don F. needed an operation that cost 700 to save his life, he would not pay for it. So it is worth it to him to kill Fanucci, a plainly terrible man.
He then assumes Fanucci's place in the neighborhood but due to far greater talent and respect, he goes much further. He probably never planned on being a criminal per se but in a world where violence was used frequently, casually, he had no choice.
Perhaps the having had his father, mother and brother(? -- I forget) murdered, the die was cast long ago or perhaps he could have avoided it if circumstances had been different and he might have just become, for example, a grocer.
