We have seen that Tyrion has asked for trial by combat two times. How realistic is it? Was it a common practice during medieval times? Is it inspired from any medieval time custom? It does not look logical if someone like the Mountain, Barristan or Oberyn Martell can do any crime and can easily ask for trial by combat.
The other answers are quite good and on-the-mark, but I want to bring the spotlight on a specific point:
Yes, trial-by-combat existed in our world. But was not a case of "might makes right". If your guilt could be proven, you could not claim trial-by-combat: This was reserved for cases with a severe accusation, but no proof whatsoever.
So what are we looking at, here?
What was the purpose of real-world trial by combat?
The accuser says the accused is a murderer, rapist, traitor. A person dangerous to others, who needs to be stopped from doing further damage. But he has no proof. By taking his accusation to court, and winning in trial-by-combat, he might be able to take the accused down, removing the danger / getting vengeance. Without hard evidence, that's probably the best the accuser can hope for.
The accused, in turn, cannot prove his innocence either -- his reputation is severly damaged by the accusation. Demanding trial by combat is basically a counter-claim of libel and slander. The accused can "prove" his innocence, and (hopefully) strike down the person who slandered him. By the right to demand the accuser-without-proof facing you in combat, the probability of libelious accusations is reduced.
In either case, by putting the case to court and going through the proper proceedings of trial by combat, instead of just getting in a shouting match, then draw swords and go for it, the victorious party ensures that it will not be accused of assault by supporters of the opposing party afterwards.
The ability to nominate a champion is, again, the very opposite of "might makes right". The idea is that those who were not fit to fight -- due to age, disablity etc. -- would be "protected" from wrongful accusations, e.g. from someone who wants to use trial-by-combat to get rid of a rival. I am not sure if a person who would be capable of fighting for himself would actually be allowed to nominate a champion. (But then, a person of enough power would probably just have the champion make the accusations in his stead. I didn't claim it's a perfect system.)
So, all in all, trial-by-combat is not even that bad an idea, given the limitations of the judical system of the time. It's certainly not a "get out of jail free" card for someone like the Mountain if there's evidence against him.
It's not only inspired, but almost identical to a Western European medieval custom.
Trial by combat (also wager of battle, trial by battle or judicial duel) was a method of Germanic law to settle accusations in the absence of witnesses or a confession in which two parties in dispute fought in single combat; the winner of the fight was proclaimed to be right. In essence, it was a judicially sanctioned duel. It remained in use throughout the European Middle Ages, gradually disappearing in the course of the 16th century.
Unlike trial by ordeal in general, which is known to many cultures worldwide, the trial by combat is known primarily from the customs of the Germanic peoples. It was in use among the ancient Burgundians, Ripuarian Franks, Alamans, Lombards, and Swedes.
The custom had its roots in German tribal law, but it was also included in various legal codes at the time. What is important in regard to your question is that many of these codes strictly defined when a trial by combat could take place.
For example in modern-day Germany:
Hans Talhoffer in his 1459 Thott codex names seven offences that in the absence of witnesses were considered grave enough to warrant a judicial duel, viz. murder, treason, heresy, desertion of one's lord, "imprisonment" (possibly in the sense of abduction), perjury/fraud and rape.
and in Great Britain:
The wager of battle was not always available to the defendant in an appeal of murder. If the defendant were taken in the mainour (that is, in the act of committing his crime), if he attempted to escape from prison, or if there was such strong evidence of guilt that there could be no effective denial, the defendant could not challenge. Similarly, if the plaintiff was a woman, above 60 years of age, a minor, or if he were lame or blind, he could decline the challenge, and the case would be determined by a jury. Peers of the realm, priests, and citizens of the City of London (the last pursuant to their guarantee of ancient liberties under Magna Carta) could also decline the battle if challenged.
This is probably the main difference with the GoT world as apparently in Westeros a trial by combat can always be demanded. This indeed seems to be less realistic, but obviously strongly increases dramaturgy.
Other, older movies and stories.
Our depiction of the medieval era is skewed, beginning with the names we use for it (medieval era or dark ages). In the great heyday of history in the late 19th and early 20th century a lot of the research wasn't very accurate, and not only by today's standards. A lot of wrong reconstructions were made, and exaggerations and basically adventure books filled whole libraries.
This picture is from 1888. People didn't want to finance Columbus' travels not because they feared that they would fall of the brim of the world. They knew that his circumference calculations were incorrect.
When something looks like a real nice knight's castle...
...it probably was built in the late 19th century, 1867 in this case. The real things were not as impressive.
This isn't a new development. The Arturian sagas were exaggerated romantic knight tales from the beginning. They were a novelist hype throughout Europe... in the 12 century. People at that time criticised them for presenting a romantic past that never existed. Tintagel castle was built in the 13th century...
...probably as a themed villa for a fan of the stories. It wasn't suitable as a fortress and is located in a quite absurd location.
Game of Thrones (and the books even more so!) present us a fantasy medieval setting that satisfies our expectations. It is very successful in this, and real fun. But it is full of anachronisms, exaggerations, and repetitions of older misrepresentations would it be applied to our history (which is invalid as it is fantasy).
Of course the misrepresentations have a true core, as @Tautologist already mentioned. But please read the source that he linked to. Most of the time trial by combat wasn't just not common, it was even forbidden or only used in very specific circumstances. Depicting it as a "right" that anyone can demand in any trail is as wrong as e.g. depictions of the droit de seigneur. The medieval era was not a time of dirt, injustice, arbitrariness, and raw violence. Solving issues by sword was not common - and certainly no one could commit crime and fight himself out of it. Of course Game of Thrones basically takes place in a setting off high houses where wounded pride and slights are reason enough for centuries of war and fighting between the houses. In this setting of imaginary honour and false pride, of patrilinear family lines of thousands (!) of years, it doesn't seem too unlikely that this instrument could have much more importance than it had in reality.
While it all sounds very unfair the principal behind it was something called judicium dei. Medieval people were very superstitious/religious and believed God decided the outcome of battles. This meant that when someone won a battle, they hadn't just seized physical control, but proved that God agreed with their goals.
They used this belief as a way to 'ask God's opinion' when there was no other way to settle the matter. In trial by combat they believed that God would only allow a fair outcome and would protect the innocent even if it meant performing a miracle. This is the same principal behind trial by ordeal, in which an accused person would have to do something dangerous or painful and God would protect them from harm if they were innocent.
The accused would often hold these beliefs as well and so people who knew they were guilty would simply refuse the challenge rather than risk death and injury without the protection of God.
Like the other answers said, there are actual historical versions of trial by combat and trial by combat could indeed limit bloodshed when the alternative was a blood feud or pitched battle.
It does not look logical if someone like the Mountain, Barristan or Oberyn Martell can do any crime and can easily ask for trial by combat.
Modern justice systems favor the rich and the smart/eloquent, trial by combat favored the strong/well-trained. One isn't inherently more fair than the other in that regard. Of course over time trial by combat came to be seen more as trial by ordeal and then people were allowed to use champions (similar to what happened on Game of Thrones) so that rich people who weren't strong or well-trained could still obtain an advantage.