In Season 2, Episode 8, Jaime says that there are only three men in Westeros who stood a chance of beating him in a fair fight. Who is this referring to?
Since Jaime himself has yet to reveal who those three are, we simply don't know.
In the books Jaime idolized Ser Barristan "The Bold" Selmy (the Lord Commander of the Kingsguard at the beginning of the series) and has always spoke in admiration of his swordsmanship, so it's safe to assume that he's on the list. Other than that it's pure speculation. Among the living the Clegane brothers (Sandor the Hound and Gregor the Mountain) are often described as powerful warriors even by Jaime, so they might be the other two. There are many others, but they are either dead (Ser Arthur "Sword of the Morning" Dayne), not included in the series (Ser Garlan Tyrell), or haven't appeared yet (Oberyn "The Red Viper" Martell).
Ned Stark was holding his own against Jaime. That lead a Lannister minion to attack Ned and thus embarrass Jaime because it seemed that he needed help to defeat the older man.
Brienne was actually trying not to hurt Jaime in their fight. Jaime blamed his performance on being rusty and shackled. That is certainly true, but it seems that Jaime brags (used to) a bit much about his swordsmanship.
I would like to add that Oberyn probably wasn't one of the three. I don't think him and Jaime were well acquainted, and Martell did spend quite a number of years in the Free Cities, so I wager that most people outside of Dorne wouldn't have been aware of his fighting skill.
Also, I'm not sure how it was shown in the books, but in the tv series, when Oberyn fights the Mountain, we unquestionably see how impressed Jaime is (the camera went to his face about three times during the fight, each time him looking marvelled). If he'd already known of Oberyn's skill, I doubt he would have looked as impressed as he did.
So the three people were probably Selmy, the Hound and the Mountain.
I don't think I can name all three but we all know that one of them is Barristan Selmy. The only thing that puts that to doubt is that at the time, Ser Barristan wasn't actually in Westeros.
Despite his wimpiness Loras Tyrell is a marvelous warrior. When Catelyn Stark is trying to find the owner of the dragon bone dagger, Littlefinger says that he lost it in a bet with Tyrion when Loras Tyrell unseated Jaime during the tourney on Joffrey's name day. On top of that Jaime also watches Ser Loras unseat the Mountain, so Jaime may have thought him able to best him. Ned Stark wasn't one of them because Jaime says "in Westeros" and at the time Ned Stark was dead.
As for the third, I can only guess, the only possibility that comes to mind is Beric Dondarrion, but I can't think of any evidence to support it.