TL; DR. The OP gives Vader too much credit, and Luke not enough.
There is an ongoing drama of Vader plotting against Palpatine from the get-go, but it seems that in order for Vader to survive, he must remain subservient to Palpatine unless it's highly likely that Palpatine can be overthrown.
I'm completely unaware of this ongoing drama that's being referred to. AFAIK, the closest Vader comes to saying anything remotely like this is when he and Luke fight on Cloud City in Empire Strikes Back:
VADER: You can destroy the Emperor, he has foreseen this; it is your destiny. Join me, and together we can rule the galaxy as father and son.
If Vader's telling the truth in that the Emperor foresaw it, and that it's apart of their plan, then this wouldn't necessarily be considered plotting against the Emperor (since the Emperor was the very one who came up with the plan).
During the lightsaber duel with Luke in the throne room in Return of the Jedi, Vader is basically a push over. It's clear from rewatching the scene that Vader is either a push over or has completely given up.
Remember that, between films V and VI there was a lapse of about a year (it's never officially been established, but this is the most widely-believed amount of time). During this time, Luke continued training in the ways of the Force, and got stronger both physically and spiritually. Vader, however, only got weaker, and more dependent upon his machinery.
Just before Vader took Luke to see the Emperor, while looking at Luke's crafted lightsaber Vader says,
VADER: You've constructed a new lightsaber. Your skills are complete. Indeed you are powerful as the Emperor has foreseen.
This supports the idea that Luke had continued his training and gotten stronger. Vader, on the other hand, had at that point become more accustom to hiring bounty hunters to do his "dirty work", as was shown in Episode V. And so, perhaps over the years Vader was getting less and less combat experience, given his poor health (end of Episode VI); or, when he does get into combat, he uses the Force to compensate for his saber (Episode V duel).
With this in mind, what may seem like Vader being a "pushover" could just be a result of Luke's significant increase in skill and strength since the last time they fought, making Vader's diminished skills that much more obvious.
(side note: Luke was ~24 y.o. and Vader was ~45 y.o. during the throne room duel.)
At the same time, Luke isn't exactly good either; he's fine against blaster-armed enemies, but he's really a complete novice in a lightsaber duel, whereas Vader has killed countless fully-trained Jedi masters during and after episode III.
First and foremost, Episode III happened over 20 years before Luke and Vader's showdown in the throne room. IMO, I don't think the events in Episode III can necessarily support any arguments regarding the throne room duel (within the context of the OP).
If we're to still consider Episode III though, the only Jedi [equivalent] that Anakin kills is count Dooku. Aside from that, Anakin doesn't kill any Jedi masters. Instead, all that's shown is him: (1) slicing off Mace Windu's arm; (2) slaughtering a bunch of padawans; and (3) killing Nute Gunray's group on Mustafar.
When Anakin fights Obi-wan though, he loses miserably.
Again, I'm not sure the events from Episode III should be considered, however, even if they are, there doesn't seem to be much evidence to support Darth Vader being as skilled of a swordsman as the OP suggests he is. Also, AFAIK, most all the Jedi were killed by clones during Order 66, and not Vader.
The final duel in the throne room seems to indicate that Vader wanted somehow to betray Palpatine, so perhaps this is why he went easy on Luke. However, if Luke agreed to join the dark side, would Vader and Luke really be successful where Vader would not have been?
After watching the throne room duel again, I'm not seeing any indication of Vader wanting to betray the Emperor, except until the Emperor begins attacking Luke.
Vader did though, to some degree, seem to not be as aggressive as he had been during their last encounter (i.e., he doesn't use the Force to push/throw, etc.), but he was still exercising Luke's abilities (example).
If there was any hesitation on Vader's end, it was a result of the conflict within Vader to not kill his own son, just as Luke mentions several times throughout the movie.
As for the last question, "if Luke agreed to join the dark side, would Vader and Luke really be successful where Vader would not have been?", I'm not sure what's being asked here, but if the OP can clarify then I'll attempt a response.