The mob bosses had sent Nicky to Vegas to basically represent them and control the situation in Vegas. Nicky, at first had kept the bosses satisfied, but eventually he got completely out of hand and had transformed himself into the new "Godfather of Vegas". He got so out of control that the bosses couldn't handle him and Nicky just had to go. Thats how the Mafia works, you go a little out of hand and you're out.
Now, everyone in the Mafia, The Bosses, The Capos, The Soldiers, and of course the associates too have to follow the Omerta. Following is a list of possible reasons why Frankie killed Nicky and his brother Dominic so ruthlessly:
He was just following orders and did what had to be done. In short, just fulfilling the necessities and following the bosses orders,
Secondly, assuming he had a "personal problem" with Nicky and he knew that Nicky had it coming in the end, Therefore he held it all in and took his "revenge" by simply following the bosses orders.
Now, even if he did not have any problem with Nicky, you must have noticed that Nicky had an "understanding attitude" towards Frankie when he was beating out Dominic in front of him. He understood it was all the bosses's orders and he had to go out along with his brother and of course being made and example of.
So at first he was furious but eventually he understood and let it all go. Frankie, on the other hand, If he did not follow the orders he'd be in danger himself, Hence he just did what had to be done