I could not find any definite answer, but I guess it should be "The Capitol" where the hunger games were conducted. The reasons being:
1) If the dictator would have run the game in the Districts in front of the people, it might have caused another rebellion.
2) From wikia
The arenas were also a popular tourist destination for many Capitol citizens, who would spend months vacation visiting the arena from their favorite games, where they would be able to rewatch the Games, visit the sites of the deaths and even take part in reenactments.
So the arenas were a popular destinations to the capitols people. So it is assumebale they were created in the Capitol itself.
3) It would be unbiased to set up the arena in capitol because every district were participating in this game. Setting up in a particular district may attract rumors of being slanted.
Each of the districts was named 1,2,...,12 to make them separate from each other. Also the numbering helped the Capitol to administer them efficiently. Each district was unaffected from others in culture, living etc. and it was illegal for them to interact with each other. There are also more unique features to every one of them. Here is a short list of those:
District 1- Luxury (Great comfort or living)
District 2- Masonry/Stonework/Peacekeepers(not said officially in books, but later revealed.)
District 3- Electronics
District 4- Fishing
District 5- Power (not said officially in books, but later revealed.)
District 6- Transportation (not said officially in books, but later revealed.)
District 7- Lumber (Type of wood)
District 8- Textiles (Manufacture of cloth or woven fabric)
District 9- Grain
District 10- Livestock (Farm animals kept for use or profit)
District 11- Agriculture (The science or practice of farming)
District 12- Coal Mining
District 13: (Was thought to be terminated) Graphite Mining (Unofficially they manufactured nuclear weapons and among that housed and produced other weaponry).
More of the districts could be found here.