It obviously wasn't the British Royal Family as her first stop on her tour of Europe was London. All I recall from the movie is that it was one of Europe's oldest royal families.
The wikipedia article says:
Ann (Hepburn) is the crown princess of an unspecified country
Princess Ann visits the UK, the Netherlands and Greece (Greece still had a monarchy back then). Those three countries are therefore not possible. As of 1953 these countries remain possible options: Norway, Denmark, Sweden, Belgium, Luxemburg and Monaco. (Spain was no kingdom in 1953) However, in the movie it is mentioned, that Ann is part of one of Europes' longest reigning royal families. The norwegian royal family reigns since 1905, the danes have their dynasty more or less since 1448, the swedish Bernadotte family reigns since 1818, the belgian family of Saxe-Coburg-Saalfeld since 1831, the royal family of Luxemburg governs since 1890 and the Grimaldi family of Monaco since 1331/1342. Ann is referred to as "her royal higness", which implies that her father is a King, yet Monaco is just a principality. So DENMARK is the most probable option.