There are two reasons why Hindostan is spelt like this.
1. Historical reasons.
India had different names in the history. Its names include Bharat, Hindustan, Hindoostan, Hindostan etc., This movie is set in late 18th century when Europeans had visited India for trade. Mughals were ruling Indians by then and the name was Hindūstan (meaning the land of Hindus) but the spelling was not the same. There were variations for this name. It was written Hindostan, Hindoostan etc., to reflect the pronunciation of long o. But in this process, the pronunciation changed.
India was named like that by the British scholars and historians of that time.
From Wikipedia,
These dual meanings persisted with the arrival of Europeans. Rennel produced an atlas titled the Memoir of a Map of Hindoostan or the Mogul Empire in 1792, which was in fact a map of the Indian subcontinent. Rennel thus conflated the three notions, 'India', 'Hindustan' and the Mughal Empire. J. Bernoulli, to whom Hindustan meant the Mughal Empire, called his French translation Lar Carte generale de l'Inde (General Map of India)
There were books published by authors using these names.
The Complete Hindoo Pantheon, Comprising the Principal Deities Worshipped by the Natives of British India Throughout Hindoostan (1842) was one such book. So, this name Hindostan was one of the variations of the name Hindūstan. So, it is a type of changing the original word. They did it for many places. For example, a town in the state Andhra Pradesh, India was originally called "Raja Mahendri" and it was changed to Rajahmundry by the British.
2. Indication of the emblem of Thugs (the rebellion group)
If you observe the poster and the trailer of the film Thugs of Hindostan, one thing is clear. The movie is about a rebellion group led by Khudabaksh (Amitabh Bachchan) and the East India Company. The rebellion is referred to as thugs. There is a special way of writing the film name on the posture. There is a logo of an Eagle in place of 'O'. This is the emblem of the rebellion group or the thugs. This eagle is seen in many places in the trailer. Such instances are below at 0:29 and at 3:10 when the name is shown in the trailer.


In the film, Amitabh Bachchan is the leader of the rebellion. He has an Eagle as his pet. So, I think this is chosen as the emblem of the group and has an importance in the film.
The film posture is below showing Amitabh Bachchan.

Eagle is shown in the image and also the logo of an Eagle in the place of 'O'.
Hence, it is shown to indicate the emblem of rebellion group Thugs of Hindostan
So, these are the two reasons why Hindostan is spelt that way.