I didn't understand why the tribespeople suddenly left Papillon alone in their village?
They'd clearly invested time and effort building the huts but then they just up and went. Why?
The tribespeople seem to be nomadic, traveling between the islands on their collection of boats, trading for supplies and collecting seasonal fruit and vegetables from the (otherwise non-inhabitable) islands.
They made a brief stop at the island where they harvested a crop of oysters and their pearls, which they then traded for artifacts of value from the nearby prison colony.
You may wish to note that they ultimately left the remaining (worthless to them) pearls in the makeshift village, presumably as a symbol that they would no longer return in future.