Clarification: Trystane did leave Dorne on the large ship, along with his fiancé Mrycella and Jaime. They arrived at King's Landing, where a row boat was sent to pick up Jaime and Mrycella. Trystane stayed on the ship.
Trystane was waiting to return to Dorne at Jaime's order/request because King's Landing was too dangerous. At least that's what the note that Jaime sent the Prince of Dorne Doran Martell says (as published on HBO's official blog about Making Game of Thrones). Jaime knew that Cersei would have Trystane killed for revenge.
The Princess Myrcella died by poison on our return journey. I suspect Ellaria and not you, but my sister will demand war. I doubt Ellaria's head will appease her but it's a start, along with your neices. Your son cannot stay in King's Landing so I am sending him back on the same ship.
The reason that Jaime didn't return to the harbor at Sunspear wasn't explained, but the obvious reason is that there was nothing he could do. He would be outnumbered the minute he stepped off the boat and likely be killed instantly.