The men working didn't have any reason to take the flag or the mast as part of their assigned duties, and as prisoners cared little enough they didn't avoid trodding over it.
Javert however as one in power cares deeply for the institutions of the republic. Not only the law, but also representations of it such as the flag. His request is not really a specific duty that he is trying to submit Valjean to, but a simple request that the flag should not be left in the muck. In his mind a minor, but necessary thing that should be done before releasing the prisoners from their duty.
Later in the novel (but not in the play/movie), Valjean has further tasks on a ship where his raw physical strength is described.
So I interpret the scene as both a way to show Valjean's physical strength, and to show that he is not willing to submit to the demands of the law over his desires/needs, even if it is the easier way. Stay put in jail for five years? No, 19 because I kept trying to escape! Javert has one more degrading thing for me to do? He wants the flag? Fine! I'll bring him the whole stupid mast!