Reliant's shot was not meant to damage or alter Enterprise's course; there's nothing that implies it was a "shot across a bow" as you mentioned in your question.
The intention of the shot was merely a warning: you're within range of our weapons; if you keep moving your ship forward, the next shot will cause real damage.
Khan was counting on Kirk ...
Khan wanted the Enterprise intact. He was hoping that Kirk would rather give the ship up than see it potentially destroyed in the Nebula. The purpose of the warning shot was to let Kirk know they would further engage if he kept running.
Obviously, it didn't work.
Primarily because there was no way that Kirk was going to give up his ship.