I don't think the whole problem was ever explained and I guess in the end it's really just a plot device, especially considering how often McCoy scanned and treated alien lifeforms without any problems.
While I can't name any "precedent", the movie's story is essentially a trope: two parties want to make piece, hardliners sabotage it, inconvenient situations make everything worse while also masking the hardliners' plans. Later episodes will repeat similar plots, e.g. Voyager's crew trying to make peace with the Kazon.
However, if you really want to, you can roughly make sense of it:
McCoy said he doesn't know anything about Klingon anatomy. He probably meant he doesn't impromptu know anything significant about their anatomy.
This essentially left McCoy in a situation most of us could find themselves in:
It's considered basic knowledge to know the rough position and workings of human organs. What the heart does, the airflow through the lungs, etc. However, this basic knowledge doesn't make any of us capable of properly taking care of deep/penetrating wounds to the torso.
In a similar way, McCoy probably knew how Klingon organs are aligned, what they do, etc. yet he won't know without looking into books/databases on what to do e.g. to stabilize some organ or stop internal bleeding somewhere else (assuming such cases are documented at all). These wounds weren't just scratches or amputations. As far as I remember he even stated that he doesn't know which medication to try, as it could kill him. Even modern medicine faces such things in emergencies, considering allergic reactions and other things doctors have to consider.
Starfleet (and other races) probably gathered anatomical data, but also don't forget the Klingons' approach to warfare: it's considered shameful to get imprisoned, so chances are high Starfleet had next to no encounters with living wounded Klingons to "treat" or study. Also don't forget that many Klingons would rather die right now rather than being treated for longer periods of time (see Worf getting paraplegic in TNG), so chances are also high Starfleet could never steal any such information either.