In short, it isn't simulated. It is a real but small leak, and in addition they exaggerate its size by tampering with the wearable dose-badges that the crew wear.
In the book, someone makes a small radiation leak by drilling a small hole through a thread of a connector in some part of the cooling system. There is only enough pressure to leak when the reactor is fully powered up to run the engines. The radiation detectors near the reactor have also been disabled so that the leak isn't revealed immediately while they are still close to the USSR. Its left unsaid, but you are meant to presume this has been done by the Chief Engineer.
In addition, another member of the conspiracy tampered with the radiation badges (by exposing them to an X-Ray machine in the doctor's surgery).
A few days into the plot the doctor (who is not part of the conspiracy) routinely collects the badges and photographically develops them - and discovers that they show that a leak has occurred. This is part of the plan to unsettle the crew, and provide an excuse for unexpected behavior from the captain. There really is a leak, but the conspirers have made it appear bigger than it really is.
Initially the conspiring officers send some middle ranking engineers to find the leak with a hand-held detector, and they cannot because the reactor is powered down and the steam from the cooling system isn't under enough pressure to leak - so the non-conspirators do not know whether there is a leak or not, perhaps the dose badges are faulty or someone has deliberately exposed the badges to radiation. So they power up the reactor and continue the journey to Cuba.
It is only later in the plot that the small leak has been running long enough to trigger a detector further forward in the submarine. At this point the Chief Engineer joins in the search for the leak and 'discovers' how the leak was made and fixes it.
By this time the crew is very unsettled by the leak even though the levels of radiation are low. So the leak has performed its purpose by the time it is fixed, promotes rumors of sabotage, and sets the scene for them using the reactor as an excuse to do whatever is needed later.
I think it was important to the conspirators to have this take a few days to work through, as this kept the crew busy and worried, but that it was eventually found and fixed. An uncontrolled leak would have given the conspirators only few possible plausible courses of action, and at this stage they do not know how events will work out. So they manage to engineer a situation where they can use the reactor as an excuse if necessary, but only if necessary.
The book is quite long and includes all this detail - a lot of this is left out of the movie obviously, but the principle of the plot is still the same - the officers engineer a deliberate leak.