When filming on location for a major production, as in out in public, you usually have to obtain permits to film wherever you're about to film. Roads and streets may be closed down. You usually also need to notify the local police of your actions so they know what's going on. The production company may also hire security to keep people away from the set or area they're filming. When an area is closed off for filming, this is referred to as a closed set, meaning no one who's not supposed to be there is allowed to be there.
The city of New York, for example, will actually provide free police assistance to help out, likely for things like guiding traffic and general security of an area being used for filming. They also state that filming with a hand held camcorder (like what you'd buy at any consumer electronics store) or tripod setup will usually not require you to have a permit. Further, they state that the production company must allow any residents or citizens that work in the area access to places they're supposed to be. They even dictate that if production lighting is pouring through someone's window, they need to provide blackout masking for the window so they don't disturb the resident.
However, if you and a friend simply went around with a hidden cam and were acting like police officers without the backing of the city and local police force for the area in which you're filming, you'd almost certainly run afoul of the law. Remember: ignorance of the law does not mean you can't get in trouble for breaking the law.
Generally, if you're going to be filming something that involves actors who look like emergency service personnel, contact the police for the local area and get the lowdown on what you need to do in order to film. Major cities like New York have entire departments dedicated to helping production studios or even small indie projects film within the city, and can also guide you on how to properly handle yourselves to avoid trouble with the law. These laws can be different for each city as well, so what one city allows may not be allowed in another.