What does the canary in the cage, seen inside the Alien ship, and brought by the human scientists, refer to or symbolize?
I assume you're referring to the canary in the cage that they bring into the alien vessel the first time they go inside. I can't find a screenshot of it, otherwise I would include one for clarity.
Historically, people working in mines would bring a canary with them underground. The idea was that if the cavern began filling up with otherwise undetectable toxic gasses (for example - carbon monoxide or methane) then the bird would die before any people did. As such, the bird's death would serve as a warning signal for the miners to get out.
When they entered the alien vessel, they wore radiation suits for protection, even though they explicitly said they knew radiation levels were nominal. They had no idea what to expect inside, after all, as this was a vessel from another world. So, they took every possible precaution. Including, apparently, a canary in a cage - to see if it died from being exposed to anything.
Coming back to an old answer years later, but just wanted to add something I recently learned myself: Apparently, the canaries used in the mines didn't usually die, they just fell unconscious. The miners would bail out when the bird stopped singing, and as they tended to treat the bird like a pet, they would take it with them so it could recover once outside. I'm sure some of them did die, but at least it wasn't a foregone conclusion.