What does the canary in the cage, seen inside the Alien ship, and brought by the human scientists, refer to or symbolize?
I assume you're referring to the canary in the cage that they bring into the alien vessel the first time they go inside. I can't find a screenshot of it, otherwise I would include one for clarity.
Historically, people working in mines would bring a canary with them underground. The idea was that if the cavern began filling up with otherwise undetectable toxic gasses (for example - carbon monoxide or methane) then the bird would die before any people did. As such, the bird's death would serve as a warning signal for the miners to get out.
When they entered the alien vessel, they wore radiation suits for protection, even though they explicitly said they knew radiation levels were nominal. They had no idea what to expect inside, after all, as this was a vessel from another world. So, they took every possible precaution. Including, apparently, a canary in a cage - to see if it died from being exposed to anything.