By listening to the Director's commentary in the Blu-ray, the idea is that Loki's scepter does not only affect people, controlling them by touch (as with Barton/Hawkeye) but the mere presence of it can create confusion and insert anger into people's hearts around it, at least in the near proximity of the scepter.
The shot where the camera flips and everybody is upside-down behind the scepter is to indicate that it is turning everybody not evil, but at least angry and "out of character". Banner and Stark are the ones that spend more time around it in the lab, but they are also the more intelligent and reasonable so, the scepter affects them more slowly, but in the case of Natasha, Fury and Rogers, they become angry very easily.
Anyway, the one taking the scepter is Banner, cause he's the one with the rage problems, thus, the one the scepter has more power over once he lets the anger flow when he starts arguing with the others, the scepter appears to give him direct "commands" without him noticing it, indicating that the people under it's effect are not turned evil, but are under mental control. That's why when he becomes Hulk he goes berserk, he's not evil, but the influence of the scepter has turned him angry beyond reason.