Hulk is based on comic character known for his anger being out of his control. In fact, intially he converts into Hulk only when he gets angered by someone. He started controlling himself in Avenger first time which leaded to many questions e.g. thisthis and otherother . So Bruce Banner prefers to live away from city either alone in forests or some peaceful tribes where people will not bother him.
Bruce Banner and Hulk are almost two different entities like Dr. Jekyll and Mr. Hyde. Hulk may not be evil as Mr. Hyde , however his out of control anger results to lot of destruction. So Bruce Banner needs lot of confidence to live in city and other side , how much he has control on Hulk, someone will find way to piss of him by experiments or other means.