In Mystic River, Sean and Whitey don't have a clue about the real killers of Katie until they finally listen to the 911 tape. Sean notices that the caller refers to the victim as a she, although he couldn't know that the victim is a girl since he saw only the car and the body was dumped in the park. This tells Sean that the caller probably knew more about the case - but how did they make the connection to Silent Ray and his friend Johnny? Not once in the movie do we see them being in contact with the police and they didn't give their names either...
This break through might look a little vague but still there was a shot in the dark for Sean and Whitey. The only connection they could make from the case was that the shot fired to kill Katie was fired using the gun of 'Just Ray'. So there was a possibility that someone from the Harris household would have committed the crime. However the elder Harris had passed his polygraph with flying colors as mentioned by Sean. When they went to question Katie's boyfriend Brendan for the first time at his house they had met the Harris junior with his friend John O'shea too.
After listening to the 911 tape they found out that the boys knew more about the murder. Hence they might have taken a wild guess that mute Harris junior might have access to that gun and he was always with his friend John. Although it looks to be exaggerated but for investigative purpose they went to meet him.
One more feasible explanation can be that after listening to the tape Sean and Whitey had gone to question Brendan about those two boys whom Katie might have known or have grudge against. But at the time they entered John had already drawn a gun over Brendan. Once police got them they confessed everything.