In Mystic River, Sean and Whitey don't have a clue about the real killers of Katie until they finally listen to the 911 tape. Sean notices that the caller refers to the victim as a she, although he couldn't know that the victim is a girl since he saw only the car and the body was dumped in the park. This tells Sean that the caller probably knew more about the case - but how did they make the connection to Silent Ray and his friend Johnny? Not once in the movie do we see them being in contact with the police and they didn't give their names either...
This break through might look a little vague but still there was a shot in the dark for Sean and Whitey. The only connection they could make from the case was that the shot fired to kill Katie was fired using the gun of 'Just Ray'. So there was a possibility that someone from the Harris household would have committed the crime. However the elder Harris had passed his polygraph with flying colors as mentioned by Sean. When they went to question Katie's boyfriend Brendan for the first time at his house they had met the Harris junior with his friend John O'shea too.
After listening to the 911 tape they found out that the boys knew more about the murder. Hence they might have taken a wild guess that mute Harris junior might have access to that gun and he was always with his friend John. Although it looks to be exaggerated but for investigative purpose they went to meet him.
One more feasible explanation can be that after listening to the tape Sean and Whitey had gone to question Brendan about those two boys whom Katie might have known or have grudge against. But at the time they entered John had already drawn a gun over Brendan. Once police got them they confessed everything.
Sean and Whitey knew enough facts about the case to at least make a weak connection to Silent Ray and John O'Shea after listening to the tape.
So all of this adds up to (in Sean and Whitey's mind): How can all the evidence point to Brendan and yet, based on the polygraph, he is not the killer? Answer: They were looking at the wrong brother. That at least warrants a conversation with the boys. The rest is explained by Qrious above.
As I stated in another post, from a writer's perspective, the call is simply the mechanism that allows the characters to break the case. Whether the call is a plausible case-breaker or whether the call itself is realistic seems to be the question. On the former, yes, on the latter, that remains open for debate.