In the film Logan, was Caliban helping Alkali Transigen capture mutants? I remember a dialogue mentioning so. If so why and what made him stop?
As a Marvel Comic Character Caliban was originally a villain; this is no real surprise, as many Marvel heroes began their tenure as evil do-ers (the Avenger Hawkeye amongst them!)
The fact he previously used to help transigen track mutants could be a reference to his Comic Book Origin Story, or the fact that he has appeared (although in a markedly different guise) in the continuity already; here is a screenshot of him from Age of Apocalypse:
Whilst Logan director James Mangold has gone on record to state that the two depictions of Caliban were accidental, he has made no efforts to clear up whether they are explicitly unrelated characters (more likely to not been seen undermining another film as less canonical, rather than to establish a direct link between them).
So in both Movie and Comic book appearances, Caliban has previously debuted as 'a villain'; this is upheld in Logan.
Given that Transigen's ultimate goal was to remove Mutants from the world, and this was a hidden agenda (likely hidden from Caliban), it's not unreasonable to assume he would have had a 'change of heart' at some point whilst working for them.