The primary target of the U-Boats were the cargo ships...
Germany's U-Boat fleet existed to interdict the flow of supplies from the US to the UK, and cargo ships were the primary target of U-Boats. Typically, they would conserve the very expensive torpedoes and sink cargo ships by surfacing and firing their deck guns whenever possible. They had torpedoes so that they could shoot at the cargo ships when escorts (or aircraft) were present and surfacing would be far too dangerous. Remember, a U-Boat that sank ten escorts and no cargo ships would have failed in its mission. By contrast, a U-boat that sank multiple cargo ships per patrol without ever sinking an escort would be considered wildly successful.
...but U-Boat crews were people and they valued their lives
With all of the above said, U-Boat crews wanted to get home alive and sometimes that meant using those expensive torpedoes to attack the escorts that were trying to kill them. Torpedo attacks against escorts would be rare, since the U-Boat's first and best option for survival would be to go deep and slip away; however, such attacks definitely happened. In addition, a U-Boat commander might attack an escort to clear the way to kill cargo ships. He might even attack to help a fellow U-Boat being prosecuted by the escort... or to avenge one that was just sunk by an escort. And then, of course, you have accidents: a U-Boat mistakes an escort for a cargo ship, or an escort inadvertently gets in the way of a torpedo aimed at a cargo ship.
The focus was on sinking cargo ships and evading escorts, so attacks on escorts would tend to be rare, but they did occur and it is not unrealistic.