Generally in thriller-type movies, there is a scene where A & B are talking.
But the scene ends with C listening secretly with an intensifying thriller sound.
Is there any single name for this common effect?
A musical sting can be used in drama, comedy, horror or any genre, and in radio and television advertising. It is a part of the music director's lexicon. It is often used to build tension. Stings are often used in horror movies to accentuate jump scares.
A cousin to the Sting, the Scare Chord is a sudden, sharp sforzando of dissonance or sheer noise intended to make viewers jump clean out of their seats. The thing that separates it from a Cat Scare is that it's non-diegetic: nothing onscreen causes the noise; instead, it's added to the soundtrack as a way of eliciting a jolt in the audience, often preceded by a deliberate lull in both narrative and musical action.
I think you are referring to the "Shepard tone":
A Shepard tone is a sound consisting of a superposition of sine waves separated by octaves. This creates the auditory illusion of a tone that seems to continually ascend or descend in pitch, yet ultimately gets no higher or lower.
This effect is named after Roger Shepard, an American cognitive scientist considered a father of research on spatial relations. He's also the father of the optical illusion called "Shepard tables."
I'm not sure of a film term, but musically, it's building tension by not resolving a chord progression.
Generally music is in a particular key. A key is denoted by its tonic note. So for example, C Major has its tonic note as C. You can think of the tonic as the stable base of the key. Composers form chords by combining different notes in the scale of a given key. These chords typically produce certain common psychological responses in people. And one of those responses is tension.
There are certain chords that feel like they want to move back to the tonic. In C Major, that would be a G 7th chord or a B diminished chord. They have notes (particularly the B) that make it feel like they're moving to, but haven't quite reached the tonic. And if you follow them by a tonic chord, it feels resolved.
So composers use this to produce or enhance a feeling in the soundtrack. When a character is running around trying to get away from the killer, that's pretty tense! So composers will often compose music with lots of unresolved chords to further that feeling of tension.