"No Sale" means the plan isn't going ahead
His plan was to see if a Russian trade specialist wanted to defect:
Mr Guillam sent me to Istanbul to check out a Russian trade delegate who might be persuaded to defect.
He recognized from the behavior of Boris that he was in fact a spy, probably on the lookout for an approach exactly like Ricky was intending to make, i.e. Boris has some sort of counter espionage role:
Anyway, I took one look at him and I knew I'd wasted a journey. Boris was no delegate. What delegate bothers to play drunker than he really is? You get to recognise your own, don't you, Mr Smiley?
So he communicates back to Guillam that the plan was not going ahead. "No Sale" is just his shorthand for "the plan isn't going ahead". It might literally have been what he put in his cable to London but more likely it's just conversational shorthand in his discussion with Smiley.
"No Sale" almost literally means "No deal" or "No agreement", and the expression is often extended to refer to any kind of request, offer or plan.