The inscription on Mjolnir reads;
"Whosoever holds this hammer, if he be worthy, shall possess the power of Thor."
We know from Thor, The Avengers, Thor: The Dark World and Avengers: Age of Ultron that only those judged worthy can lift Mjolnir. When judged worthy, the hammer gives the lifter the power of Thor. I elaborate on this in this answerthis answer, but the short of it is that those worthy of Mjolnir can not only lift Mjolnir, but they are also granted power by it.
Importantly, when we do see The Vision use Mjolnir rather than simply holding it (when he hits Ultron with it), we see that he hits no harder when 'wielding' Mjolnir than he would have anyway. The Vision is not granted the power of Thor, as the inscription on Mjolnir would suggest. This means that Mjolnir does not judge The Vision worthy, and as such can only be lifted by The Vision because he is not counted as a person from Mjolnir's perspective.
Additionally, when we see Thor regain his worthiness in Thor, his lifting of Mjolnir is accompanied by various special effects indicating him being 'powered up', none of which are present when The Vision lifts Mjolnir - because he isn't worthy, and has not been granted the power of Thor.
This means that when Tony and Steve are trying to comfort Thor at the end of Avengers: Age of Ultron, they are in fact correct - The Vision lifting Mjolnir makes him no more worthy than an elevator lifting it would, or as the Helicarrier in The Avengers or a coat hanger in Thor: The Dark World does.
In short, The Vision can only lift Mjolnir because his status as an android would seem to make him exempt from the rules.