Tarkovsky, at least in the 60s, thought that color in film was a problem in that he seemed to feel it was pandering and commercially oriented and was some form of imposition on the audience.
That said, he would frequently use both black and white and color and (if I recall correctly) he even went as far as to have a scene in B&W with people watching a screen that had color footage playing on it.
Here is a quote from an interview he did in 1966:
On the screen colour imposes itself on you, whereas in real life that
only happens at odd moments, so it's not right for the audience to be
constantly aware of colour. Isolated details can be in colour if that
is what corresponds to the state of the character on the screen. In
real life the line that separates unawareness of colour from the
moment when you start to notice it is quite imperceptible. Our
unbroken, evenly paced flow of attention will suddenly be concentrated
on some specific detail. A similar effect is achieved in a film when
coloured shots are inserted into black-and-white.
(source)
While I feel the other answers are correct about potential symbolic meanings, he is also very much interested in the subjectivity of the moment.