Sometime around the later years of The West Wing (season 4? 5?), Dulé Hill (Charlie Young) tends to only appear for one brief scene in most episodes. (Fans complained about it, e.g. in Television Without Pity.) Was this because Dulé Hill was co-starring as Burton Guster in Psych? Was Hill, maybe, flying down from Vancouver (where Psych was filmed) to L.A. (where The West Wing was filmed) once a month or so to record a month-or-so's worth of scenes?
I think it had a lot to do with the departure of Aaron Sorkin. Sorkin wrote the first four seasons, then left the show after which it was written by other people. The end of season 4 is when the President's daughter Zoey gets kidnapped, and Charlie was a big character up to that point since he had the unrequited love for her.
So I'd say a few episodes into season 5 is when Charlie started to appear less and less. It's also, in my opinion, when the quality of the show took a nosedive. Season 7 is excellent but it's basically a different show, it's a show about a presidential election campaign with eerie parallels to 2008's real life election.