Which movie started the tradition of black-and-white movies with a first person narrative of a cool, suave private detective that investigates on behalf of a client. (generally wears a fedora and a trench coat, maybe smokes, probably white)
Just to put some constraints on my search. You're looking for:
first person narrative of a cool, suave private detective that investigates on behalf of a client. (generally wears a fedora and a trench coat, maybe smokes, probably white)
This is what is colloquially known as the Hardboiled Detective (WARNING: TV Tropes link), a genre generally credited to Raymond Chandler's Philip Marlowe and Dashiell Hammett's Sam Spade, who started to gain popularity in the 30's. Hardboiled detectives tended to be tough, cynical, narrated in the first person, had a penchant for attracting dangerous women (the equally iconic Femme Fatales), and true to the fashions of the day wore hats all the time (usually fedoras).
Now taking those two iconic detectives we find that the earliest movie that any of them appeared in is The Maltese Falcon (1931) starring Ricardo Cortez as Sam Spade (10 years before the more famous version starring Humphrey Bogart).
Here's a clip from the 1931 movie.
Though it is quite hard to find the very first film that set the trend for Detective Stories.
At the Villa Rose (1920 film) is a film that I know fits the criteria (First Person Narration, Private Detective and White) of your question and is roughly a century old.
The link below is a filtered search of IMDb for the very first Detective Films ever Recorded.