This is addressed explicitly in the biography First Man: The Life of Neil A. Armstrong, by James R. Hansen, which the film is based on, and whose author is presumably the Jim mentioned in wcullen's answer.
Armstrong never released any information about the contents of his [Personal Preference Kit]. He agreed to do so for publication in this boo, but reported that he was unable to find the manifest among his many papers.
"I didn't bring anything else for myself," Neil would declare. "At least not that I can remember." As for Janet, the only thing taken to the Moon for her was the olive branch pin. "He didn't ask me if I wanted to send anything."
Perhaps surprisingly, Armstrong took nothing else for family members--not even for his two boys, a fact that still distresses Janet. "I assumed he had taken things to give to the boys later, but I don't believe he has ever given them anything. Neil can be thoughtful, but he does not give much time to being thoughtful, or at least to expressing it."
Another loved one that Neil apparently did not remember by taking anything of hers to the Moon was his daughter, Karen. What could have made the first Moon landing more meaningful "for all mankind" than a father honoring the cherished memory of his beloved little girl, by taking a picture of the child, dead now over seven years (she would have been a ten-year-old), one of her toys, an article of her clothing, a lock of hair, her baby bracelet? Astronaut Gene Cernan, just before he left the lunar surface on Apollo 17, had written the initials of his nine-year-old daughter, Tracy, in the dust. Buzz Aldrin carried photos of his children to the Moon. Charlie Duke left a picture of his family on the surface.
What if Neil did something for Muffie but never told anyone about it, not even Janet, because it was of such an intensely personal nature? How much more would posterity esteem the character of the First Man? It could have elevated the first Moon landing to an even higher level of significance. Among those who feel so are Neil's sister June, who knew her brother as well as anyone.
"Did he take something of Karen with him to the Moon?" was June's rhethorical question.
"Oh, I dearly hope so."
Perhaps the mystery will be solved when humankind returns, as it surely will, to Tranquility Base.
As mentioned in the existing answers, this makes the incident a plausible educated theory, but I do find the added context w.r.t. Neil's living sons to be certainly relevant.
I'm bothered by the exact textual match between Hansen and Singer's report of June Hoffman's response. Were they both present in the same interview? Did she give a word-for-word exactly identical response on two separate occasions? Is either of the two accounts a paraphrase of the other? (Just things to consider.)