I was likewise more impressed as a child by that small effect than (for example) the "passage of centuries" sequence.
I don't know how the special-effects team accomplished it, but I know how I would have done it.
The single book he handles first would be more difficult, but taking sheets of thin paper and then heating them to near their ignition temperature, say 400°F, for several hours should make them brittle and friable like that. Then carefully make them into a "book".
The book shelf is a much simpler trick: there are no books! There is simply a long sheet of paper with the spines of books drawn on it. The sheet is stretched across the space where the books should be and liberally dusted with flour. Rod Taylor just sweeps it aside with his hand; the curtain on the right hides the bunched-up paper.