I watched the full movie and did not got any hints about what the Book of Shadows is. Maybe I missed some part or it means something else?
I can't find anything definitive, but:
- The two main characters are researching the Blair Witch for a book they are writing.
- The characters are a Wiccan and a Goth Psychic.
- A "Book of Shadows" is a Wiccan text containing spells, rituals, etc.
- The television show 'Charmed' was wildly popular at the time and the "Book of Shadows" is a critical element to the show.
- The "Book of Shadows" is never mentioned in the movie.
Based on those items, barring anything else definitive, I think the title is tenuous and convenient and has nothing to do with the movie itself.