I've gotten to the point where I can't forget a single holotape in both recent fallout games. TES has so many books though! I really liked the holotapes in fallout though because most of them tell the story of where they were found.

The books from TES feel like they are telling stories from different times in different worlds, but I'm really trying to get into the TES lore like I did with fallout.

So should I even try? Do the stories better prepare me for references in Skyrim?

Should you try to read every single book? If you really want, but a lot of books can and will make your head spin if you do not crack down on its purpose and what it's trying to tell.

Otherwise, http://www.imperial-library.info/. Every single in-game book, Pocket Guide to the Empire (great for background information, and was given in the TES games), and obscure text (don't read till you understand the in-game books that deal with myth and metaphysics). The big thing that'll help prepare you for Skyrim is to read the pocket guides, books on religion/myth of TES, and hanging around this section of the forum and read the discussions.

No. Unless you're really bored or have a ton of time. Reading everything isn't a good way to absorb information. I'd read over the introductory basics and then research whatever interests you. Check the http://www.gamesas.com/index.php?/topic/1112211-how-to-become-a-lore-buff/ for a list of important books.