Dumb question, but where does it come from? Is it all from books?

The vast majority of it is in-game content, such as books. There are also the 'Obscure Texts' which are out-of-game things by devs and former devs, most prominently (and bizarrely) Michael Kirkbride, but they're in a sort of canon limbo. Sometimes, things from the Obscure Texts make it into the games, like the idea of the Towers, so they're probably a good thing to look into if you're interested in the lore.

First, there's the game storylines themselves. A lot of lore gets explained or shown through many of the quests and characters in the games. Then there's the books, which offer a ton of background on specific subjects, and you can even glean culture and other things from the in-universe fiction. Then there's the Pocket Guide to the Empire, first and third editions. The first edition was a pamphlet packaged with copies of Redguard, back before Morrowind was made and about the time when the lore really started to evolve from generic Tolkien/DnD clone to something of its own - it describes all of the provinces and races in Tamriel, among other things, from the perspective of the Septim Empire just as it was starting to conquer Tamriel. Then there's the third edition, packaged with Oblivion, that describes the provinces and races at the point in the timeline where Oblivion starts. Then there's the obscure, out-of-game texts, which are vague, open to interpretation, and mostly only canon if you want them to be.

Remember that since most of the lore comes from an in-universe source, almost all of it is biased in one way or another. The Pocket Guides are pretty much ethnocentric propaganda. (Cyrocentric?)

The most fun is when there are multiple different accounts of the same event, told with different biases. A good example would be the books Biography of Barenziah and The Real Barenziah, the former of which has an obvious pro-Imperial bias and the latter seems to value entertainment more than historical accuracy.

Heck, Morrowind even made this conflict of historical accounts a major plot point! (what happened at Red Mountain etc.)