the number of books in tamriel might be a little... surprising, however even looking at the real world it's possible to find examples of literacy in early cultures (example - the Book of Kells)
true, the greater population wasn't usually too literate, and it was mostly just churchmen and nobility, but: i make the assumption that, in tamriel, church and state saw early on that a literate population might be more beneficial than one that didn't have the opportunity. after all there's no real point producing written propaganda if only a handful of people can understand it
and tbh, the mages guild probably impacted this too - taking in anyone, and teaching them a great deal about magic, would have encouraged note taking. researching and sharing results would have
required literacy, or impeccable memories and conversational skills.