Let's use the Alessian Order as an example. In the First Era, this group ran Cyrodiil (and whatever other territories the Alessian Empire had picked up on Cyrodiil's borders with other territories), and had such great power that Skyrim's King Borgas aligned himself under them as well (which meant that Skyrim, itself a rather powerful empire at the time, was officially sanctioning the Alessian Order's actions in its territories). The Order was extremely anti-Mer, so any balance between the Alessian Empire and the Ayleids was destroyed during this time and Ayleid records and libraries would have been destroyed as soon as the Alessians could get their hands on them. Then when the Order itself began to fall, its own libraries became targets for its enemies--and the Order had many enemies, including Borgas's successor Wulfharth (who gained the title Ysmir for destroying Order outposts in Skyrim) and anyone else who was displeased at the Order's oppressive and expansionist policies, to say nothing of its rather radical theology. In other words, not only did the Order destroy plenty of records, but the majority of the records that were left (which were their own) were thereafter destroyed as they themselves fell into decline. And this is all in addition to the long amount of time between that period and the Third and Fourth Eras, nor does it take into account all of the other disasters (such as the Interregnum) that also took place during that timespan.
And that's just the amount of knowledge that is completely lost. I wonder how much sway Hermaeus Mora has over information that is secret to most, but known to a very select group of people--that is to say, because the information is forbidden to people that are not [insert group here; the Thalmor are always a good subject for discussion, so we'll use them], does that mean that it falls under HM's purview simply because most aren't permitted to know it or even know about it?