Well given the 200 yeah gap who knows what violates the lore. The books may not be that accurate of a predictor. The thing is while everyone physically can use magic as you said they need to be taught. How rare that teaching is can vary. With the falling of the empire they might be entering something like a dark age and knowledge might be lost or become more restricted. Then again it might not, we just don;t know yet.
D&D not the crappy 4th edition but the original stuff magic could be learned by anyone of average intellect or higher, you had to be brilliant to learn the highest levels of magic but that seems fairly similar to the ES in requirements for magic. And yet mages were rare and feared for having magic in most of their settings. How magic is taught is a big variable on a things rareness even if everyone can do it.
I can't remember who it was, but some forumer made a comparison that I think couldn't be more accurate: magicka is like math. Everyone knows it to a certain extent, but only some people really have a knack for it and excel at it. To get really good at it, you'll probably need guidance, but you could figure out a lot on your own.
To the OP, magic IS kind of like a sword in TES. People would no sooner run from a skilled swordsman than from a skilled mage.
Personally, I prefer it this way. I don't like it as much when magic is this rare trait in some people and only a select few can learn it, mostly because those who can are completely over-powered and they usually dominate the whole series and the stories.