Exactly!
I always say this to people who object to levitation because 'lore does not support everyone whizzing about everywhere.'
(Or other things the player easily does but not NPC's.)
I try to say that the people we play in the games are not common citizens by a long shot.
Isnt there even an NPC who comments on this?
Something along the lines of: 'Its ok for you adventurers, you can just go loot a dungeon. I have no such skills.'"
It isn't only the number of people who have trained to use magic, it's the fact that magical items aren't uncommon, that it seems to be a skill that only needs to be learned rather than a natural inclination shared by rare individuals, and the fact that magic and the fantastical in general is so widely accepted and viewed without superstition by the populace.
Certainly not everyone can cast high level destruction spells, but it seems fairly common for at least the upper classes to have spent some time learning minor enchantments or acquiring magic items.
There's nothing wrong with it, and it fits the setting perfectly. I don't think it would work particularly well at all to suddenly decide that the populace would burn people for witchcraft unless they were a sanctioned imperial wizard, and that any magic user was at constant risk of insanity or mutation just for casting a spell. I'd love a game set up like that, just not a TES game set up like that.