I also don't think it helps that you started with -10 INT in Oblivion and Morrowind, but still.
It isn't unfounded (just look at Nord NPC's in every game since Morrowind--they don't come off as particularly bright) but for the most part it's just a generalization.
Sounds like a potentially interesting roleplay opportunity to me. A character might make it their life's goal to prove to dismissive, prejudiced people that they are not stupid.