Uh, this is The Elder Scrolls game right? I mean the whole point of these games is that you can be anything you want, go anywhere you want, have any skill you want.
But not without consequence or penalty. In Oblivion, you could wear heavy armor, but you were encumbered, or you could wear light armor for less encumbrance and less protection. In fact, IIRC, Todd mention that "unarmored" was supposed to work in Oblivion by not having the penatlies of armor (though the overpowered perks ruined that by removing the penalties with the armor, which made unarmored not as viable a choice).
There is also the whole "lore" thing going on with magic users that wear armor. Most, if not ALL, of the original classes in Morrowind and Oblivion not only encouraged you to wear armor, but it was built into the class system. Battle Mages could wield armor and cast destruction spells, Crusaders were like Paladins.
Of course you can still play mixed classes. But they need to be balanced so pure mages are just as viable as Crusaders or Battlemages.. that means restricting your mage abilities when using non-mage skills. A Battlemage isn't going to be as good of a mage as a pure mage, nor as good of a warrior as a pure warrior.
Balance is still important in a single player game, otherwise the whole system can get out of hand and devolve into punishing everyone except for those using a select few play styles.