But as far as practicality and game mechanics goes, there are perks to mastering one type of armor, and you only get those perks if every piece of armor you're wearing is either Light or Heavy, so it really pays to specialize unless you're extremely into roleplaying or fashion.
The problem with the game mechanics is that the perks of wearing all one type are
less than those of not wearing any at all. At Heavy armor Mastery, your armor "weighs nothing" but limits your spell effectiveness to 95% and gives you 85% "weapon resist" even with enchantments. The same for Light Armor, capped defense, limited spell effectiveness, so you can do better without it. Enchanted clothes give you the same 85%, but spell effectiveness is 100% and the weight is still minimal (and if you pick the right ones, can be zero).
I like the idea that magic is as effective as physical armor, but it shouldn't exceed it. It would have been better if spell effectiveness had been tied to defense directly, so that a mage has to choose between defense and offense. And it has never made sense to me how arrows work with armor, reflect damage etc. and why there is no magical way to deflect them.