I've played Skyrim for 3000+ hours now, and this question has preyed on my mind much of that time. Given that Tamriel is a hack-and-slash world (well, Mundus actually) where Magic works, there seems to be no penalty at all for a mage to cast Magic AND wear armor at the same time. So why, why, why, why would anyone willingly risk his neck and other bodily parts to the ravages bestowed by hackers and bashers by NOT wearing armor? Nearly all robe-type garbs have a ZERO armor rating. Worse, most magically imbued robes are specialized, restricting the robe's magical properties to just one School of Magic.
I could see the eschewing of Light and Heavy Armors if wearing those items diminished a spellcaster's proficiency. But from what I have seen after thousands of hours of gameplay, there is not one iota of ability lost by wearing armor while casting magic. Shouldn't simple self-preservation motivate nearly all magic-users to improve their survivability by donning at least Light Armor? If anything, encountering a spellcaster wearing just robes should make most warriors go weak in the knees just thinking about the fact that the wizard is soooooo powerful, he has no NEED to wear armor. Instead, I find that when I encounter robed spellcasters, I'm thinking "Goody! Easy pickings!" And I imagine that pretty much all hack-and-slashers would be thinking the exact same thing. So the hordes of spellcasters wearing robes instead of armor must be really, really stupid to think that they are gaining any benefit by wearing robes anywhere except at the College of Winterhold.
Additionally, since nearly all apparel can be imbued with one enchantments, the robe-wearer is cheating himself out of the ability to be wearing four or five (body, head, hand, feet, and shield) rather than just two (head and body). [EVERYONE has the ability to also wear a ring and a necklace at the same time.]
To motivate a spellcaster to wear ONLY robes, there really has to be something in effect that makes it more beneficial to willingly sacrifice the body protection provided by armor. Something like, "Thou shalt not be able to cast spells while wearing armor," or "An armored individual will NOT be able to employ a magical staff while so armored." Something.
“No matter how subtle the wizard, a knife between the shoulder blades will seriously cramp his style.”
― http://www.goodreads.com/author/show/27704.Steven_Brust as Vlad Taltos in Taltos