I always thought wearing a robe over armour was pretty stupid actually. Now if they accomplished the same goal by letting you wear a cape that would be OK.
In Morrowind, you could wear clothing under your armor. I mean - who wears their armor right next to their skin? The chafing! :cryvaultboy:
In Daggerfall, you could wear a robe and at least two cloaks over your armor. You could also wear pants with that, though I believe the robe took the shirt slot.
I just want it because it looked really cool. Especially when you got mods involved so the robe could be open to reveal parts of the armor underneath. Cool look.
In Daggerfall, the same piece of clothing could be worn different ways. A robe with a hood could be worn with the hood up or down. Many shirts could be buttoned all the way up or left halfway unbuttons. Yes, beware of cleavage showing if you are female

It's an opportunity cost issue: dev time spent on implementing this feature is dev time not spent on developing other features. Making a system where cloth with deform realistically over a variety of armor shapes is not trivial.
Just make the robe baggy. That's the type my Daggerfall character is wearing... with a cloak over top!

Naaah... Having everything at once kind of defeats the purpose of chosing between higher protection and higher magical abilities.
You're turning it into a balance issue. Fashion is hardly about balance. It's about realism and choice. Why should we NOT be able to wear a robe or cloak over our armor if we choose to do so? Daggerfall did it. Morrowind did it. Not to mention the role-playing aspect it brings. My Daggerfall Nord likes to wear robes and cloaks over her armor in order to hide what she's wearing underneath