In Oblivion you started out wearing leather/iron and as you progressed in level you went from looking "meh" to "bad-ass!", but for characters wearing robes you went from "my robe is red" to "my robe is blue". There are a bunch of games with robes that have shown me you can do cool things with robes, like giving them spikes, skulls, cool embroideries, jewels. The order cultists' robes were the only ones that looked different and they looked awesome.

I also believe only having a 5% penalty for spells when wearing armor was a tad too lenient. Instead of your fireball doing 50 damage it only does 48!