It depends on what difficulty you play on, and of course, your playstyle.
My vanilla experience was:
Melee Warrior/Archer/Thief were (too) easy on Master.
Mages that focused on Conjuration were about the same,
Mages that focused on Destruction require constant kiting, impact abuse, and care (potion spamming, enchantments, etc).
Also, since Destruction doesn't scale, you are dependent on the next level of spells becoming available when you near a threshold. So when you're around a 45 Destruction skill, firebolt (or whatever) is a peashooter that pings off small chunks of damage from enemies. There are difficulty spikes until the next "level" of spells become available.
And of course, you begin to grow weaker relative to enemies starting around level 45 as a Destruction mage.
If you play Adept, or even Expert, I think Destruction is probably just a tiny bit more difficult, but on Master you notice the difference. But it's not so noticeable if you play a "well-rounded" mage, because most damage will be dealt with other schools, like Conjuration and Illusion.