Think about it. If I am a one-handed warrior, I max out my damage by taking a ton of one-handed, smithing, AND enchantment perks. If I am a mage using destruction magic, I max out my damage simply by taking the destruction tree. So the one handed warrior has used far more of his limited number of perks increasing his damage than the mage has. The mage essentially has free perks to use on other stuff that the warrior would be unable to get. Let's say he decides to use those free perks to take enough conjuration perks to get two dremora lords and a bunch of powerful illusion spells. Is the mage really less powerful in this case? I'm not sure. Maybe, but it's certainly close.
So yes, destruction does less damage, but you don't have to put many perks in it. There's a bit of merit in that, I think, and it is CERTAINLY unfair to compare a perked out warrior's damage with a mage's destruction damage when he has used 1/3 of the perks of the warrior.