Most of the whines about destruction magic on this forum, that I've read, came from it not compare to the damage you get if you exploit smithing and enchanting, through which it's possible to craft items which are five to six times better than what you'll find and then enchant their damage % to insane heights making it possible to one or two hit everything except bosses and never take any damage. Technically you can also exploit smithing a bit as a mage, making your spells cost nothing or next to it, but since there isn't a % damage enchant for destruction your spells will stop scaling before an axe or bow would.
Of course playing like that is pretty much the equivalent of using a God mode cheat and while some people think that's awesome I found it pretty [censored] boring.

Magic is fine though, a character that only does destruction magic isn't going to do very well. But then again neither would a melee ignoring armor/smithing or a thief ignoring sneak. So if you level up a few different schools of magic and actually use it you can play the game just fine.
It'll never deal the same single target damage as a 2handed mace in the face, but on the other hand you'll be the king of AoE and it involves more than just mindlessly bashing away until you win.
+ it's a singleplayer game, who cares if it does 20dps less if it's more fun?