Magic is probably the weakest of all the archetypes. Your strongest attack only does about 200 points of damage which is far lower than the 500+ damage per swing for warriors and the thousands of damage thieves can get.
Also, as mentioned above, if you don't want to have infinite destruction spells why don't you just take off one of your enchantments?
Yeah, weapons scale out of control damage wise, but magic can infinitely chain stagger any group of enemies that are within visible range of you. Dual cast chain lightning over and over may be slow, but you don't even slightly risk any sort of danger unless
A. Enemy uses a ward which blocks the stagger effect, then you simply need to shatter their ward and stagger them before they reapply so you can get them in a stagger lock.
or
B. They're stagger immune. There's almost no enemies in the game this applies to though. Tsun is stagger immune, and maybe Alduin. Also, I think there may be a dragon priest that is stagger immune.
All archetypes have ways of breaking the game. Sure, wizards can't match the weapon broken damage, like I said, but they're much safer. Almost any "boss battle" typically involves the boss standing there repeating one animation while getting stabbed by two dremoras and then disintegrating when its HP gets low.