A Sword & Board fighter gets constant damage upgrades throughout the game from One-Handed Perks, Sword Perks, better material weapons, improved quality weapons and Enchanting (which includes adding elemental damage to weapons and a straight percentage increase to skills). And i'm not even taking crafting exploits into account. A mage gets exactly 50% increased damage to each element from all Perks and has to chug potions to get beyond that. Further, it doesn't cost anything to go into combat with a Sword & Board unless you choose to use a Power Attack, while every single thing a mage does costs Magicka from a limited pool that has a horribly slow in-combat recharge rate.
Should a Destruction Mage be able to one-hit Dragon Priests? No. But they shouldn't have to stun-lock and whittle away at something a physical damage character can take down in a few seconds, either.