The problem with Destruction is that compared to every other direct attack skill (Archery, One-Handed, Two-Handed), its damage does not scale as it levels. Sure, you can pick stronger spells, going from Flames to Wall of Flames or from Firebolt to Incinerate, the same way you can pick a stronger sword, from Iron through Dwarven to Daedric. Sure, you can pick a few perks to increase your spell damage, the same way the very first perk of each weapon tree eventually lets you double your damage. However, while the base damage of weapons also gets multiplied by your skill as you get progressively more experienced, the spells remain the same eternally, experience rather decreasing their mana cost.
This reduced damage output per strike is compensated by the ability, through the Impact perk, to reliably and continually stun enemies, and through enchanting to massively reduce spell costs. This leads to a very boring game of attrition where fights against tougher opponents consist of standing and spamming your strongest direct damage bolt until they keel over.
To function on par with other damage skills,
A) Destruction damage should also have increased along with the skill's level, same as weapons.
It would also have helped to invert the effect of Enchanting and Alchemy, where it would have been Enchanting which would have increased effectiveness, same as Smithing for weapons, and Alchemy which would have reduced mana costs for a time, so you could have used a potion to set yourself into spam mode for a difficult fight.
C) Finally, Impact and the elemental capstones should have been reworked. The current capstones affect enemies that are already dying, adding style but nothing more. A much more useful version would have had fire inherit Impact's stagger at a reasonable but not absolute chance, ice could have had a similar chance to freeze opponents, and lightning could have knocked back. Not guaranteed, something along the line of a 10% chance, but both more useful than the current capstones while being less abusive or monotone than Impact.
Now, this doesn't mean that Destruction, as is, is useless. After all, I am playing a pure Wizard right now. But it works better as a supplement to your other casting disciplines. Once Lydia and a Frost Atronach have charged in, once you buffed her with Rally, once you Soul trapped one of the enemies, once you shouted Marked for Death, then you snipe Firebolts just so that whatever your cohorts are fighting stays locked down and easy to beat up. It is definitely not the charge in, tear stuff up style of a sword and board fighter.
Since what you want to make actually looks more like a Battle-Mage, my best advice would be to skip Destruction altogether. Go something like...
One-Handed
Heavy Armor
Restoration
Conjuration
Smithing
Enchanting
Use Restoration wards as a shield and battle alongside your summons. Perk up One-Handed's two weapon fighting branch and summon yourself a second, soul-trapping, minion banishing sword when you want some more damage.