A few points to make to establish my playstyle:
-I don't carry a weapon of any type.
-No stealth.
-No armor or shield.
-No followers or summons.
-No using illusion spells to make the enemies fight each other.
My character is a Breton. I used the Mage stone until I found and switched to the Lord stone.
Conclusion for those who don't want to read the following very long post: Destruction as your ONLY damage source becomes less and less viable as you level. Enchanting is a must. Bring the cost down to zero and let Impact stunlock everything and feel like you're actually a powerful mage.
Meta-conclusion: Enchanting is too powerful and too useful. Destruction without it is an exercise in gulping magicka potions like a fiend.
Things were pretty easy at first. No surprise there. I made my way early-on to the college to get things started in that department. I favor the lightning spells because I like the instant hit. I'll skip to where the story gets interesting, which was around level 15.
Fights were getting harder. A lot of things died, many of them me. Quicksave was my best friend. It was starting to take my entire Magicka bar to kill things, which was fine when facing solo enemies, not fine when facing multiple enemies. Dragons were a real chore. I developed quite the addiction to health and especially magicka potions. Magicka potions, in particular, were in short supply, so I had to turn to alchemy to keep my stocks up. I'd have to leave a dungeon part-way through to go get more. That was annoying. But then I had to stop making potions, because my alchemy skillups were starting to cause me to over-level. I spent tons of my early funds buying potions, but that also makes Speechcraft go up. Basically you can't do anything without slowly and inexorably levelling up.
Around level 20, things were getting ridiculous. Impact is a nice perk, but dual-casting (even with the appropriate half cost perk) drains magicka like crazy, and things just weren't dying. Without armor, even bandits were chopping me in half, and I was having to abuse every AI pathing trick I could to make it through dungeons. It was fun for a while, but then it just got really, really tedious.
Even having put all of my level ups into magicka, the health of the monsters was outpacing my ability to damage them, and I was a 100 hp wuss. Finally, in a random outdoor fight with a couple mages, I hit a wall. I couldn't damage them fast enough and they kept healing themselves. Even with my 50% magic resistance, their spells were devastating. I kept backing away, trying to buy time for my magicka to regenerate, and if I hadn't accidentally found my horse, I would have had to run away, tail between my legs, for the first time. Watching my horse kill both of the mages was pretty humiliating.
By itself, the main problem is the massive cost of spells compared to my magicka pool. I had to either increase the size of my pool, or decrease the cost of the spells. I'd found a couple items, and I went shopping and managed to pick up a couple more, but at my level, there just wasn't that much out there to help. A ring of +10 magicka, or an amulet of 3% cost reduction, is just a small step away from being useless.
I had to turn to Enchanting. I maxed it out. The levels I gained while doing this worried me. I ventured out with earlier versions of my final gear, and things started to become noticeably easier (despite the level gains) at around -50% cost. Once I got into the -90% range, with spells like chain lightning costing less than 10 magicka, it was like the lightswitch had been flipped from off to on.
Now I could dual-cast to stagger the enemy, and keep them nicely stunlocked until they died. Everything. Dragons, dragon priests, master vampires, master necromancers, draugr deathlords, it doesn't matter. If I can get a dual-cast spell into them, it's all over except the clicking. Once I got -100% cost, I can cast forever. Not exactly a revelation there, I know.
My character is now level 41. Destruction is long-since maxed, so I've been increasing the other mage skills. I still don't carry a weapon, use armor, use summoning, the fury/frenzy illusion line, followers, or stealth. An elder dragon landed behind me and it crouched there on the ground, stunned by the impact of my dual casted spells, until it died. 20 seconds, maybe? In a certain dungeon, I fought the final room's guardians (3 draugr deathlords and a draugr wight) all at the same time, while overburdered, with no summons, no paralyze, no nothing except destruction. Then I killed the dragon priest Hevnoraak on my way out, easy as pie.
Melee and bows may do more damage. I don't care. This isn't an MMO so it really doesn't matter. Destruction's damage, considered by itself, is fine. What is out of whack is the cost of the spells and the size of your magicka pool and the very slow speed of regen in combat. Increasing the size of your magicka pool isn't a viable solution because you simply can't increase it enough to matter. Even with 500+ magicka, tough enemies would still be alive, and lightning-using mages were a nightmare.
Whether by design or by accident, Bethesda has made enchanting, or a lot of loot-luck, necessary to use Destruction as your primary, let alone sole, damage source.
Maxing enchanting and having -100% cost also brings other issues to light. With Destruction spells costing nothing, the half-cost perks are useless. A properly-planned mage would avoid taking those perks except where they are pre-requisites for other perks.
Speaking of perks: Rune Master (cast runes 5x farther away) was very useful to me earlier on, but I never cast runes any more, at all. I was surprised there are no stronger rune versions in the game. Impact isn't very useful until you can chain-cast dual spells, and then it's overpowered. Intense Flame, Deep Freeze, and Disintegrate all seem useless. Who cares what low-health enemies are doing? Just blast them one more time and they're dead anyway. Augmented Flames/Frost/Shock are the best perks, other than Impact. +50% damage is nice, and one of the few ways to boost spell damage.
So my Destruction-only mage lives on, and is thriving, and frankly, I'm not sure there's anything left in the game that will challenge her. But without Enchanting, it would be a very different story.