It seems to me that this particular skill is the hardest one for which to lose all the perks at once. You're suddenly down to single damage for your spells, at full cost. You retain your high Magicka pool, so you can at least cast a decent number of spells, but the enemies have more hitpoints, so you need them!
Does anyone know a strategy for using other skills to achieve some kind of synergy with Destruction, the way that Smithing a better weapon compensates for the lost damage multipliers when you make one- or two-handed legendary? The only thing that seems to help is Restoration's boosts for regeneration.