There ARE ways to increase destruction magic damage. You can make or buy potions that can almost double the damage output. If you poison a dagger with weakness to magic, there is another significant damage increase. Paralyze your opponent for a few seconds, poison them with weakness to fire/frost/lightning, down a potion to increase damage, and blow that SOB apart! Sure it takes a little THOUGHT and PLANNING but you are playing a freaking MAGE!! You are supposed to use your head!! You are not some blood thirsty orc charging into the fray with reckless abandon! Outsmart your opponent! I think there are enough ways to obtain these potions and poisons so that when you do come across a tougher than average opponent you will be well prepared.
Also, if you are leveling skills outside of your mage playstyle simply to level up more you will probably end up gimping yourself and spreading yourself too thin while the enemies continue to scale up. And don't discount the usefulness of a nice staff! When you run out of magicka and need those last few hits to finish off your opponent this can be a life saver. You should not have any problem keeping it charged either.