If you're playing the Vanilla game, pure destruction mages tend to be underpowered at the lower levels and at the higher levels.
You'll need to supplement your magic with either weapons (bow/melee) or conjuration/illusion at the start. Rage/frenzy, or fear/turn and familiars can really help you tilt the balance until your destruction skills start to round out. I find bow + magic is a very good combination as Bows not only have range, but they take 2 hands which means if you unequip a bow (hit the hotkey while it's equipped), you'll bag the bow and have your spells in your hands. IE, twice as fast as cycling through your spells one key at a time.
If you're talking higher levels, you're looking at long casting bouts with dual cast/impact for low damage unless you get rebalancing mods like Destruction scaling mods (highly advised at higher levels).
Strategically as a mage, I tend to try to get Lydia asap (can't be killed), and send pets in early. Getting Frost Atronach is a priority as it's the beefiest of the summons until you can get Dremora Lord. I also tend to start fights by casting summoned creatures on top of enemies to start the fights. Also, make smart use of traps, doors, etc. Also, don't under rate flame cloak when you can get it. I've won a LOT of fights by simply casting flame cloak and running around like a headless chicken when things have gotten tough.
Use the environment to your advantage as much as possible. Hide and cast. Get the high ground (hit them from ledges). Make use of traps. For example, I'll do dungeons and lure enemies directly into traps to soften them up or kill them. I love swinging axes and spiked doors.
In terms of character progression, the first things I do is level up Health. Magicka is nice and all, but survival is your top priority in the early going. Enemies effectively do double damage, so that means that your 100 health is essentially equivalent to 50 health on Adept. Don't start putting lots of levels into Magicka until you get to around 150 or 200 health (so you can at least eat a hit or two). Look at it this way, if you can cast 5 fire bolts, how many more fire bolts does 10 magicka get you? None. Increase your survivability, wear +magicka and +health gear when you find it, then start fine tuning your character once you can live through a few hits. Put perks into Novice and Adept when appropriate to half your casting costs. Your eventual target is to get to at least 300ish health (and eventually 400+). It's at that point that you can really load up into the magicka schools.
Put simply, the mechanics of Legendary difficulty means that it's going to take some time to reach the point that you can effectively play pure destruction. As for College quests, don't worry about them so much. The fastest way to a high power mage is level your alchemy and enchanting so that you an make Fortify school gear for reduced casting costs. Doing relatively easier dungeons around Whiterun is often a better choice (for gear, ore and skill raises) than to go running off to the College (which actually has somewhat difficult quests at low levels on Legendary for a pure mage).
Also, get used to running, kiting and doing a lot of running around if you're a mage at the lower levels. If you stand still long enough to get hit, count on getting clobbered.