That's the thing... I want to be a full-on mage, not a battle-mage

I and should i really need to spend loads of points and time on enchanting to have fun

Plus, you're much better off going 1H + Heavy, 2H + Heavy, or Archery + Heavy than you are Destruction + Heavy. Sure, you can get "free" destruction spells if you enchant your armor like that, but archery and melee are also free for regular attacks and have these advantages:
- Gain +100% damage from perks, rather than +50%
- Gain damage for raising your associated combat skill. Spells gain no extra damage.
- Weapons can be smithed to be made significantly stronger.
- You have, I believe,
four equipment slots that can receive the +archer/1H/2H/whatever damage enchant. At +40% damage each, you're looking at +160% damage on top of skill, perks, and smithing. And that's before you start talking about 100 enchanting where you can add 2 enchants per item or abusing enchanting/alchemy. Spells have no such enchants.
Destruction is fun, at first, but it really starts falling behind. Yes,
as a mage you cannot use just Destruction, but it
IS the offensive tree for casters, just as 1H/2H/Archery are for the warrior types. It'd be pretty lame if Block/Shields got all the damage buffs which made shields do significantly more damage than swords or bows past L30, but that's kinda where mages sit currently.