About to hit level 36 and I am currently doing the warrior thing, shield, heavy dragon armor, 1h sword. I was thinking about switching over to mage but was wondering if that would really gimp my character once I get past 40's? Any thoughts?

Just pick conjuration and summon a dremora lord or other summons, bind it to your shield hand. Or go illusion and pickup spells like frenzy... basically anything that could benefit your current setup, if you really want destruction magic I would go with frost since it slows the enemy, you could even prepare a frost ward/trap pre-combat on the ffloor. Or you could go for restoration since it also has a magicka regen perk.

You mean stamina regen perk.

No, he probably means magicka regen perk. Restoration doesn't even have a stamina regen perk.