This idea is less stupid than the OP.
My ego thanks you.
Personally I don't have a problem if non-mage/non-enchanters have a slight disadvantage in recharging their weapons as long as they can do it. I tried a non-magic using Redguard in OB who would only use enchanted items if he found/bought them. Didn't even use the healing spell that you start with, only potions as he saw alchemy as using healing herbs etc.... But I willingly excepted the challenges because I imposed the restrictions on myself.
I don't begrudge a stealth character who can sneak thru a dungeon and take out the main badguy at the end with an assassination strike (like in the trailer) because my warrior has to slug his way thru in his clunky armor. I chose that character so I deal.
My idea, which you thought wasn't as stupid as the other guys, just tries to differentiate an enchanter from everybody else thru perks. Which I think is the reason to have perks, to broaden customizability. Every playstyle has it's advantages and disadvantages. And if you don't wish to dabble in enchanting for RP reasons then you just have to face up the this particular disadvantage.
I'm fine with MW style recharging and if this is how Skyrim works I will be happy. Just thowing out ideas as the OP suggested.