This feels terribly wrong.
My favorite type of character to play is a "Shadowknight." When you picture a Dark Knight, you picture someone wielding a hulking huge blade casting spells. For a Paladin, you picture either a sword and a shield, or someone wielding a big claymore. A Ranger, you would picture dual wielding casting nature magic.
I've never played an RPG where a combat/spellcaster had to keep a hand free to cast his spells.