The problem is that the game doesn't reward you at all for using 1h + spell. It's basically the most challenging combination you can pick. And unless you plan on powerleveling enchanting, you have to split your stat allocations and put some into Mana, which dents your HP pool.
1H+paralyze, 1H+fear, 1H+frenzy are quite possibly the least challenging combinations in the game.
Of course you lose a great deal by offhanding a destruction spell. No surprise here right? For this to be really viable we'd need more high-tier spells that could be channeled in the ice/lightning trees. As it stands only the novice spells channel. There's wall, but that doesn't channel in the way I'm discussing.
For OP, if you want to battlemage think ahead to paralysis; alteration has more tools than stoneskin. Mage armor keeps up unless you're a smith.