I found it freakishly difficult as a mage with no destruction. One-Handed/Restoration char was trying at the beginning, easy in higher levels.
I think the difficulty is fine. If you want a challenge, the highest difficulty works quite well for that.

http://www.gamesas.com/index.php?/topic/1286170-skyrim-way-too-fun/