With the right build unarmed really isn't half bad. Beast characters have higher unarmed damage than the other races, khajitt having the highest. Plus a high enough enchanting really helps, you can put the fortify unarmed enchantment on a ring and gauntlets. Plus, the perk in the heavy armor tree. And finally the Poison Talons in the Vampire Perk tree (DG) adds 20 damage unarmed.
i have a khajitt that uses stealth unarmed combat, with necromancy and restoration. it's really a lot of fun.
Lower the difficulty a smidgen if you have to, and conjuration will really help (as to take some of the attention off of you once in a while).
and the Fists of Steel perk only affects the base armor stat. So smithing your gauntlets up won't make a difference.
Trust me, there's tons of fun to be had with unarmed combat, not to mention the coolest cinematic kills