Hand to hand should definitely stay in the game and should be a viable skill to use agains all and any adversary (if it is not, then they should not even bother with implementing it).
H2H bare-handed should not kill an attacking marauder. Ever. A disabled, stunned, or otherwise unconcious marauder? Sure. And H2H should be able to create those situations. But it should never kill an armored opponent on his feet. It doesn't make sense.
When you fight anarmed, you should still be able to augment the damage output by special items, likek nuckles, katars and so on for regular damage, or special "combat rings" for dealing magic damage.
and these situations are valid and may alter the equation some. Knuckles can be a damage enhancer for unarmored targets, and otherwise simply increase the chance of a debilitating blow being delivered. That's fine. But they should also not kill an armored opponent on his feet. Punching a Daedric armored swordsman to death FEELS silly when he never even falters, then just drops dead. Of course, with magic damage on punch, I could see it happening and NOT feeling quite so silly. Likewise, something that involves a puncturing mechanism (punch dagger, etc), or simply a massive amount of weight with a small surface area... they also can sensibly damage through armor. I have no objection to that. But please, no bare-fisted death to a Dremora Valkynaz.
Unarmed comabt should not be reduced to boxing, but should include things like kicking, or grabbing your opponent and tossing him, or grabbing things, like barrels of crates and throwing them on your adversary to cause damage.
Kicking is about all you could reasonably expect to see in the near future, honestly. Throwing part of the background requires quite a bit more control than the grab button allows for, and more of the world having physics and collision. Read: not going to happen any time soon. Why? Daggerfall included a "put me back on the floor" key for a reason. Morrowind thought it was clear (but yet I still plunge through Vivec every time I visit at least once). Oblivion got the player to stay in level. too bad anything else can fly through walls just often enough to be annoying (mainly weapons, though I've lost whole corpses) You get the picture, right? The collision detection isn't anywhere close to perfect, which is what tossing more crap around demands.
If you are using your bare arms, you should also be able to block attacks. It sure is not a good idea to block a claymore with your forearms as you did in Oblivion, but if you do not place your forarms in teh way fo the blade, but instead you try to catch or block on your opponents hands, you can avoid quite a bit of damage.
The window where that actually works is so extremely narrow that it's not worth spending the money on animating that possibility. If you include it, H2H fans will claim it never works if you actually calculate it based on the blade length and speed. If you satisfy the H2H fans, swordsmen will have a much more legitimate complaint that they hit a random townie in the head with a claymore. THEN he blocked it. There is something to be said for weapons that are longer than the human arm after all.