In real hand to hand combat, especially against armed and armored opponents, you would realistically have more disarming moves or moves to attack joints of other weak spots in the armor. Also, in a hand to hand fight, it's not just punches and kicks. There are holds, throws, submissions, headbutts, etc.
That would go fairly well with Don't Forget This. Training in Submission, Disarming, Attacking, and KO. If animations were improved in TES V these could be variants of of 2 basic combat controls. Submission and Disarming could work off of block and Attacking/KO off of the current "flail" button (:brokencomputer: curse you OB animations)
Disarming could rip off shields, weapons, and (against H@H) throw. -Timing the move to a block or attack with less sucess further from the que. Most attempts would be a block. EDIT: speed oriented, and the most unique attack to H2H
Submission would work to immobilize, possibly similar to fallout location damage. - Endurance oriented
Attacking is general health damage, maybe poison-able. -Strength oriented
KO for general fatigue/crippling moves. -Agility oriented
While H2H would still generally be speed oriented.
While defensive moves could be executed regardless of armorment, attacking a heavily armored opponent would require some equipment or magic enhancement.