So, there isn't an unarmed skill, but one-handed perks effect our fists.
Using your fists does not raise your one-handed skill.
We can up our hand to hand damage....
....but only if we invest in heavy armor.
Seriously.
They are just too damn good at this.