Personally, i imagine the fighting technique and style of a one-handed swordsman would be closer to that of a two-handed swordsman than that of a one-handed axeman (i hope that makes sense).
I'd prefer to choose sword, blunt or axe as a skill then have one or two handed as a perk.