I disagree with OP completely, and I daresay I don't believe he has ever worn a full suit of armour.
Different armour styles affect your movements heavily, and thus greatly affect the combat style of the wearer. Dodging, parrying and attacking are all very different when wearing a full suit of plates as opposed to say, leather armour. If "if you've worn one suit of armour you've worn them all" sounds true to you, then I am afraid you are sorely mistaken.
I recommend you read: Hallgerd's Tale, The Armorer's Challenge
Yes, I know fully well how heavy armor restricts your movement. Which is why I think it's bloody stupid that he will be WORSE when putting on a lighter armor!
All of a sudden he is less encumbered, has less restricted movement, and can move about quicker, and he's supposed to be clueless about how to fight in it?
It works the other way around. Someone used to light armor will have trouble with heavy armor. But someone who is used to heavy, will have no problem whatsoever with lighter armor. This system is flawed, and as such, it would be better off with simply ONE armor skill, with the option to specialize for a certain style via perks.