One must not only consider the protection granted to the wearer by the armour itself, but the overall advantages different armour styles grant the user. It should be obvious that a light armour user will be terribly awkward, unused to fighting, and perhaps not even strong enough to function in a heavier armour (anyone who has worn a full suit of plates before will confirm this).
On the other hand, imagine a heavy armour user going to light. How does a light armour user's fighting style differ from a heavy armour wearer's style? You must remember that part of the advantage of light armour is not only the (relatively little) protection it offers, but also in the maneuverability it allows. Thus, a light wearer will not rely primarily on his armour for protection against sword and axe blows, but rather attempt to evade such attacks, while hoping the leather will protect him in case his evasive moves fail him. What about a heavy user? He rarely dodges - he usually can't. But that's perfectly fine for him, because he relies on his armour for protection against brute force. Thus, when a heavy armour user goes into battle wearing light, how adept do you think he will be at dodging attacks compared to the nimble light user? Sure any warrior can dodge attacks, but the heavy user will be nowhere near as good as the light user, who dodges constantly.
Thus, in summary, light armour users rely on the maneuverability + little protection of light armour for defense, whereas heavy armour users rely on the pure protection of armour for their defense. Both are quite different techniques, and if you think armour is armour then I'm afraid you are sorely mistaken.