The game is easy enough however that if you want to go for looks and like the idea of wearing whatever/whenever you want (like I, mpls Big Daddy, lolzz and others do) it can certainly be done.
This.
I used to specialize with my characters, but I really don't see much of a need, and I'm tired of the default sets. They'll generally end up with sufficient armor skill eventually anyway, and anything they're short, they can make up with enchantments. Plus I almost never play characters who just stand there and get hit - I tend toward lightly armored stealth characters, and they're better off moving around a lot and avoiding attacks rather than just standing there and letting the armor soak them up.
My basic starting gear for an armored character is leather boots, iron greaves, iron gauntlets and... some sort of cuirass (or maybe even no cuirass). The cuirass (or lack thereof) just depends on the character, the skill they most want to increase and the look they want. And from there, there's no telling which way they'll go. Most often toward light or no armor, but they'll still mix in a heavy piece if it looks good. I tend to avoid heavy cuirasses, just because they're so bulky and so heavy, but heavy greaves and gauntlets are often nice, and sometimes even heavy boots once their sneak skill gets high enough that the weight doesn't matter any more.
That said though, from a purely practical standpoint, yes - you're best off to pick one or the other and stick with it.