With perks, light armor lets your character move faster and crit more often PLUS decreases the likelihood that an enemy crits you. I don't know about you, but an enemy crit is one of the most dangerous obstacles for me; well worth losing the extra DT just to avoid those. Normal clothing can offer faster movement and ocassionally some nice stat bonuses (+1 Charisma, endurance or agility. Don't underestimate any of those; Endurance is self-explanatory, CH improves your followers and agility improves your reload) but doesn't provide any good DT. Medium armor provides slightly more DT, but nothing more than that. Heavy armor provides even MORE DT, perhaps some STR, and, through a perk, more resistance to the seldomly used energy weapons (Fiends are the only EW enemy that one can expect to constantly run into, and they're hardly dangerous). This is usually all at the cost of -1 to agility, a huge weakness to Pulse weapons and again, you move slower.
So sell me on this: why would anyone pick anything but Light Armor? I get that not everyone is going to invest two perk slots into LA and sometimes people do just seek pure DT, but my point is that if someone is truly looking to have the best possible armor and the best possible defense and they're willing to invest in those perks, HA, clothing and MA really don't have anything going for them. Shouldn't they get some perks too?