The upshot of having 'civilian' clothing equipped could be a further reduction in hostility from strangers. For example, when entering a new town, your character walks in with his weapons drawn, and is quickly swarmed by locals bearing pitchforks. Your character walks into the same town with his weapons sheathed and he'll get a coo, wary response from villagers, with some villagers maybe even a little aggressive or hostil. But if he were to walk into the same village in a tunic and trousers, with his weapons stowed away, then he would be greeted almost warmly from everyone. The downside would be that outside of the villages and towns, he would almost always be attacked by opportunistic highwaymen, giving you a reason to switch back into armour.
What do you think?