not hard at all, Daggerfall had it, so did skyrim, with the Werewolf transformation. Line of sight. do they SEE you transforming back to human? if so, attack, if not, you are fine. in this case, if they SEE you changing cloths, then nothing happens, or maybe they ask why you are wearing it.
and like i said, wearing Caesars arrmor should make the LEGION attack you, and the NCR wonder why the Fudge yo uare wearing it.
by your logic, no one in the NCR would have attacked someone wearing it at all, because the logic "they do not know what he looks like" works both ways, if they have seen him, they would know it is something only he wears, if not, they would not reconize it as legion at all.
Great Khan and Kings and similar armor is mostly just regular cloths with a sign, or something else, yet everyone reacts to that as well with whatever their thing is to that faction, even when it is entirely possible to have killed them, liked the looks of it (not realizing the reactions of people, because hey, its a coat/shorts/whatever.
raiders armor have zero effect, yet your logic would dictate they would be shot on sight as well.