If you do tasks or attack a faction while wearing faction armor your faction will go up or down and remain that way so long as you keep wearing the armor. Take the armor off and put it on again and your faction will reset to neutral/hated depending on that factions relationship with the faction of the armor you're wearing. However if you take the armor off again your faction will reset to the level based on your actions.

For example if you do quests for the NCR while wearing NCR armor your faction with them will go up. If you take the armor off and put it on again your faction will return to neutral. As there are some quests that require that npc have a high diposition towards you you may have to take off your armor if you've been reset to neutral in order to get the quest.