What are the downsides to wearing Legion armour if you're playing a Legionairre?

Obviously the NCR are always gonna be hostile, but what about other places? Do certain towns turn you away? Can you get into New Vegas?

I think New Vegas will turn you away since it is under NCR MP jurisdiction.

What about random settlements? Goodpsrings, Jacobstown, general quest givers, DLC people (Joshua Graham?)

Does wearing it kinda limit the game?

I'm pretty sure you can wander around the strip, I had a Legion companion with Legion faction and everyone ignored him. I'd save it in case though.

Companions' factions will be ignored though. For example NCR troopers will attack you if you wear Brotherhood PA, but not Veronica even if you equip her in Brotherhood PA.

You can get onto the strip via the nearby air force base if the securitrons turn you away.

Just the NCR are affected by it, since Khans are friendly same with Omertas and WGS, the Tops i'm unsure about.
All towns that are not controlled by NCR should be friendly, you might have a problem with NoVac because of Boone and Manny since killing them when they are hostile will cause the whole town to turn on you.
Kill them all anyway.

If he's dressed up in Legion armor that's probably not the safest idea.

You gotta learn to fight sometime.