Im on the mission where you infiltrate Caesers Legions fort.

Everything is fine, im walking through the camp, and constantly for no reason someone shouts "Youll pay for that", then i get [censored].

So what has caused this damn glitch and how can i switch it off?

What mission now? I don't remember infiltrating the fort at all.

What's your reputation with the legion like? If they dont like you or it's at all negative then yes, they'll attack you on site. This also can happen even if you're wearing legion outfits, for some reason.
I found that even the mark of caesar didnt protect me once I'd started pissing off the legion.
If you're up there to get the chip/find benny/do something for Mr House, then going on a killing spree up there might be your only option. It's hard, but there's some good loot. Take Boone, he loves killing legion soldiers..