Just wondering has anyone else experienced this?

I don't bother doing the settlement defense missions, nor do I do any of the Preston missions. I just ignore them but if I travel to any settlement that has been attacked after the defense mission failed at least one settler will start shooting me. My companions can kill them and nothing happens. If I even attempt to kill them the whole settlement attacks me.

Anyone got any ideas apart from not going to any of these settlements?