Sometimes a settler is a synth and they attack. That might be the origin of the raids spawning inside the base myth if it's not accurate.
Every attack I've observed has been from a predictable location. Usually it's red Rocket (Mostly super mutants), they attack from the south, and they die before they get under the firing ark of the missile turret I placed on the roof (admittedly it's a tossup if the missile turret does them in or the machine gun turrets at ground level).
Also, on a previous playthrough Covenant seemed to be under nearly constant siege by raiders crouching behind the low fence on the far side of the road trying desperately to stay out of the fire of the line of turrets I squeezed between the wall and the edge of the building boundary. I'm not sure if it's relevant but that's also where I stashed the bodies of the covenant dwellers after I moved in (so the pile of bodies often got quite large before the raider corpses got cleaned up on reload)