It seems sometimes the spawn point is inside the perimeter. The obvious example is in Covenant, a settlement heavily fortified with only one exit. I was attacked twice by raiders while I was sleeping in the inn ??. The others seem to spawn just outside the exit and to the right.

Sometimes the spawn point of enemies is inside the constructable zone ( In Starlight dive-in is just near the tower, so it is possible that you have build in there XD).

In other post they seem to clarify that your settlement is "safe" when your defense is more than your water+food. It worked for me but i prefer to be attacked once in a while so I left some settlements "unprotected".