I have not had any spawns inside my bases but have watch raiders enter my settlements in the gaps of my wall that i thought were narrow enough but were not actually . I have spawned just outside a settlement in the middle of a battle when i fast travelled to a settlement under attack and nearly died. That happened ONCE. I simply used the resources > Misc. > fast travel point to a position away from the boarders that is safe and secure for all my bases now. I have lost a settler or two from attacks but i have never lost a settlement. But I "waste" perks on things like salvage and getting more bottle caps from loot and better deals from vendors so I can BUY and salvage lots of resources to beef up my defences. And there are mods or tricks/exploits for consoles that allow you to build past your settlement build size. So great defences are more than possible. I think it is worth the effort to get the cash to buy lots of shipments to have the resources to build really advance settlements. So for me the perk investment is worth it. Am I as powerful in combat as i could be? No but I am already a 'death dealer' so its not an issue.
I get that things can be frustrating but gaps in the wall can and do allow enemies to enter. Little things like having enough guards on duty so ALL entrances have settlers guarding them at all times is critically important. Using the Alarm tower, allows guards to summon settlers to the danger also helps in defence. Every guard post should have an alarm tower stationed beside it. Sure it cost one power each and seems pointless but it actually helps with defence, I have seen settlers use it. The alarm sounds and settlers come to defend the area. Larger settlements mean it takes a longer time for people to respond but it does work.
My favourite Building 'block' is the shack foundation as you can use it to create multiple story concrete walls that seals off the outside AND it can be placed in the water. Using the stairs from structures> wood > floors allows you to 'snap' in a foundation on top of an existing foundation allowing for some great defensive measures. You can in turn use this defensive wall as part of a building. Much more efficient than the junk fence system of creating a defensive wall. The first block getting it to line up straight to the settlement boarder can be tricky but it is worth the effort.
I also have found that creating supply lines is very helpful with defence, as it allows you to drop off all your junk at established workbenches and simply make the first settler of a settlement part of your supply train. So long as All your settlements are connected you have access to ALL the items in ALL your workbenches for construction. you can't remove items from the workbenches unless you are at the workbench that item is in but consume it to build something? No worries. I have one settlement, sanctuary creating all my water needs, and the other settlements don't die of thirst because my supply lines allow settlements access to my water in sanctuary. So you don't need your first settler in a settlement creating food so long as you have a surplus within your supply lines. Sending that first settler to establish a supply line to a new settlement really allows you to beef up its defences early as you can get people on guard duty faster as well. And build up a based using all of the materials you have collected so far.
The key is establishing specialized settlements. Sanctuary can be a great place to get lots of industrial sized water purification plants in the water and it has huge areas to plant food. you don't need anyone to be producing food at the hangman's alley or Jamaica plains if you set up supply lines and have other more established settlements supplying all your water and food needs for these locations which means you can assign these people to guard duty and scavenging duties right away. Which makes the next settlement easier to build because you have two urban settlements getting lots of scavenged supplies each day. Both of these settlements have small building zones and limited dirt areas so being able to support large populations without having to grow food or place down water pumps in the settlements themselves is more than a little helpful.
If you are having problems with your settlements ask players for tips on how they solved some of these issues. I am not sure settlements need to be "fixed" as I have yet to experience any issues that the settlement mechanics doesn't have a fix for in it's tool box. I think it would be nice to have the ability to manage settlers easier and to be able to look at any given settler and see what job/duty it is assigned but as for the system being "broken?" I just haven't seen that.