I did a settlement experiment and set a few of them up with 600+ defenses, building a perimeter with heavy gun turrets and putting some up high in the center, like heavy laser and rockets along with inaccessible generators to power them.
I prefer to actually fight the battles but the game doesn't give me the chance most of the time. I either don't get the alert or miss it since it is a micro-second long popup box. Sure enough and soon enough on one of my experimental settlements, I get the dreaded "Abernathy defense mission failed" popup. I must have missed the mission quest on that one. When I went there, despite massive amounts of defense, I got the same result as having little to no defense - every single turret broken, every single generator broken, every single water pump broken, every single crop burned. Having a large defense is not only useless in defending a settlement for auto-resolved battles, it creates a large chore to repair everything when it inevitably gets 100% torched.
Another buggy behavior I notice is that the "break script" or whatever breaks all your equipment and crops seems to be lazy. Sometimes I arrive and everything is working even though the pipboy says 0 defense, 0 water, etc... Then as I do the rounds on the perimeter, I notice equipment I had checked a moment ago and was working, does not work now. Worse, I notice as I go around the perimeter repairing everything, when I get back to where I started some of the turrets are broken again! I think the "break script" is lazy and kind of continues on in the background even while you are fixing things.
Considering when I do get the opportunity to defend a settlement and fast travel there, the attackers spawn inside it in the same place every time, a robust set of defenses seems a waste.