This is a very real concern, considering just how energetically the game tries to push the player into the MM faction at the beginning. Once having built a settlement, the settlers that live there essentially become family. Understandably, having sunk the effort into building sufficient housing, food, defenses, and Happiness-makers into a settlement, it would be a shame to just let some random Raider attack destroy the settlement's infrastructure. So when you get the message that one of your settlements is about to be attacked, you feel that you MUST drop whatever you are doing and run to the settlement's defense. Why? Because it is absolutely guaranteed to lose if you fail to appear. Granted, no settlers get killed, but a varying amount, up to ALL, infrastructure WILL be damaged and require repairs before it becomes productive again.
Now multiply by 20. Even if the probability of a settlement is low odds (say, 1-5%) every turn unit, one or more settlements will send out distress calls. Low odds per settlement adds up to a significant percentage the more settlements you have. For example, say the percentage of a settlement being attacked is 5% per time unit. For any given settlement, the probability that it will require assistance suggests that within 20 turns, it WILL most likely be needing assistance, at least once. Sometimes more often, sometimes less. BUT if you have 20 settlements, then the probability that at least one settlement will need help is almost certain EVERY TURN. For example, I currently have 24 settlement sites under my control. (Four of them are unpopulated because they are in what I consider poor locations.) Last night I was planning to go adventuring to a specific site. I fast traveled there, but when I arrived I got the message that "(Settlement ____) is under attack by _____." So I turned around and ran to the settlement's rescue. When I finished there, I fast traveled to my adventure site. AGAIN, when I arrived, I got the message that "(Settlement _____) is under attack by _____." So again I aborted my adventure and ran to there. Finished that distraction and fast traveled to my intended adventure site. YET AGAIN I arrived to the message that "(Settlement ____ is under attack by _____."
The settlement system, a central component of the game -- just like the College of Winterhold and Bards' College in Skyrim -- is getting in the way of playing the game. It doesn't help that no matter how well you fortify a settlement, if you don't run to them when called, they WILL lose. And if you are unwilling to allow settlements have the snot kicked out of them, you can't finish the Main Quest. That to my mind is shoddy game design. "Here, come do this. Now, go do that, even though everything you worked for earlier will be undone."
By the time Bethesda releases the first DLC, they MUST loosen up the settlement-building so it doesn't keep demanding so much of the player's game time. Allow settlements to be delegated to unused former Companions, or make it possible to appoint "Senior Settlers" to be responsible for a settlement's well-being. Something.