Yeah, I'm to the point where I move or "eliminate" settlers that can't be move from smaller settlements just because of this. I only focus on the big ones. It's sad that it has come to this because I like the settlement idea overall. Hopefully Bethesda takes notices. There's no reason for having your settlements thrashed if you're not around if you have more than enough defenses.