I guess my real concern is that your adversaries will just hang around your base waiting for you to show up, so that they can launch an attack directly on you. Thought this was ludicrous in Skyrim, I'd just led the giant stand around using his club as a back scratch tool, and get around to killing him when I grew bored with him. If you ran away, the bandits would just hang around until you got back, and they'd ignore your steward totally.
My ideal in fallout would be a random scenario in which that bandits/raiders/super mutants , could attack at anytime, and you would have someone left in charge (possibly a companion) to maintain defenses. If it wasn't working out well, then you'd get a mayday on your pip boy. You'd either come running back promptly, or let fate decide. Consequences!
I don't see your settlement getting too thrashed, just maybe the power grid and armaments destroyed,/ and non essential NPCs either killed or running away never to return... It will be interesting to see how this is balanced... and if it's realistic and quasi believable. Probably asking for too much...