The only time I've had this happen is when the attack is by Synths.
For instance, I've had the Castle get attacked a few times. (It's defense is almost 3 times the food/water total...which is incredibly annoying...I guess it's programmed to be attacked at least occasionally...) The attacks always come from the road leading into the gate, have been either Super Mutants, Gunners, or Raiders, and I have multiple guard posts there and various turrets all along the wall (including several missile turrets.) Every time, all of the attackers are dead well before they even get close to the buildable area of the Castle. Mission has never failed.
However, in three instances now with Synth attackers (once at Sanctuary, twice at Abernathy,) the mission has failed despite all of the Synths being killed. I've even resisted fighting back myself and shut down the generators powering some of my turrets just to ensure that a few of the Synths make it into the buildable area. In every instance, despite all of the Synths being killed, the mission fails.
I've found the following solution to get around it:
I always get the message "defend X settlement" after I fast travel it seems. When I get this, I reload the autosave from right before I fast traveled, and then go directly to the settlement in need of defense. This prevents the mission from appearing, but the attack still occurs. Kill the Synths. Talk to a Settler to verify (you should get the "thanks for helping" dialogue.)
Annoying, but effective so far.