I received no warning about Covenant being under attack until I got the update saying that it failed, I reloaded to my previous save and used fast travel to get there. Covenant was under attack by Super Mutants and all NPCs were hostile and being shot by the turrets that I placed, once I killed the Super Mutants and got the objective completed, all the NPCs became friendly again, but all the named NPCs were stuck with a extremely happy facial expression and none of them had voiced dialogue anymore and they were stuck in an eternal loop of the "Thanks for helping" conversation, but only my lines were voiced. I went somewhere else, waited, then returned and they were still the same. I reloaded to the same save file as before and let the attack fail then teleported there and everything was normal, the super mutants didn't destroy anything, no settlers were dead.

Failing to defend had no negative impact on anything at all, while defending it pretty much broke the named NPCs there, the rest of the settlers still had voices.