I was assigned coastal cottage as the mercer safehouse and after clearing it out when I go to use the workshop it tells me to clear all enemies in the area - they are clear.

I used killall in console and that didn't help. I traveled around with tcl and couldn't find any enemies. I've gone back and retried it from an old save and same issue.

Only thing I haven't done is go back to save before picking it up and try and get a different settlement (this is because I spent hours and hours building two settlements in between picking the quest up and going to do it. At this point, my game will be stuck as I can't do anymore RR quests.

Not sure how to force it to complete with console since it's a random quest, there are no bug lists in the wikia about it or setstage commands. Does anyone know how I might be able to force it forward with the console? Or any other fix?

Either way load a previous save and get another radiant quest from Preston,

OR

if you are already in that quest: "Taking Point" radiant quest -> Location: Coastal cottage

use the console command: setstage <400> to force the quest to skip to the last objective, and then go to talk to Preston Garvey.

Also killall can be used to kill all enemies, however this command includes any regular non-essential NPC that may be in within range as well. So don't use that!

I've kept a save before using anything like killall so far, so I can go back to before various console commands.

It's a quest issued from PAM in the Railroad instead of Preston, and to go back and change it I have to go back to a save before turning in the Butcher Bill quest and like I said, I literally spent all day building a settlement and would lose that by going back to an old save. Plus, the fact that it's a RR quest means I can't move forward on that entire questline at this point either.

I'll try that console command and see if it works for this quest I guess.

I went back to an older save just to clear the settlement without being assigned and still get the 'must kill all enemies in area' even though all enemies are dead. I guess this is a broken settlement in my game now.

OK, I was able to use a similar console command you suggested to set the stage to the next phase (opening up the workshop) and then enter god mode and build defenses enough to get the Mercer Safehouse NPC to show up and trigger completion. I still can't use the settlement, but at least it's a railroad safehouse and will have some purpose and I can advance the RR questline.