Remove all the original crops and replace like for like, not sure what the issue is but that fixed it for me.
Cats are evil. This is well known. There is nothing I won't blame on cats (except the Holocaust; even then, I'll bet many prominent Nazis also liked cats).
Weird.... I had all my settlers randomly unassign then not want to assign when prompted.
Then when I fast travelled/walked back to try and fix it derped and only the trade stands/defense people were being wonky.
I reloaded a save like 10 mins earlier and turns out it was just a bug. Think it was either from building or assigning someone to the defense I built when it was done.