Why would they be worried about being overran by ferals? Shouldn't they be more concerned about Ferals attacking caravans rather than being overran? Would Ferals even overrun their community?

Well look at it this way: if the other ghouls weren't actually feral, but merely a rival group, then could the Slog be overrun?

Feral's seem way different in FO4 as compared to FO3 they are basically mindless killing machines who will attack anything