I think the legitimate response to your point of "insignificant mutations" would be, "yeah, same thing for ghoul."
There are at least two types of ghouls: 1 normal, and 2 feral. There are of course several leveled variants of the ferals.
We all know that, ferals cannot be "reasoned" with. They attack on sight, and if you do not escape them or dispatch them they will kill you and any living human (anything other than other ghouls I suspect). They are monsters. Arguably they too are also "humans" but humans that have been irrevocably transformed into deadly threats to life and human life in particular.
Raiders are also humans, but much like ghouls, they cannot be reasoned with (most of the time, I suppose Charisma perks might change that).
We don't really have too many of these "monster" things in real life, things that one justifiably "kills on sight." Among the sole examples that most of humanity would agree on: a. the Nazi regime between 1939 and 1945 (although some would disagree); b. rabid animals, or wild animals that have become a verified threat (generally when a bear attacks or kills humans most conservation entities regard killing it as a reasonable preventative). c. I suppose a criminal who is demonstrating a threat of harm more-or-less belongs in this category although it is more fluid.
Ghouls are categorically NOT a confirmed threat. Many of them are quite nice and helpful. They only possible justifications for hating them would be:
1. Bigotry (which your previous claims of them being "not human") seem to reflect, or
2. Some sort of argument that they are a "walking timebomb" in the sense that they inevitably are going to go feral eventually.
My knowledge of the lore is that, there is not much basis to fear any particularly ghoul is going to go feral at any time. I'm not even sure if it happens at all, except under exceptional circumstances.