By not letting people in, they keep not only their race pure, they keep their organization as well.
Keep in mind, in the Enclave's case its not really about race, at all. That's a term that doesn't really apply here. The Enclave makes no distinction between human races. In the Enclave's view, they are supporting the continuity of the human
species. In this way, its about keeping their population healthy and genetically free of defect.
Anyway back on topic.
In my opinion, I agree with what Mr. Enclave said. The Brotherhood at this point has proven itself to be an organization which runs counter-productive to current progress.
Now as for exterminating them, is it justified? That depends on what you mean by the term. Is it morally right? I don't think so. But its justified in the sense if you look at it from the view of an organization which has been troubled by the Brotherhood in the past. IE. the NCR. The NCR is at war with the Brotherhood and thus a destruction of their bunker would be a part of the Brotherhood-NCR war and thus a justified military attack.