Where would they go? Up north is slaver territory, Legion is on the west, Midwestern BoS is well midwest, I think... correct me if I'm wrong please. Plus most people are poor and weapons, if they have any, are in terrible condition. Hell some wastelanders have baseball bats as weapons, that won't get you far. Not sure about the south but there's probably more [censored] going down there as well.
I think people have a POV of their own cultural biases, and project that onto the people of the wastelands. For me, I pretty early on concluded that the Lone Wanderer was unique in many ways from the others he met in the Capital Wastelands - for one thing he had an education. He could read and write. Most of the people outside the settlements probably could not read and write, and would have to settle for myths about the other places that haven't been directly observed.
If I wanted to make a game of the Irish potato famine, I wouldn't make a game about the irish in New York City and what they knew. I would make a game about the Irish in Ireland and what they knew. The myth of America would probably be larger than life, filled with dreams and hopes and unrealistic aspirations. The poor farmers dying in Ireland would have a myth construct about that place that spread by word of mouth, and which grew in the telling for very human reasons. The tellers would make themselves more important, more in-the-know, and so on.
The same is probably true about the wastelanders in DC, and the myth-construct about other places are probably negative. Instead of "Go beyond the horizon, because the streets are made of gold and honey", it would be "Don't go there! The super mutants are huuuge, and they eat children!".
People have probably left DC. Lots of people have probably left DC. But in a world without communication and education, we can't know anything about them. We only have to deal with the ones that remain. The ones that remain are the ones that huddle in basemants, hiding from raiders and super mutants and whatever that's out to kill or rob or enslave them. Maybe more bombs fell on the DC area, which made the short term poisoning of the soil last longer. Maybe, instead of a halving-time of 60 years the DC area needed 120 years. Or something. It's not stated, so we can't know, and we can only speculate.
There are hints that the chaos has been going on for a long, long time. Dialogue between Sentinel Lyons and LW seem to indicate this, as when BoS came to the DC area it was totally overrun by super-mutants, and that was years ago. It's taken this long just to carve out a few safe havens for the BoS in the DC area: the Citadel, the GNR-building. Even now, the GNR is under constant attack.
But all of what I've said is just theory-crafting, because that's all we have. Much isn't stated in the games due to various constraints. But the theory-crafting I presented here is as valid as the theory-crafting that tend to knock FO3. Both are valid, and invalid at the same time. Both want to make the POV the true and real one at the exclusion of the other. It becomes silly in the end. It becomes worthless mind-games.
The truth is - both FO3 and FONV are great games that appeal to slightly different demographics and play-styles, and arguments against any of the games becomes an argument to invalidate the play-style and the preference rather than an argument against the particular game.