As opposed to the normal "Killed enemy re spawns 5 feet away for your convenience" let us assume that there are no "Kill X ,Y times" quests. Without said missions, or a dependence on certain enemy types appearing regularly in an area, the devs could be a little more creative with where and how enemies spawn.
My idea, is that enemies spawn from a "stronghold" of a sort that relates to their type. IE, radscorpions spawn from a nest within a cave, while enclave troops would have a well defended camp, and raiders might take up residence in a burned out school.
While there are always enemies guarding said location, others will spawn and begin wandering the wastes, possibly even scavenging or hunting and returning the loot to their lair. Players have the ability to find and destroy or clear out these lairs for the accumulated loot gathered by the NPC's, though obviously this will be no easy task.
There are a few ways to take out a stronghold.
The option exists for a relatively easy destroying of said stronghold with proper and extremely powerful weaponry....though you won't have any loot to re-claim unless you have extremely advanced biological, chemical or radiological weapons which all have their own drawbacks.
Alternatively you can clear the structure in the classic dungeon run style of simply running through and whacking everything that moves, until it stops moving. This way, while far more difficult, is quite cheaper and more available to those without extensive resources, as well as leaving plenty of loot to be gathered up without the nasty side effects of destruction weapons.
Once a stronghold is destroyed or cleared, it will stay as such....for a time. Depending on where the stronghold is situated, if it is in a zone which players can build up (Make a base, Colonize) they can feel free to use this existing stronghold to found their own base or town. Or of course, they can simply leave it be.
The now displaced enemy type will have more spawns generating from existing spawn points, making them more difficult to eradicate...but with far more loot being generated as well. The displaced enemy type will also roam the wasteland, and eventually find a new location to build a stronghold to begin the process anew. With the stronghold created, spawns in other locations will return to normal.
Most enemy types will have a set amount of spawns and strongholds in the wastes at any given time. A few, more aggressive types however, will not.
Enemy types such as super mutants, raiders, and enclave will be able to establish footholds.
Footholds act similar to a stronghold, though with less spawns and loot. The major difference however is that there is no limit to the amount of footholds NPC's can have, and they will expand aggressively. Footholds will act as something of an outpost, or a siege camp, and usually reflect their temporary nature. If left unchecked, footholds can create a major nuisance as you suddenly have raiders attempting to plunder your town, super mutant ambushes waiting just outside the city gates, and enclave strike forces assaulting your town.
Overall, this would be a far, FAR more organic, if difficult to implement, method of NPC's and NPC interaction that would fit ever changing and evolving nature of the wasteland perfectly, and make each and every trek out into the wilderness new and exciting....not to mention dangerous.
TLDR:
Enemies spawn in bases and roam the wilderness. Bases can be destroyed or cleared for loot. Once this happens, enemies roam until they find a new location suitable for making a base. Certain expansive enemy types can create small camps to make life in the wasteland even more difficult.