Because games are not about "challenge" particularly RPGs?
Challenge is only one of many possible elements of game design and for many games, it is unimportant or even undesired, to the point where including it actually undermines the quality of the game. People who are hung up on "challenge" for gaming need to play games where challenge is key such as time-based games (e.g., puzzles, racing, etc.) or fighting/competition types of games (e.g., sports of various kinds). Story-focused games should have little to no challenge as challenge only interferes with telling the story and presenting characters.
Having variety for spawns is most certainly pointless for many people because you are still tasked with simply killing the spawns. That's not variety, it's the same repetition people here claim they want to change. It doesn't matter if you are killing a Raider or killing a Deathclaw, you're still just fighting something to the death. Might as well be in an arena (e.g., fighting games). Same thing.
In many instances, it doesn't make sense, anyway, as the reason for certain entities to be present is due to environmental factors. You'd have to change the context for it to make sense.
We do have some variety of encounters where it makes sense to do so, as has already been mentioned.
People might also consider that doing this for everywhere is simply impossible, at least if you want a large game world. It would take enormous resources to offer such variations in encounters everywhere due to different coding and assets for each variation. Not practical... but people could try to do it with their own game and discover this on their own, of course.
Why not assume that Bethesda, as an experienced team of senior developers, has tried to do what they can within modern technical limitations and stuff that isn't there simply could not be done? Makes a whole lot more sense, really.