there are adventurers there, you see them at the beggining attacking the gatekeeper, also the issue is covered when you activate xedilian
and as for oblivion what adventurer would survive accept you
yeah, that combined with chats with the inhabitants of Passwall led me to believe that Lord Sheogorath has a system for readying people to enter into his realm. I'm sure this doesn't mean that he can't create new inhabitants on a whim, but i think he much more enjoys seeing the new unexpected insanities that are produced by people being brought in. Additionally, if events in Crucible are any indication, it seems that the inhabitants of the Isles have a rather high mortality rate, certainly higher than the birth rate (if one existed), which would also indicate that new people are actively being added as time goes on, not a simple full repopulation
Spoiler following every greymarch
(if mr. Chemist insists on using spoiler tags on this i suppose i won't ruin it, but i hazard to guess that anyone reading the lore forum is already knowledgeable about Sheogorath's life cycle).