Morian Zenas entered all realms although he is supposedly a very powerful mage. As well as "Visitors to this isle have historically come mainly from the Dunmer of eastern Morrowind (as Azura is one of the Dunmeri three Good Daedra) and the catfolk of Elsweyr, whose people both hold a great affection for the mother of immanence, though by separate roads. "
Morian Zenas also went bat [censored] crazy after it all.
Remember, they're visitors, not residences, and it doesn't give a description of how many people go there, other than those two people go there more than other races. Daedra don't just leave doors wide open for people, even Sheo's door was only there for a champion to serve his needs. They keep the doors to their realms pretty closed, unless they want to invite some people inside, you are that powerful enough to break through the barriers, you are part of a witch coven, or you are a psijiic.