It makes cosmic sense when you think about it.
Basically like this, if I've got my stuff straight: Nirn is created, Ehlnofey and Hist populate the land. Tamriel is like some mythical mega-continent. Ehlnofey break into Aldmer and Men. Dawn Era ends with Tamriel split across the oceans (and the Hist are said to have mostly sunk under the waters in this time of awful fighting, only remaining in the swamps of Black Marsh). Left-handed Elves and proto-Redguards end up on Yokuda. Some Men leave for Atmora, others stay in Cyrodiil (the Nedes, et al). The Aldmer, spreading out from the Summerset Isles, become Ayleids, Bosmer, Orsimer and Velothi (Chimer, later Dunmer), and those remaining in Summerset become Altmer. The Dwemer split at some unknown point in the far past.
Atmoran Men return to Tamriel, some split off and mix with Altmer in High Rock becoming the Bretons, others become the Nords.
Yokuda is destroyed, Redguards return to Tamriel.
Bosmer and Khajiit come from the same aldmeri forest-people, with Khajiit taking their distinctive form because of a blessing by Azura or somesuch. (not entirely sure on either point)
I think that's everybody.
edit: Oh, the Argonians. Produced by the Hist, the only sentient race of Tamriel that isn't descended from the Ehlnofey.
Pretty sure giants trace back to the Ehlnofey as well, actually.
You're saying Ogres, Goblins and Dreugh are descended from Ehlnofey? I'm 99% sure the Dreugh are not.
And, I doubt the Khajiit are (descended from) elves. I know there is evidence that they are, but there is also evidence that they are not. I'm willing to bet they are something different, like the Argonions and Dreugh. Probably related to Lilmothiit (and obviously related to Ka Po Tun)
Also, I kinda laughed when I saw the title because it seemed so bad.