I never said once that I have evidence behind what I was saying. It is just a theory that I have. The Ayleids came from another continent and were beat by the rising Empire. In Oblivion they said that the surviving Ayleids feld back to their home after they were defeated.
I think you're confusing the Ayleids and the Akaviri.
The Ayleids homeland was Cyrodiil, the Imperial City was their capitol and they're called the Heartland High Elves for a reason. Once Alessia defeated them they fled into Valenwood where they merged with the local population and remote parts of Cyrodiil where they became basis for myths about the wild elves.
The Akaviri came from the continent of Akavir, east of Tamriel, and were defeated by emperor Reman Cyrodiil, founder of the second empire of Cyrodiil. Most Akaviri invaders were killed or fled but some pledged their allegiance to Reman, founding organisations such as the Blades, later becoming the Akaviri Potentates, trusted adivisors and stewards of the Reman dynasty who were finally assassinated mid second era.