The Dunmer are of average height.
The Orsimer although stocky and muscular are of average height
The Dwemer ghosts were of average height
The Bosmer who may have also absorbed the last of the Ayleids were of average height
The Falmer (the Betrayed) are of average height
In addition to this the Breton who are human but have absorbed some Aldmeri genes are also of average height.
Is it possible that the Altmer have in fact evolved to be taller than the original wave of Aldmer and aren't as unchanged as they would like to think?