1. The Dunmer only had their skin and eyes changed. Not their height.
2. The Altmer are not necessarily the closest approximation of the Aldmer. They try to approximate an ideal but as it goes with ideals, they have no negative qualities. So where some Aldmer were exceptionally tall, some were exceptionally pretty, some were exceptionally good at magic, the ideal Aldmer has all of those.
3. Genetics are somewhat irrelevant in Tamriel as it's a fantasy world where the mothers traits are most dominant.
4. You don't know the height of the Nedes.
5. Other factors then genetics influence height.
So all you could conclude is that the Nedes might have been as tall as the Bretons but not the other way around.
And some Aldmer were exceptionally buff. And look wha happened. *sigh*
http://julia.hanovercomputer.com/toy/nov01/catalog/Uncle%20Sam/pict0082.jpg