Chimer, Ayleid, and Altmer all looked genetypically Aldmeri.
Also, would you consider the Ayleids Altmeri or Aldmeri? Ayleid is just the word for the civilization and it's denizens, not a species name.
Those things are one in the same.
Seeing as no one has ever seen a Chimer or Ayleid in a TES game ever before, I really don't know how you can actually draw those conclusions. But anyway, according to the texts, both races did look different when compared to Altmer.
He demonstrated the right way to wear their skin.
-http://www.imperial-library.info/mwbooks/changed_ones.shtml
she blinked, and the bird was gone and in its place, a naked Elf man stood, not as dark as a Dunmer, but not as pale as the Altmer. She knew at once it was an Ayleid, a Wild Elf.
-http://www.imperial-library.info/obbooks/2920.shtml