It can happen in reality too. It's not that rare that you might look more like your mother than your father. But they still have the intelligence of Aldmer, so I say it's a fair trade off. Besides if they looked more like Aldmer, Altmers wouldn't exist and neither would the Thalmors.
You may only inherit certain parts of your mother, but not everything.
For example, if you have a black man and white woman breeding, the offspring will be half black and half white, like President Obama.
If two half blacks breed, the offspring may be everything between full black and full white, but it will most likely be more in-betweens (odds for full color is ~1%).
This seems not to be the case in Elder Scrolls. As the mother's phenotype is always inherited you can have a long line of Altmer women no matter who they mate with. This is odd because phenotype is just another part of your chromosomes as is everything else.
Of course the universe contradicts itself here - Bretons are in art shown to be in-between mixes but in-game you can hardly tell them from Imperials.