I think that's well enough explained already: they're the longest lived mortal race, and before Skyrim, they were consistently among the healthiest. I won't get into the magic debate (re: nature or nurture), but even without it, there's good reason to believe that the Altmer have a special genetic stock worth protecting. If you managed to somehow get most reasonable people to seriously consider which mortal Tamrielic race they'd rather be, I bet most people would choose Altmer. Sure, the other races all have something truly special to them (except for the cyrodiils, AFAIK), such as a redguard's special adrenal glands, or an argonians ability to breathe water, but I think most people consider living drastically longer to be more important than high-performance athletics or underwater exploration.

Racial purity is totally ridiculous in our world because there's no reward to it. Variation between "races" is less than that within "races", et cetera. But in the fantasy world of Nirn, that just isn't the case. Altmer really do have some awesome inherited traits.

It's all the mannish wives they breed with. Keeps them pure.

In the imperial librairy on the redguard myths there are story about how goblins came from another dimension, be it another continent, a daedric realm, or the twilight zone

STOP RIGHT THERE CRIMINAL SCUM!

Who'd want to be a [censored] Altmer? Right now, the Thalmor is looming over Summerset, there is (or was) an caste system similar to India, the eugenics (these are bad primarily for ethical, not scientific, reasons), and a bunch of other stuff I forget about.

There is an Altmer in Redguard that says the community treats him better than the one back in Summerset, because shopkeepers are considered valuable in Hammerfell.

Well, maybe being an Altmer would be good if you don't live on Summerset.