Racial things aside, all Altmer you see outside of Summurset are exiles and outcasts and probably not considered true Altmer by the islanders at all.
Precisely. The Thalmor should be the embodiment of arrogance and racial pride, all the things the whole race gets hated for. No reason, however, for an Altmer raised in Cyrodiil or any other province to have the same personality traits, at least not to the same degree.
Same as a Dunmer Nerevarine being 'outsider', the Thalmor might actually treat non Summerset Altmer with even greater contempt than humans,seeing them as stains on the Altmer purity. I can easily see reasons why a mainland Elf would actually sympathise with, and actually side with men, against the more traditional elements of their own race, even if in petty revenge for the disdain they are treated with.