Depends on how you wish to play. High Elves in general are probably more likely to side with The Thalmor, and The Imperials by proxy. But more because they feel that they OWN The Empire, after the White Gold Concordant. If you're playing more as a mage character I'd definitely side with the Thalmor in your head, anyway. 

There are also plenty of Altmer who live normal lives and don't appear to be concerned with politics at all. Not all of them high and mighty mages, either. There are Altmer farmers, even married to Nords, and the couple who owns the Windhelm stables is Altmer.