The majority of Altmer are actually against the Dominion.
From http://elderscrolls.wikia.com/wiki/Legate_Fasendil
Sounds like you've seen you're fare share of hardship.
"The life of a soldier is full of hardship, that's nothing, but they send the Legion to places that have become too bad to be settled with out violence. What's hard is seeing people warped by evil, and I've seen the face of evil, it was in the air above Sentinel on the night of Green Fire."
What happened on Green Fire?
"Back in 42 in was on leave stationed in Hammerfell on leave in Sentinel trying to track down refugee family members who were trying to escape persecution in Alinor. Suddenly, an explosion of magic in the refugee quarter. Thalmor mages were attacking Altmer dissidence, who were resisting with magic of their own. I ran with some other legionaries who were stationed there, but the entire quarter was a smoking ruin by the time we got there, everyone was dead, wholesale slaughter. The Dominion, not content with killing dissidence of their own back home, came to Hammerfell to finish the job. We're supposedly at peace now, but I put in to be stationed here to keep an eye on the Thalmor, I have a feeling they're behind this unrest here in Skyrim."
It'd make perfect sense 