The Altmer are holding back the mortal races from true achievements. Killing them and sacrificing their souls to Daedra is doing both them and the world a favor.
Last I checked, unless I missed something, 'twas not an Altmer who seized control over a 1,000 foot-tall Dwemer construct and proceeded to unleash it on anyone who wouldn't submit to their rule. Also wasn't an Altmer who enabled said person to do so.
Last I checked, 'twas not a line of Altmers who ruled the Tamrielic Empire for four centuries. Ocato was an Altmer, and headed the Elder Council, but he was no Emperor. And his rule, if you can call it that, was taken from him by an Imperial. (I'm assuming Titus Mede was an Imperial, even if he was not, he doesn't sound like an Altmer.)
Last I checked, 'twas never the Altmers who ever ruled the whole of Tamriel. Indeed, no province of the Third Era, or Fourth (as far as I know) suffers from being conquered by Altmers.
Last I checked, 'twas not an Altmer who once dwelled beneath Red Mountain, brainwashing their own people, with the intention of ruling over them one day.
So, do you care to explain to me how the Altmers are so responsible for oppressing everyone else?
(I'll grant you that Mannimarco and Mankar Camoran were certainly trouble-makers, since you'll probably bring them up. I'd also argue that the majority of the Altmer race wouldn't back them up on their goals.)