Elves are big doodoo heads. After the Ehlnofey arrived on Nirn, the scattered, wandering Ehlnofey tribes (proto-men) managed to find each other as well as the Old Ehlnofey (proto-elves), and the wandering Ehlnofey just wanted to get all back together and be a happy family again. But of course, since the elves are doodoo heads with severe entitlement issues, they saw the men as unclean and tainted, went "no way jose" and started a war that ended up sinking the majority of the Hist's realm (who up to then were content staying uninvolved and not getting in anyone's way).
Then after the Hist created the Argonians to try and integrate with the growing societies of man and mer, the Dunmer (continuing the elven tradition of being doodoo heads) had to go and enslave them while cutting down more Hist. After the Oblivion Crisis, the Hist/Argonians were finally able to get some retribution, though I wouldn't quite call it even considering just how many Hist have been lost due to the Ehlnofey war and the long years of slavery. Running the last of Dunmer out of the land that was already kicking them out doesn't quite compare, IMO.
So, yeah. Kinda hard for me not to sympathize with the lizards there.