I can't help but think it's a much darker time than the Nord entry on UESPWiki states. As with any war there are atrocities on both sides, here arguably much worse because of a definite lack of sympathy between the mer and men. For the men to establish as strong a foothold as they did, the Night of Tears would just be the beginning of the horrors, since the Nord culture has a strong sense of vengeance. To drive the Falmer from Skyrim would mean something worse than the Ayleids enslaving Nedes, which was in part a reaction to what was going on up north.
I tend to disagree with anyone who thinks that there would be lots of (actually, any) Falmer ruins scattered around Skyrim, like Ayleid ruins in Oblivion. In this case, the dominant culture was the usurper, who would be as interested in destroying symbols of power of the previous culture, as in destroying the previous culture itself (Fall of the Snow Prince is pretty clear on the latter). I'm not against culturally-degenerate, isolated Falmer hiding in Skyrim (they're long-lived elves after all), but most should have interbred and effectively disappeared, and in their original form should be nowhere any group of Nords have frequently travelled to in the past few milennia.