I've been wondering how could 500 warriors could destroy an entire civilization and put a single race to near extinction. Could Ysgramor himself be an unstopoable warrior? Could these 500 men perhaps have some blessing to make them powerful? Or could 500 is not a litteral term and that Ysgramor came back to Skyrim with a massive army?

They might have had dragons as well as Pelinal Whitestrake.. good enough for me.

They were also assisted by Ahzidal, who was a master enchanter. He imbued their weapons with magic that he learned from the elves, making it easier for Ysgramor and his men to cut the Falmer down.