Don't you think it's too much of a coincidence that they were happy to let the Dunmer have Solstheim after Red Mountain exploded? That left Vvardenfell completely open for invasion.
Do you honestly believe the Telvanni coast was so easily destroyed? The magic there was just too deep and entrenched. Thousand-year old wizards wouldn't let such a minor thing as a volcano ruin their home. They do have fire-resistance. They have magical shields. They can even cause the illusion of destruction.
Why else do you think Skyrim is so underpopulated? The vast amount of the people went east to leave the freezing cold and left the bandits to take over. The new residents of Vvardenfell are suntanning, netch herding, guar riding, and basking in the heat of everlasting geothermal power. Not to mention that as soon as they crossed the Morrowind border, they gained the ability to create spells to fly through the air.