V-man seemed pretty adamant to not touch his giant dookie in the sky when he froze it in time. He could have removed it, but he didn't want to. He wanted to make sure people's love of him remained. And if you go into the whole Vivec's CHIM deal, Vivec is also love, so if the dunmer forgot love, they forgot him, and the giant dookie in the sky will crash down. Plus, if he lost his godhood completely, I doubt he'd be able to keep the spell intact, even if he wanted to.
As far as defending Vvardenfel, it would have been a smoldering crater anyway, no point in defending a land you cannot even feed yourself from. In all likelihood, I feel V-man would most likely leave Morrowind and go back to wandering, or help his people relocate.
Another thing to look at was how badly the Oblivion Crisis would have affected the morale of the dunmer. If V-man could not have used his awesome powers as previous eras, the people would lose faith in him, especially of what happened.
In the end, Vivec "leaving" was probably the best thing he could do. The dunmer were extremely stagnant, and depended on the Tribunal almost too much for defense. Not to mention, they would never learn certain lessons if there was not going to be consequences.