I still don't understand why the Crystal tower would 'hold the world up', if it is the altmer's desire to return to the et'ada. Am I missing something?
It doesn't. Crystal tower wasn't a part of Mankar's plot.
The powers also created Red Tower and the First Stone. This allowed the Mundus to exist without the full presence of the divine. In this way, the powers of Ada-mantia granted the Mundus a special kind of divinity, which is called NIRN, the consequence of variable fate.
After these two acts, which is commonly called the Convention, the gods left the earth.
Red Mountain prevented the Oblivion invasion, so the Mythic Dawn deactivated it. That was Morrowind's plot. The masters of White Gold actually sat on a door to Oblivion, sealed by the barriers of Red Mountain.
White-Gold Tower is a conduit of creatia, aad sembia sembio, built to bring about a reversal of the congealing spiritual bleed caused by the Convention. In other words, it was a focus point for (re-)reaching the divine.
White-Gold Tower was made by the Ayleids, the Heartland High Elves that would have none to do with their isle-kind. Where the Altmer sought to focus on dracochrysalis, or keeping elder magic bound before it could change into something lesser (and act which ironically required Aetherial surplus), the Ayleids harvested castaway creatia from Oblivion by entering a pact with the masters of the Void, the Princes of Misrule.
White Gold's design is unique, because it harvests creatia from Oblivion. Mankar took the Amulet to open that door for Dagon, who broke the barriers once inside. As the Red Mountain barriers were now broken, Martin mantled Akatosh and sacrificed his divinity, for the new barriers. Crystal Tower had nothing to do with Oblivion's plot.
The Ayleids were not content, until they ate creatia from the table of the Daedra.