The Adamantine Tower is basically the anchor of existence, from what I understand, and since all the other Towers are built as an echo of the Adamantine Tower, they also assume that role. I take all that to mean that the Towers basically maintain the laws of reality (causality, linear time, and the barriers of Mundus, and so on), and if a Tower is disabled, it weakens those laws.
I for my self always thought, that the towers are holding Oblivion at bay. But after I readed http://www.imperial-library.info/fsg/nalionarticle2.shtml I began doubting it. It seems very clear, that not the towers play the primary role to protect Mundus from Oblivion. Notice, that just after the death of the Emporer (and his heirs) the Oblivion crisis has begun. Ok, the Chim-El Adabal was gone and in the hands of the enemy, but it wasn't destroyed at all. The Whitegold tower and it's stone was functionally.
As the previous poster said, the fact that Red Tower (and at a few intermittent times, Walk Brass) was functional explains why the barriers didn't crumble the other times the Dragonfires weren't lit. However, if all it takes to start another Oblivion Crisis is to disable three Towers, then it would end up meaning that will have another such event with every new game in the series (Bethesda has a habit of destroying Towers in their games), so I'd be inclined to believe that, while all Towers contribute to maintaining the barriers, that White-Gold Tower plays a bigger part than the others. It actually makes some sense when you consider the fact that the Ayleids evidently had a fair amount of traffic with Oblivion, and it would be reasonable to expect them to want the ability to control Daedric movement in the mortal world (not only keeping them out, but if they controlled the Tower, they would possibly be able to willingly weaken the barrier when it suited them.
Yes, the "Whitegold-Polydox" is a wheel in the wheel and it is the strongest lesser tower. But I cannot assume, that a lesser tower is stronger then a "higher" tower, like the Red tower. As known, the Ayleid harvested very much creatia from Oblivion (the goal was to get back the divinity), so they had to make a pact with the daedric lords. I believe, that the daedra weren't as much interested to enter Mundus at that time (except for Mehrunes Dagon, but he is just one lord instead of 16), as you might assume.
A question that just occurred to me, while I'm on the subject; the Amulet of Kings contained a portion of Akatosh's divinity (which is how Martin did what he did at the end of Oblivion), do you think that was part of the Ayleid's original creation of the Amulet, or was it something that was added when Alessia took control of the Tower?
I think it was an aedric thing. Notice, that it was Akatosh, who give Alessia the amulet. I don't think, that Alessia knew about the towers and it's stones and if she did, she would'n take the Chim-El Adabal - a symbol of ayleid power - as her amulett. She would burn it, destroy it or give it to the Maruhkati Selective... I don't know. But then there came Akatosh, take the really important stone (the Impossipoint of the wheel in the wheel) and said: "Hey girly, this is da Amulet of Kings, yo. It will protect you from da freakin' daedric [censored]s from Oblivion, yo. Care about it, bia[...censored]". Ok, maybe his word choice may was different, but the point is the same