I found the Empire pretty badass in Redguard and Morrowind. Disciplined, organized, and cunning. Not to mention when you have a dragon named Nafalilaargajumpinjehosephat on your side, you just know you're up there on the list of coolest armies in Tamriel. The boring, memetic legion in Oblivion made me hate them.
Though the intro of Redguard showed that the organized empire couldn't do much against the Redguard armies until the country was split by a civil war. The Imperial Legions were almost never conquerers of locations, the Numidium did most of that. They're a guarding force that rarely gets into large scale conflicts.
Still, I don't think that the TES series should ever be completely be without an Empire. Even if they lose a great deal of their power, I'm hoping to see traces of them in TES V as a sort of "shadow organization", like with the Blades in Daggerfall. Especially if the next game is set in a very decentralized province such as Skyrim or Black Marsh - I could definitely imagine agents sent by the Emperor to try and sway local nobles and militia forces to their cause.
Even at that, the empire and the legions are too large to collapse overnight. In disarray, they still will have large pockets of organization. A legion commander with his troops could build their own kingdom wherever they are and become a new city-state. In our ancient world, armies were often used for construction projects more than fighting.