I'd say so. Personally, I think the empire has rotted from within. Take a look at the situation in Vvardenfell. The Great Houses, along with a few Imperial organizations, control the little arable land that exists. Raiders, bandits, and monsters are relegated to the more dry and inhospitable areas of the island: The Ashlands, Molag Amur. Even the bandit caves lie along the road away from good arable land. And the books that talk about Imperial involvement in the East speak of it as a drain upon their resources, driven more by the personal avarice of the powerful than the good of the Empire.
Look at Cyrodil in comparison. Settlement is sparse, and heavily interspersed with goblins, bandits, and marauders. The countryside is dominated by ruined forts, long since abandoned by the Empire but hosting healthy populations of outlaws. Goblins prey upon farmers no more than a day's ride (scaled up from the game model) from the walls of the great towns (Chorrel in particular), and mines are constantly infested by the little buggers. Actually clearing them out (Fighters guild quest, Cheydinhal) is the exception, not the rule. My guess is that the heavy taxes (including currency depreciation) necessary to keep the empire running are slowly destroying the communities that form its base. As people find it harder and harder to live in a law-abiding fashion, they drop out of civilization, gathering in the remote places to find strength in numbers against rapacious tax collectors.
And as this process goes on, "remote areas" become progressively more numerous and closer to the official centers of Imperial civilization. And not only do human outlaws gather in these places, so do goblins. With the strength of Cyrodil perpetually abroad, little can be done about it... and little could be done anyway due to the atrocious economic policy that leads to the problem in the first place.
This decay can also be seen in the nobility of Cyrodil. Bruma and Cheydinhal have a competant lady and lord, respectively, but Bruma's is due to the constant influx of hardy Nord stock, while Cheydinhal's ruler is a foreigner. The situation is much worse to the south: Bravil's lord is a drunken wastrel, while Leyawin's is a man who trusts nobody, and is not worthy of trust... particularly by the Khajit and Argonian populace that inhabits this area. Skingrad benefits from the fact that its lordship hasn't changed hands for at least several hundred years, while Anvil, despite its beloved ruler and position as a center of oceangoing trade to parts West, still suffers from a growing urban blight. And Chorrel is ruled by an old lady, who spends most of her time ruminating over her lost husband (and her lost painting), and, so far as I can see, has no heir other than her daughter, who is married to the Duke of Leyawin and is possibly even worse than her husband. We don't know what conditions were like in Kvatch before it fell. Hmm... I've never really explored Kvatch to try to figure out what it looked like before it fell to Oblivion.
With the tax base eroding, its only a matter of time until the Legion starts missing payroll on a regular basis. I'd say that all Oblivion would break loose when that happens... but it looks like it already has.
So thats why theres so many outlaws in Cyrodiil?
The Empire has been going in and out of crisis ever sine the reign of Pelagius I, and every time, the Empire has managed to weather said crisis and come out of it with the power mostly intact. However, then, Cryodiil and the other Provinces had an Emperor to guide them. Since Mehrunes Dagon had his little hissy fit, there now is no heir to the Dragon Throne directly from the Septim line.
The devs seem to forget that there are descendants of Tiber Septim ruling Sharnhelm, but I will leave that little bit alone for another thread.
Anyway, the Empire, as it stands now, is destined to crumble. That doesn't mean that all hell is gonna break loose. I forsee something along the lines of a Confederacy between a number of states. Cyrodiil, Hammerfell, and Skyrim will likely remain close to each. Morrowind will be totally independant. Another war will errupt between Valenwood and Elsywere (shocker /sarcasm). Summerset Isle might make another grab to form the Dominion again. High Rock has two possibilites, go back to the chaos before the Warp, or become a more united country. And then Argonia will continue to be itself.
I think the new book coming out mentions something about a descendant.
But anyway yeah I geuss it makes sense for the Empire to crumble, but going off from lore in ES 5 I don't want to see a "fall of Rome" scenario where the Imperial City literally gets sacked and all of Tamriel gos into Dark Ages, I'd rather see the empire fall apart so that provinces like Cyrodiil, Hammerfell, and High Rock become groups of city states, that govern themselves and have their own armies ETC, rather than all hell break loose and everything becoming destroyed and everyone living in mud huts like the Dark ages.
I mean I'm somewhat concerned over this because alot of what I like about these fantasy games comes from the Dark Ages, but for the game to actually become that would svck.