Not being able to consolidate their own province is dire, and does not shine well on Imperial fortune it all. But all sources and ideas indicate that the Empire isn't that weak. Tiber Septim only died a couple of hundred years ago, as "the richest man in history".
I think the entire representation of the Empire in TES 4 was completely problematic in this regard. Disproportionate and misleading.
Although descriptions of the grandness of the past and descriptions of the decrepitness of the present condition of the Empire were often so overlapping that it was often confusing (especially since books about the past will still use the name Septim, which can be misleading), I think it's been made fairly clear that a lot of people don't rightly know how Tiber Septim did it the first time at all. In Daggerfall people casually speculated help from the daedra or some type of "deal with the devil" in order to do what he did.
In the present day, I think what little power the Empire is able to amass is so diluted across the provinces that yes, Cyrodiil would be the very first province to be neglected in order to hold the Empire together. Morrowind seemed to have, proportionately speaking, a high occupation force to population ratio. And the attrition and fatality rate of serving in the Legions on these 'fringe' provinces like Morrowind, Elsweyr, Black Marsh and Skyrim seems to be high. The rest of the Empire certainly seems to regard them as still somewhat wild and dangerous. Also, a lot of the forts in Cyrodiil itself are outdated. They're like castles spread across the various countries of Europe from the times when the countries used to be broken down into individual principalities or fiefdoms, before imperial nationhood emerged. And a lot of them were used for things like the Akaviri invasion, in a time in the distant past when Cyrodiil routinely expected foreign invasion (either from overseas or other provinces/kingdoms.) So when the Empire doesn't restore and man all these ruins across their own province (and most are ridiculously closely placed together anyway, even assuming a feudal society) it's asking an overstretched empire to focus too much of its power and resources at home. For a lot of its history Rome itself, for example, was not heavily defended during the Roman Empire.
I do think that the fact that these ruins invariably became homes, often within eyesight of farming villages or cities, of cults of necromancers, the undead, rogues and thieves and cutthroats, goblins, vampires, psychotic cult worshippers and the like, they should have at least levelled these fortifications at some point.