The Simulacrum was the start of it. Jagar Tharn (the Usurper) had control of the Empire for a decade, and instigated conflict accross Tamriel for an unknown reason (though he was connected to Mehrunes Dagon in some fashion.) Most of these were old, deep-seated grudges between provinces (i.e. "Hey, Black Marsh... doesn't it make you angry that Morrowind has neither apologized for slavery, not stopped the practice? You should do something about that!")
With a continent-spanning Empire being purposefully run into the ground for a decade... well, it should tell volumes of how effective Uriel Septum was as Emperor that he managed to keep the whole thing together until he died in the Oblivion Crisis.
While it's true that had the Oblivion Crisis not occured, the Empire as we know it would not have fallen, the reasons for the fall in the first place were instigated during the Simulacrum. Hell, in Morrowind there's talk of pulling the Legions out of Morrowind due to ongoing problems back in the Imperial City- one issue was that a riot was forcefully put down after claims that Uriel's heirs were impostors planted by Jagar Tharn. While unlikely, the effects of the Simulacron were still being felt thirty years later.