The first thing you learn in the novel is that the Empire fell shortly after Oblivion. I also think it would be interesting, but it's being dealt with in a less "hell-for-continuity" way, because as much as I would have loved to be a major player in such a conflict, how could there be choices and consequences in such a significant and unstable period without murdering future continuity?
Well, its actually perhaps simpler than you think. It wouldn't necessarily mean the series would have to end, even without the empire the world of Tamerial still exists. There would be a major change in the political atmosphere of the game. Its definitely a possibility and could make for an epic main quest.
I could just see my character walking down the pathway of one of the provinces and seeing the army of another providence marching in the other direction. Lots of skirmishes, the guilds in complete uproar and looking for a hero to hold they're institution intact. Chancellor Ocato could be completely corrupted or made into a tyrant by other powers and assassinated.
In the end the world would be twice as changed as is it was in oblivion.
Though it could possibly be saved for a final game..
The elder scrolls themselves being highly connected to the empire right?