There's a lot of stuff that happened in the 200 years between Oblivion and Skyrim, and I'm keeping both fingers crossed for a plethora of new books on the subject in Skyrim. I'm sure the Mede Dynasty (or whoever ended up taking over Cyrodill..Lord of Souls will fill in that blank. Will Titus be assassinated by Aldmeri Dominion? Will Attrebus take the throne? Will Vuhon or some other evil force? Who knows?) probably focused heavily on internal affairs for a while before beginning the aggressive Imperial expansion into other territories.
This.
Apparently, between the events of TES IV and TES V, the Cyrodillic Empire manages to lose most of its fringe territories (Valenwood, the Summerset Isles, Elsweyr, Black Marsh), as well as some of the southernmost counties of Cyrodiil itself (Bravil and Leyawiin). Titus Mede manages to reclaim the lost Cyrodiilic territory, but it's clear that the major decline-period of the Empire has already begun, and he evidently never reconquers the secessionist Elven lands.
The next novel and TES V will undoubtedly be bridging those events together, but it's something we're going to be forced to wait until this fall to finally learn about in full, unabridged detail.