I see from the Tamriel map that it would be a long journey east or west by ship to get to Vvardenfell from Cyrodil but there is alos a large mountain range between Cyrodil and Morrowind so how did the Imperilas get there in the first place? Ship seems to be feasible but slow so did they take Morrowind first and then moe into Vvardenfell?
There are several passes through Velothi mountains and major roads connecting Cyrodiil and Morrowind.
At the time of Tiber Septim's conquest of Morrowind Skyrim was already part of the Empire and his troops could launch invasion from there as well.
Don't trust everything you see in games, game mechanics and sandbox approach ruled out "realistic" borders.
When it comes to Vvardenfell, Empire gained foothold at much later date, originally it was Tribunal Temple's Reserve with limited Imperial influence.