I have just re-read the Pocket Guide to the Empire: Cyrodiil 1st Edition and I took notice of the differences between the Colovians and the Nibenese.
Accroding to the PGE the Colovians are known for keeping more in touch with their nordic ancestry and that "Their primitive ferocity was disinclined to magic or the need for industry, preferring bloody engagement and plunder instead". On the other hand, the Nibenese culture were/are known to have a mage and battlemage aristocracy.
So, the question is: Could it be that the playable Imperials in Morrowind and Oblivion are based on the Colovian, while the playable Imperials in Skyrim are based on the Nibenese? To me this seems likely, but more importantly, lorefriendly. Well, with the exception that the Imperials in Skyrim doesn't have Speech bonuses.