The Imperial Race in Elder Scrolls resemble Romans, their weren't many brown skin skin Romans. The Etruscans who started Rome depicted themselves in paintings and many had brown skin but as the Roman Republic started not so many people were brown skin. I would say during Roman times about 30% of the Italian population had a dark brown skin tone, maybe less but in modern day Italy you rarely see a brown skinned person and everyone has olive-fair skin.

So why are Imperials in Skyrim so dark? Many of them I see have a medium brown skin tone, In Oblivion they were olive-fair skinned and never that dark. It makes no sense, skyrim is cold and it is rarely sunny, why should they be so dark? I keep making it up in my head that they migrated to Skyrim from Hammerfell since war brokout there and the Empire lost its hold on Hammerfell over the past 10 years since the events in Skyrim.