I don't think a lore explanation in needed. The way I see it, what we see in these games is not the literal worlds themselves, but an artist's representation of those worlds. When an artist paints a picture or a novelist writes a novel they do not include everything. They choose important details and leave the rest to our imaginations. We do not hear the names of every single inhabitant of Saint Petersburg in Tolstoy's War and Peace, for example. In Oblivion we do not see every single inhabitant of the Imperial City.
I think the same is true of races. On earth, our races come in a vast array of shades and variations. I think we can assume that the same holds true for Nirn as well. The skin tones found among High Elves in Tamriel are probably limitless and include those with white skin. But the artists chose not to represent all High Elves in the composition. As with all other artistic representations, some details are shown to the viewer, and some details are left to the imagination.
That is a very poetic way of looking at it.... I like it!
Thanks for the help!