There was some disagreement over whether the game world came from the "lore" or vice versa. I did some online research and it turns out that the first Elder Scrolls game, "Elder Scrolls Arena" which was released in 1994 had no "lore" books as they have come to be known. The game world, as a matter of historic record, existed before "the lore." The "lore" was added later and is therefore an outgrowth of the game world and not the other way around.
Read about it here:
http://elderscrolls.wikia.com/wiki/The_Elder_Scrolls:_Arena