You aren't going to miss much, there's many lore books in the game that you can read if you wish to catch up on details of the past, along with many historical sites that you can see in-game to get a feel for the ruggedness of the cultures. If you want to know how the gameplay works out, that's mainly the only thing different from the rest of the other games' play, it's more streamlined now with less drawbacks and more jump-in-and-play-how-you-want action.
I hope that's all true. I'll really feel like a character who was truly born after all those past events, only to learn of them through history texts and word of mouth from NPCs. My biggest fear now is that the world of Skyrim will be so immersive that it'll destroy my social life and follow me into my dreams