Of all time? I can never answer that. Filtering out nostalgia is just too hard. And then you have to figure out what might be nostalgia, what might just be changing tastes. And nostalgia versus novelty. I loved a game greater than any game I've loved before. But I can't play it now. Is it because I played it too much, the systems are too dated, or my tastes have changed? Should that matter? Woe be to my complex decision making processes.
Obviously nostalgia plays a role in it, but I think the catalyst should be the impact the game made when you first played it.
My favorite game of all time is System Shock (as if you couldn't tell by my handle), the original 1994 release: beyond the mechanics and the graphics, the story captivated me, the combining of role playing elements with action adventure elements captivated me... I couldn't map my keys, the graphics were all 400x600, etc. but the experience I had while playing that game, has not been surpassed, although Deus Ex came pretty close.
As to the OP, no Skyrim is not my top 5 simply because I have seen it all before. Not that I don't enjoy playing it and once I get the CS and Blender scripts I am going to mod the crap out of it, but that is not going to make it a top 5 game for me, amybe not even 10.
So for top 5, I got:
System Shock (the original)
Deus Ex
Legend of Zelda: Link's Awakening
Wing Commander: Privateer
Sins of the Fathers