Too many things to list, really. I could sit here all morning and think of new ones. The people above already mentioned some of them. I've only played Oblivion and Skyrim, so can't comment on earlier games.
Off the top of my head:
The range of colors. Skyrim is a very bleak, grim game. I love it, but I also love returning to Cyrodiil, where there's just lots of color. Even something as simple as grass won't just be green. There's the entire range of greens drawn up into the game, just for grasses.
Humor. Oblivion is funnier, gets me laughing more often, especially NPC conversations and occasional NPC odd situations. Some people think the NPC conversations break immersion...for me, they ADD immersion.
The range of landscapes. From the apparently humid Blackwood forest, to the plains of the West Weald. Getting lost in the Great Forest, and then riding up to Bruma, where it's almost as bleak (sometimes bleaker) than Skyrim.
The sewers! under the Imperial city
The Arena. Skyrim really needs one. I have a feeling it'll be introduced via DLC.
Attributes.
Skill system. I hated Oblivion's level-scaling, but never had a real prob with skills.
Use a skill more often? You get better at it. Makes perfect sense most of the time.
CharGen is better in Oblivion, especially with females. I've made some pretty (and not so pretty) characters in Oblivion. Bottom line: I always am satisfied. Skyrim's females all have this "scowly" look to them.
It's hard to get away from this when creating a face.
....Skyrim's males, on the other hand, are divine.
The Arcane University. Skyrim's College is cool, but there's only one magic center, basically, and it's kinda run down, and sits by a village that has seen better days. When I go to the Arcane University, on the other hand, there's a feeling of splendor that's missing in Skyrim.
Being able to make a character start his/her game just before exiting the sewers. One thing that annoys me in Skyrim is every new character I make is gonna have to start from the very beginning, which means
Spoiler they all are going to nearly be executed but then saved by a dragon.
In Oblivion, we don't have this restriction, meaning we can start a game just before exiting the sewers on a different save, and RP it that the new character never had to deal with meeting the Emperor. We can change everything (Starsign, Class, looks, etc.) just before exiting the sewers.
Oh...
Starsigns. And just the fact that every character is just a little bit (or a lot) unique from the next
at the beginning of the game, mostly due to Attribute and Starsign differences. Having every character start with the exact same stats (ala Skyrim) just feels cheap and dare I say generic as hell. 100/100/100.
...see. I told you I could sit here all morning with this topic!