So the worst parts of the past 3 Elderscrolls games are, individually...
For Morrowind I chose, personally, the AI. Or non existent AI more like. Can't chase you through doors without a mod, can't use anything without a mod, even merchants equip random stuff you sell them without a mod.
For Oblivion it was the combat, what a snore fest. Not that Skyrim or Morrowind were better, but just that it somehow stuck out for me the most.
For Skyrim it was the world design. It was beautiful, and generic, and hollow. The conceit of other Elderscrolls, that you were in a real world rather than a video game playpen, was ditched. The dungeons were linear and obviously "levels" the cities were still a loading morass of tinyness, there were barrels I couldn't open just because, objects I couldn't pick up for no reason, etc.
Just my opinions, what are yours?