To me, it's like asking which was the best GTA game. They always get better, but which one deserves the most accolades? GTA IV was clearly superior to its predecessors in every way, but it, like the newest TES games, are really just improving that which the ground-breaking title in the series (GTA3 for GTA, Morrowind for ES) created. There was a massive paradigm-shift in the development of these games, which would be followed by every game in-series (and many clones or variations by other groups) taking the majority of its function from that cornerstone-like title.
Which game, would I say, is the best right now? The one I'm currently enjoying most? Well, that'd be Skyrim. Which is the game which made this all possible? That would be Morrowind.
Plus when people say one or the other is the "worst" we're still talking about a game that scores in the upper 90th percentile or so, compared to games within the 98th percentile. (These numbers are not cited or particularly factual, but a gross exaggeration to explain my point.)