they haven't been working on it this whole 5 years, they probably started 2 or 3 years ago. besides thats not the point, the point of delaying the game is not to add more stuff to the game, but to fix the bugs that ruin immersion and the fun of playing the game.
It depends what you consider working on the game. If you cosider throwing ideas out there and getting some stuff on paper of what they plan to do working on the game, they may have started before Oblivion was finished. If you cosider putting those ideas into the game world, then it might be three years ago. A few people were just working on dragons a long while before the game world, so it really kinda depends.
On topic however, I would be upset but I would also be understanding. They wouldn't delay the game just to piss us off, so they would have a good reason if they did it. I'd like to be told that reason, but even if I wasn't I would simply continue to wait.