I understand your frustration, many of us go through this. I really want to hear about a new Elder Scrolls game, but at the moment, Fallout: New Vegas is gaining my interest. I'm also kept busy by other games, so I don't spend so much time thinking about ESV.
I do believe that ESV is being worked on, and there is certainly more time being put into it than Fallout 3, as it hasn't been announced yet, and I don't think it will for at least some number of months. Fallout 3 was announced 1 year and 2 months after Oblivion, but ESV has gone past that point after Fallout 3. There's my reasoning. Plus, don't forget about the Skyrim trademarks. They keep getting renewed. Why? Because ESV is in the making? Of course.