Oblivion's voice actors were much better (I mean, you could actually tell when they were supposed to be angry, scared, or sad), but they fell flat because multiple races shared the same voices, which stripped them of their racial characteristics. A lot of Oblivion's voice actors were in Redguard and Morrowind, but the voice work in those two games had more character it, which helped make them interesting (Khajiit actually sounded like sly cats, Orcs sounded like dumb brutes, Argonians sounded like predatory lizards, Wood Elves sounded lovably annoying, etc).
The number of voice actors you have doesn't matter as much as what you can do with the ones you do have. They could've done a lot with Wes Johnson, Jeff Baker, and Jonathan Bryce alone, but they instead dropped everyone ,good or bad, from the previous games except Wes Johnson, Craig Sechler, and Lynda Carter, and relegated them to bit-parts. Skyrim had a lot of okayish voice actors with a couple good ones, and Oblivion had a handful of very good ones, but in both cases they were utilized poorly.
Although "better" is a subjective term, I broadly agree with what you say here: Oblivion's voice actors were actually very good, the main gripe being that there weren't nearly enough of them. Some of them really lent their personality to the characters they voiced from one game to the next, so I'm very sad that they weren't re-hired, even if they were to voice fewer characters. For the most part, I don't have any gripes about the actual talent of Skyrim's voice actors, but I do think that the directing and casting were rather poor, to say the least. It might've worked in a Fallout game, as I've said elsewhere, but it just sounds wrong in TES.
I'm also a bit perplexed as to the number of comments moaning about English(-ish)-accented Dunmer, since in Oblivion they were voiced by the albeit American (I think) Craig Sechler but had fairly "neutral" accents, and in Morrowind by Jeff Baker who I think is English; indeed about half the voices in Oblivion and Morrowind were English, whereas nearly all in Skyrim with the exception of the "Arnies" have American accents, so it seems a little odd to complain about the few non-American voices. Personally, though I don't mind American accents, I think having pretty much
all of the VA done by Americans sounds a little incongruous in a fantasy setting, before even considering its background: for example, Fallout wouldn't sound quite right if everyone had a French accent... or English, or Mexican etc in case someone nitpicks.