Game difficulty isn't the main issue that bothers me when it comes to the so-called dumbing-down of RPGs.
What bothers me more is a lack of content, depth, and freedom to go where you want, do what you want, and be who you want.
In general, making an RPG "more accessible", "simpler" or "more appealing to mainstream action game fans" are three things I don't want to hear.
My problem with RPG's is that too often they don't have enough realistic content. That and a lack of freedom.
Otherwise, my biggest issue with RPG's is them being unrealistic, and by that, I mean exactly what Skyrim is removing: Classes. I'm not born knowing that I'm a hunter, or priest, or battlemage; if I want to do any of those things, I learn them. I don't think that RPG's should put players in a box and say "you can be this, this, or make up your own thing" rather, I think they should be "you start with x stats at x level and then depending on your choices you can learn to be x thing, spend enough time learning y thing and then you can learn to be x and y." Rather than "You pick x or y and can never learn the one you don't pick."
I also dislike perk systems in RPG's, as they too are unrealistic and quite frankly don't make any form of sense in games. In Skyrim they sort of do, since they could be considered "blessings" from the Gods, but in games like Fallout they make no sense what-so-ever.
Basically, I think that RPG's should be less pen and paper and more "role play" than they are now. Sure, keep the stats, I like them in fact, but don't confine me to certain things. Real life is a sandbox game, and I believe that RPG's should be too.