I've played Morrowind for some time now and Oblivion for quite a while, and while I've managed to somehow wrap my head around how attributes work, I still find them a bit confusing. However, I also agree with those who say that attributes add another level to character development - one that Skyrim's system lacks. It allows for situations like when my Personality was drained in Morrowind and NPCs were like "You stink!" "I don't have time to talk to you", "Why does it approach?" etc. ...Or the addiction quest in Shivering Isles, which was brilliant.
On the other hand I like perks for the special abilities that you get, and it's also quite natural that someone who's good at something, can do things that more unskilled people can't.
I think what makes attributes hard to get into, at least for me, is that some of them are vaguely defined. What Personality does is obvious, while the line between Strength and Endurance is blurry. Yes, I would like to have attributes back, but not in their original form, and I don't want perks removed. So why not have them both, but more distinct and clearly defined?
Example:
When you level up, you choose to increase either your health, magicka or stamina, and also put points into 2 or 3 attributes (depending on how many there are). Attributes determine your overall efficiency. Such as Willpower = How well you resist magic. Athletics = How fast you can move with armor and weapons, and the stamina cost when you attack. Simple as that.
At level 5, level 10, level 15 and so on, you may also pick a perk; those perks are special abilities or allow you to specialize in certain aspects of a skill, such as fire magic in Destruction or axes in Two-handed.
...Well, that's a rough idea that can be improved, but I'm certain that it would be possible to have a character development system that has depth but isn't too hard to get into.