I never bothered with difficulty anyway, since all it did was make the game incredibly unrealistic.
And so did the elves, orcs, monsters, magic, and gods. Complaining about realism in a video game that has freaking
wizards in it is one of the most asinine things I've ever heard.
Back on topic, I don't see any reason to remove the difficulty slider. Everything you could accomplish with a set easy/normal/hard difficulty level you can do by moving the difficulty slider to the appropriate area, without the added benefit of being able to fine-tune your choice. I enjoy a slightly higher difficulty when I play so I move the slider over a tiny bit, whereas my wife (if she were to play a TES game) would probably slide it over to make the game much easier.