Sounds line it's working as it should be then.
Hardest difficulty should be unreasonable hard to play.
The game should be balanced so that most people would play on medium difficulty. Most people should get their asses kicked on the hardest difficulty.
Otherwise they might as well just go with [ Normal ] -- [ Easy ] -- [ Very Easy ] -- [ Ages 2-4 ]
Dosent really sound like that, no

Hardest difficulty is still easy most of the time if you dont put various other restrictions on yourself. Like no Smithing, Enchanting, not killing enemys while they cant reach you etc.
If you do that normal enemys become more interesting (still not hard though), but you will get killed by generic enemys that occasional pop up and are for what ever reason 20X as strong as their compadres.
So basically the difficulty suddenly rises and then decreases again by some hundred percent, no matter on which difficulty setting you actually are.