I'd agree with that. Legendary difficulty as a damage modifier is a poor substitute for ramped AI, better npc detection and smarter tactical responses, but its what we have at the vanilla benchmark.
It does indeed get easier after like 30-40 levels, but for a DiD char, getting to those levels is the tough bit, normally resulting in an early death before the easier time starts.
With all the maxed gear....regular bandit encounters, bears, sabbycats, draugr upto Scourge are no biggie making 90% of random encounters very easy. However, tomb delving will soon get you to Deathlords (and worse) with ebony battleaxes etc.
Bandit marauders are no problem...unless.....they are rapid firing shock mages....OP with thunderbolt so even at the 85% magic resist cap you take fatal hits in seconds. Falmer Warmongers...often in teams...are pretty deadly too....you need huge sneak and silence to creep past these guys.....thankfully they can be paralysed...unlike....Vampires! The undead are a really dangerous pain....powerful shock and that life drain to keep them fighting fit at your expense...Alteration does have a paralyse spell that works on them, but only for five seconds which is not enough if you are getting surrounded! Chaurus hunters....awful things...can't be paralysed by poison....persistent beyond belief and very deadly poison...Dragons, well, getting caught in the open by a Revered Dragon with that drain vitality shout is a ten second death sentence.
Well that's just a few of the enemies that will kill even a level forty legendary player char. Of course, if you don't do silly nerfs...like refusing to take healing potions or use healing spells then its is much easier....but once you get into the challenge of not doing a quick healing up during combat...well its a hard habit to break. To me now, taking a healing potion feels like cheating...crazy I know, but that is how me and the Contessa roll.