I suspect some bug at play on scaling here, because sometimes on my 400hp with max physical mitigation char, I would get one shotted by most random non boss bandit-ish encounter. I reload then I can tank the whole group forever.
Bosses (bandit chief, deathlords, etc.) actually scale consistently. It's these super grunts that are really weird, and no real way of playing around it either. But they are really rare, so rare (faced 2 in my 220 hours) to be non-concern for me.
This has been my experience as well.
I'm fine with the actually intended more powerful "boss" enemies like Arch Deathlords and Dragon Priests and encounters of Falmer Shadowmasters, but I can remember several instances where I've been absolutely murdered by run of the mill bandit enemies.
Upon reloading to fight them again they can barely scratch me, as they should with a ~520 armor rating.
I remember one time on the way to Blackreach I found that Khajiit who was still waiting in the cave before the dwemer ruin. He killed me in two hits with a woodcutting axe and no armor. I was wearing full Ebony and had Ironflesh casted.
It really seems like a legitimate bug.