Enemies scale, especially main quest enemies. At level 60, they're doing oodles more damage than they would at level 10, and I'm positive the player's stats will lag behind after a point. After perks and gear, only your HP marginally increases with each level, everything else has a natural top out in-game.
But as for the topic overall, this game was never meant to be super difficult. Inevitably one will get into a violent confrontation, not everyone or everything can be reasoned with after all, but game play needs to be open enough to avoiding violent conflict. Personally, I think it would be nice if harder difficulties simply meant more enemies, more accurate enemies, and more damage from enemies without any associated durability boosts. Sure, the game can be easy, but if even with bleeding edge equipment and perks into damage resistances and extra health we still go down in 3-4 shots from your average raider, it can be hard. Though honestly, I think placing more requirements on the player might be the smarter way to go. Less ammo found, food and drink becomes a requirement. Weapons and armor have durability (unusable when broken, or just less damage and inaccurate at lowest durability?) All humanoid enemies have stims and use chems for various effects (jet would make raiders super accurate, buffout faster and more durable, etc. Obviously better armors and weapons, as opposed to simply beefier health and damage values. If Bethesda could afford to tax players machines with smarter and more numerous AI, there's no reason not to even on normal.