And... sometimes that is the right thing for the game to be doing.
If you are playing a "thief" character (which is where most of those non-combat skills come from), game combat is supposed to be harder for you. There has to be a reason you are sneaking...
Put differently, if you reach level 50 without taking any combat perks, this means you are playing a character who is not interested in combat. This is going to make combat difficult for you, but there are ways to deal with opponents without combat, and you should be using them. For example, you should get other people (ideally your enemies) to fight those dragons for you. Or, just run away and avoid them. And, yes, this means you cannot complete every quest line on that character, but something similar happens when you fight your way through and wind up destroying or alienating your quest givers.
Put differently, there are dragons in the world, and time is passing, and everyone that wants to survive is having to deal with dragons (or deal with others who have been dealing with dragons), or die (or hide, or... whatever). Levelups represent time passing, and represent your choices in how you have passed your time.
(And, of course, when that's not good enough, there's always mods.)
Mind you, I do have my frustrations with the game, but they are different...