if you have 100 points in any skill like say single handed weapons, you will do less damage then a nooby with 20 points but 1 perk in the 25% boost in damage.
In fact, have 100 points and being a master only gives you 24% boost in damage.
I truly think that's why this game is somewhat broken with balance. Perks should be compliments to the skill tree, and increasing skill should be really what determines your character.