No no, I think you misunderstand me. You're saying they must have some perks, like 20%, 40%, 60%, etc improved damage right? I'm saying that they don't have any perks, they don't have a perk system, they don't even have ANY system. They are hardcoded to do arbitrary amounts of damage depending on level. They are not modeled the way the player is.
On topic, though, my $0.02 is that you should put a few points into those perks. In fact, I think calling them "perks" is really a misnomer. They're skill trees. A perk that does 20% to 100% more damage is not a 'perk', it is a fundamental part of using that skill. If you want your character to swing a one-handed sword with any ability, you're gonna have to invest at least a few points.