I don't think perks will be limited to skills only.
But that's exactly how they work. You pick 'perks' under a certain skill, and it makes an area of that skill better. For example some may be under '1-handed weapons' and improve your 1h swords. They dont have generalized perks as Todd said on the podcast a while back. He said something along the lines of "They wont work like they do in Fallout, in that they are instead in tree's under each skill. You have to unlock one perk, to gain access to the next one further down the tree."
So really a perk contest just wouldn't work. :/