You can get all skills to 100. You are, some might say severely, limited in perks.
Think about getting the important bit, the majority of the effectiveness, in all skills, and being strictly limited in incidental, unimportant and slight bonuses. Where is your character's uniqueness then?
Try to forget any baggage, past associations, with the words perks and skills, and decide whether skills being more important than perks will actually give you a defined character, markedly different to any other.
Exactly, i made a barbarian oday who only uses light armor and two handed swords, but i am also putting a few perks into archery to get those pesky archers, enchanting, to deal with those mages, a VERY few perks into sneak, block and a few other things to give my character a heads up a bit, but other than that, i feel I am playing as a barbarian, much different than my high mage, and my knight character.