The only way you are making fast enough progress to max out your special is if you are power leveling, which is a build built specifically to level up quickly, if you aren't power leveling then your character is either going to be missing special for a long time, or will have absolutely no special abilities besides having maxed special, which makes the game far more difficult than not maxing special asap.
Again, minimum level of 36 to max all your specials, that is no small investment, and you are not that much more powerful than you would have been out of the vault at level 1, just more hp and better randomly looted gear, because you would have to skip out on the crafting perks, and this relies on you picking up all of the bobble heads before maxing your specials, each one you wait to get 11 is an additional level with no actual advantage gained besides health, which all of your enemies gain far more of.
Going for an 11 in all special build is such a chore for almost no gain that it just isn't fun, you need to get to at least level 43 before you get the last special point, and have to hold off on bobble heads until you get there.
I am level 80, have all bobble heads, and 11 in strength, intelligence, and endurance, all my other specials are still poor, with charisma and perception being 6, and 2 luck and agility, and It is such a sacrifice to give up perk points to special that it just is not worth it.
End, Str, agl, and Int are the perks with the most gain from the special itself. Hp, melee dam, vats, and exp
Lck, per, and cha are very situational at best in their special, but their perks are very strong, especially the high tier ones,
The idea that it is too easy to get special is a bit misguided, the other games stopped at 50 levels, at level 50 in this game you are only a fraction of the way there, and wasting 42 of the easier levels on what amounts to nothing until you start reaching level 80. Seriously though, maxing all your specials does not make you strong, it makes you terribly weak early on, and then you become no different from more reasonable builds, and then you end up at level 270 and finally have the advantage of a small amount of hp, 10%(scalar - 150%->160%) damage with melee, and, and, and, wait, there are no other advantages that are gained by having 11 in the special that aren't rendered irrelevant by perks.