The problem isn't the customization that perks bring, it's how balanced Bethesda can make the perks. Fallout 3 had horrible balancing when it came to perks, so if they don't balance them out well, it could lead to extremely overpowered characters like in Fallout 3.
Seriously, by the time you were level 20, nothing in the Wasteland could stop you. Nothing could even come close to hurting you haha.
And yeah, playing ridiculously overpowered characters is no fun in an RPG, especially when you are trying to RP a character. Like people said, Fallout was ridiculously easy, even at the hardest difficulty level.
Curiously, I never had that problem with Oblivion though. You crank that difficulty slider up, and you are gonna die.