Do what you like with settlements. I agree it's tedious, but I go back and forth. Sometimes it's good to take a break from questing and shooting and just tinker around with a settlement or two to make them cool, or build up shops so I can make more caps. It's fun for me, but I do get bored/tired of it, then I go out questing some more...
Settlements are totally optional (although they seem to be shoved in your face over and over).
You don't have to build anything for anyone really, it has pretty much no effect on the rest of the game (other than doing some of the minutemen quests to get the castle and the old guns, etc... for the MM quests). Settlements will be attacked and raided, but if you don't want to bother with it just don't. Settlers will die and get replaced with new ones (if you have a recruitment beacon), defenses, crops and water will be ruined, (but from what I understand they will self-repair over time) and failed quests to help the settlers (with ghouls, kidnappings etc...) don't have any effect on the rest of the game at all.
So just have fun with it if you think it's fun, tinker as much or as little as you wish.
There are lots of problems with settlements, but I think the biggest one (for me) is that they make this supposedly "optional" feature in the game, and then they set it directly in front of you and have it pestering you with quests and defense missions etc... for pretty much the entire game. If it's optional then present it to us like it's an option, not like it's the beginning middle and end of the game!!! Sometimes I just want to "opt out".