Granted, you will need a high-ish INT and CHA (and their perks) to make a really awesome, fully functional settlement but since there are likely level restrictions on some of the higher-rank perks, and since I won't have many resources in the early game, and in the early game I'll still be tinkering with my character.... I doubt I'll start with CHA 6. More likely CHA 3 (for Lone Wanderer perk) and then build up CHA over time. Of course that all depends on what ~else~ Charisma does for me ---- if a high CHA gives me more dialog options then, yah, I might start with a high CHA.
I just don't want to get absorbed in settlement building only to find myself level-gated and unable to advance any farther until I scrounge more resources, meet more people, acquire more level-gated perks, etc. I'd rather jump in with a bigger "toolbox", if you will, when I do decide to start building.