I would definitely invest into a charismatic character, all of my characters so far have had very high charisma, this is because in New Vegas many quests have speech checks. These speech checks can do a variety of things, whether it being able to avoid violent situations to skipping trivial parts of quests or better rewards...
Basically, Charisma pays off a lot more in New Vegas than FO3...
Have you actually seen any charisma speech checks though? I've seen speech checks for int and luck, but not charisma.
I'm playing with a base charisma of 1 and so far have still made all my speech skill checks without even investing very many points into it. With naughty nightwear adding 10 to speech, the speech magazine adding +20 (with the perk), and party time mentats adding +10 speech from the +5 charisma bonus, I can pass speech checks 40 points over my base speech skill.