Speech definitely makes your life easier in the game as there is are numerous speech checks that can either make quests easier to do, better rewards, skip certain parts of quests, etc, etc...
So in the end it is up to your preference on what your character to be...
You could always bump Charisma down to say 3 or so and simply tag Speech as one of your skills to make up for the deficit also.