I would. If I had a character of low intelligence, I'd want him to use small words, choppier sentences, and in general come off as incredibly thick compared to what a high intelligence character would sound like.
As for age, I don't know about you, but I certainly didn't sound like I do right now when I was 18, and I doubt I'll sound the same when I'm in my forties. And as for Karma, that brings up the fact that even there it's not a one size fits all kind of deal, which is what we'd be getting if we only had one voice to (not) choose from. We might be getting the soft spoken sadist, but lose the option for the manic fast rambling lunatic, or the guttural growler, or the jovial and polite affably evil. We'd be locked into a road that would render the other choices we could make nonsensical because they run counter to the personality forced upon us by the voice.
As for that last part, see above. Just because you were playing as a pre-established person, doesn't mean you were a pre-established character in terms of personality. That's always been free for you to shape and mold, and I feel it should remain that way.
The problem is you're treating the lack of a voiced character as the problem of bad stories rather than the actual problem: bad writing. Samuel L. Jackson voicing your character would not have saved Fallout 3's nonsensical ending, but having Gilbert Gottfried saying all of the Courier's lines would have crippled New Vegas.
Also, pointing out that Fallout 3 failed in regards to dialogue reflecting S.P.E.C.I.A.L. and skill checks isn't really a winning argument to me, because ideally Fallout 4 would improve on those areas. And it's not just getting them to sound stupid as opposed to smart. It's getting them to sound practically brain dead, thick-headed, slow-witted, dumb, kind of slow, average, a little bright, smart, very smart, genius.
The key to a better story is better writing, and while a defined protagonist can lend itself to that, the more important thing is the world around that character, the conflicts that are going on in it, and how the characters react to you tackling those conflicts.
Giving the protagonist a voice and saying it makes for a better story is akin to trying to fix a dripping faucet to keep from wasting water while ignoring the busted water pipe in the basemant.