The money that went to writers in earlier games now goes to the voice actors. Since voice out costs restrict the amount of text, there is no point in hiring a writer who can phrase a vast amount of inconsequential small talk phrases, so they make one list and use it for everyone. Careful what you wish for.
I don't think this is so much the case. There are other games with extremely varied dialogue. Writers don't often make much money to begin with in many cases, and in others, the developers themselves write the dialogue. Also, it wouldn't be hard to say to a voice actor, "Okay, here's the list, but improvise on it a little. Where it says 'hello' for example, just give a couple of greetings for us."
I too would like to see more variety. Instead of every gender of every race saying, "I saw a mudcrab the other day," they should have a variety based on the same concept. "Have you seen those mudcrabs? Pests!" or "I killed another mudcrab yesterday" or "These mudcrabs are becoming a real nuisance."
What annoyed me more was when beggars changed voice all of a sudden when asking about a topic...
Agreed. I think that was either a bug, or they simply didn't record extra dialogue for the beggars in the same style as the other bits in which case it was a rather bad oversight.