They had raspy voices because of living near Red Mountain and all the ash storms. Even though I liked that voice, they don't live near Red Mountain anymore and the ash storms haven't been around for 200+ years. So, it wouldn't really make sense unless older Dunmer who also happened to be native from Morrowind and survived the Red Year had raspy voices. Other Dunmer should not.
I don't know where this theory originated from but it has a lot of flaws..
A ) Female Dunmer doesn't have raspy voices on Vvardenfell, nor do they have them on the mainland, or in Cyrodiil.
B ) Male Dunmer on the Morrowind mainland (Mournhold) far from Vvardenfell and the Red Mountain had raspy voices too (well there was
one certain ash storm). Dram in Hammerfell had the same voice as well.
Oblivion just had the same three people doing all the voices and none of them sounds like Jeff Baker.