To be fair this isn't always the case.
Different ethnicities do tend to have subtle differences in vocal quality, regardless of upbringing. Note I'm not talking about accent, but actual tone and texture of the voice itself. This is a well known fact amongst vocalists, but can be seen in regular speech as well.
I grew up with people of north, east, and west african descent, but I'm only half african and I'm always told I sound like a white person over the phone until people see me. It has nothing to do with my upbringing.
Likewise I know a white person who was adopted by a white mother and black father, but grew up mostly around his father's family. He sounds white regardless of his accent.
In Skyrim, it was incredibly strange hearing the redguards all voiced by white people, even though every previous game gave them black voice-actors.
There are exceptions obviously but there is a basic difference in tonal quality of different ethnic groups. It's going to sound very weird hearing my Wesley Snipes character sound like Ryan Reynolds.