You'd think that but then again there were black slavers in Fallout 3 which I've never understood.
No races are innocent of slave trade. As an example, African slave traders were largely to blame for bringing slavery to the American colonies. There was not much of a market for their slaves in Europe, where there was already feudalism and the lords were served by their serfs. Consequently, the African slave traders marketed their slaves to the New World instead, where there was a definite need for labor. In time, the slaves became 'institutionalized' there. Here's an interesting http://en.wikipedia.org/wiki/African_slave_trade on African slave trade.
In any case, there will always be immoral individuals who will seek to dominate others, regardless of color, when there is no rule of law.