Having never been to Summerset (and having little information on it) we can't really say that with any certainty. If the Aylieds & Dunmer (who are afterall just displaced Altmer) are anything to go by I'd be willing to bet they do, or at least did at some point.

I vaguely recall someone in Morrowind saying that slavery disappeared in the Summerset hundreds of years ago. Not sure if the Empire had anything to do with it.

Goblins don't really count in the mind of Tamriels denizens.

broken link

Link fixed: http://www.imperial-library.info/content/skyrim-concises-account-great-war-between-empire-and-aldmeri-dominion