Slavery and serfdom, when they are past their used by date, are a MAJOR hindrance to an economy rather than a boost. Look at imperial Russia in 1914.
But, i don't know if bringing in slaves from the outside has the same affect as enslaving the native population. The slaves in Morrowind were outside the normal population, and seemed to be only used for farm labouring, mining and service. If the rest of the popluation is encouraged to be industrious and innovative, the only drawback could be relations from non-slaver states.
Aside from the immorality of it, slavery is probably inefficient compared to the alternatives. Slaveholders need overseers, whips and chains, and slavecatchers; and live with the permanent fear of rebellion. An unwilling workforce probably sabotages production when possible. I once read an expert's speculation that Egypt's pyramids weren't built by slaves (more likely, similar to draftees) because the quality of the construction is too high.
Although it was for the Lore RP instead of canon (yet), Ted Peterson wrote that Helseth made a political marriage with a Dres noblewoman who, along with others of her generation of Dres, was inclined to end slavery. In the RP, Helseth abolishes slavery with little advance notice, partly to cause difficulties for his opponents, the Houses Indoril and Redoran. Ted Peterson's version of Helseth has as his long-term goal the abolition of the House system itself. The Dres nobility was shown as going along with the abolition of slavery through a combination of persuasion and veiled threats, giving up their household troops in favor of Helseth's centralized rule. As I've said, this isn't canon at this point but it fitted together at the time.