Hermaeus Mora is arrogant and believes everyone is bound to serve him no matter what, but that doesn't mean it's the truth.

All Daedra obtain servants by the lure of power, gifts and invitation. Think of Hircine, Boethiah, Nocturnal and so forth. Old Herma-Mora is no exception to the rule, he can't simply decide you're his [censored] and that is final.

Otherwise why would he even bother trying to tempt mortals.

Most importantly, the dialogue where Frea says you were forced to serve Mora in order to defeat Miraak, and warns you against going further down that path. If you already were there, no such conversation would be included.

Mora can decide anything he wants. The world could be his collective slave... doesn't make it true though.

Its about as effective as me saying you aren't welcome here. I can say and think that all I want, but that doesn't mean it is true.