I see. The early Atmoran civilization wasn't theirs, it was really the draconian civilization, we're looking back to...
Yet if this was the case, I don't see it in Bjorik's study:
In Atmora, where Ysgramor and his people came from, the dragon priests demanded tribute and set down laws and codes of living that kept peace between dragons and men. In Tamriel, they were not nearly as benevolent. It's unclear if this was due to an ambitious dragon priest, or a particular dragon, or a series of weak kings. Whatever the cause, the dragon priests began to rule with an iron fist, making virtual slaves of the rest of the population.
The abuses came from priests, really. Man were the ones to enslave man, while Alduin's dragons needed to bargain for his worship, which in turn supported the people's fear of the priests.
Tamriel's dragon priests, all but Miraak, who turned on the dragons and priesthood, turned on man for their survival, for the promise of safety and near deity. Alduin must have terrified them, and their avarice blinded them, to the extent they would throw man under the bus, if only they could survive the World-Eating.