I wouldn't say so soon that the rank and file Telvanni are completely cumbersome and illiterate dregs, with not enough tuppence between them to buy food... To join the House you had to sport at least "a moderate intelligence and willpower" and profess magickal ability, if anything Telvanni elitism is a barrier to peasantry and general middle age squalor. Yes, they live in mushrooms, or large communal housing, but that doesnt make them illiterate or stupid, and the non-contact with their Mage Lords dosen't make them any less aedpt in what they profess to do.
Just push gameworld and lore apart a little bit more and focus on the different between the servants and allies of House Telvanni and the general population it protects.
I view Sadrith Mora as a city of outlanders, agents, and businessmen who are already living under House bureacracy because only certain people are allowed there.
Vos and the shacks of Tel Branora are my idea of the Telvanni Dunmer, peasants who have nothing to do with the House.
Edit: Saxbass2: It's not that the other Houses 'give their people a chance.' It's that they espouse ideals and represent a culture that their people are a part of. Telvanni is divorced from the population. Sure, Aryon grew Vos a temple, but Aryon's a nice guy.