If the lesser daedra are extensions of the main daedra's personality, the sphere of Peryite is the taskmaster of the lower orders of Oblivion. This means he is the taskmaster of the extensions of his personality, and of all the other daedric prince's personalities. It's already inane for mortals to worship him considering his sphere, but if the minor daedra are extensions of the personalities of the other daedra, this means he's actually in charge of all the minor personalities of the major daedra. So the prince gets their staple, and all the attributed sections are under Peryite's control. He's the kingmaker and rental locker of daedra, but just annoying to humans.
Molag Bal, the fire stone of Coldharbor, is in charge of the domination and enslavement of humans. His subset of personality is being the king of censored (you know what), which means Peryite is actually in charge of being the king of [censored]. Daedroths were assumed to be under the Molag Bal's control, but not only do they not fit with the domination, enslavement, or [censored], they wouldn't be under his control anyway. He would need a massive cult following to get anything going in Tamriel.
If a daedra is their sphere, then why do their spheres have different names? Or if a sphere depends entirely on mortal interaction, or entirely doesn't, what's the point? It is a rather weak view that a daedra actually is their sphere, I prefer the reasoning in Redguard.