Summoning should have your typical Sorcerer's Apprentice risks. But I don't like the idea of wizard duels or whatever over the control of the summon.
If I remember correctly, some book describes how the conjurer has to call AND dismiss a daedra, AND keep it in its control. Those are the three parts of conjuring something. If you can't keep it in your control, it will run wild and you won't be able to dismiss it. And if you can't manage to dismiss the creature, you will eventually face that very problem, because you can keep your concentration up only for some time, unless maybe you're a master at this.
Might also be my imagination, but in my opinion that is how it should work. In any case, in order to snatch a summon from you, another wizard would have to make you lose control over it first; however once it's set free, he can't simply "take over" the control that's still figuratively lying around somewhere. If anything, he'd have to use a Command spell.
So yeah, it shouldn't be possible to simply steal a summon from you. I could imagine that using Dispel spells on conjurers makes them lose their control over the summon (if the spell's strong enough), at which point the free daedra should be a
third, unpredictable party in the quarrel, with their own goals. Using a Command spell then gives you control over them, but Command is not the same as conjuring something, so after the spell expires you still have to kill the daedra to send it back to Oblivion.