He says "The guild can provide you with some training," meaning other training can be found outside of the guild. He then goes on to say "but the best trainers are usually very hard to find." This implies that that the "best trainers" are not connected to the guild and the "some training" that the guild provides. It seems very clear....
The fact that the only two people who ever mention him (one being a guild member) both seem to have no real clue of who he is, where he is, or at all what he is like, indicates that he isn't being kept track of or regulated by anyone, including the guild. Of course the guild could be keeping track of him and he could be affiliated with them, but there is absolutely no dialogue or really any evidence to suppose that this is the case. If anything, when Litte says "I personally wouldn't go looking for him myself" this gives us reason to think that he might actually be a dangerous person. It is also possible, as you say, that he is practicing and training illegally and the guild just doesn't want to put in the effort to go and arrest or kill him; but why would Alberic Litte, an advanced conjuration trainer and guild member tell you that you can go and receive training from him if it is illegal to do so? Is it really more reasonable to believe that he is somehow sanctioned by or a member of the guild even though there is no evidence to show this? Or is it more reasonable to simply say that he is a self taught mage practicing on his own and teaching people legally?
I am not sure what you mean by this. All I am saying is that alchemy is a part of the magical arts as evidenced by the fact that the Mages Guild, who's chief purpose is to provide magical services to the public, sells alchemical ingredients and potions. I have never seen the Mages Guild provide any non magical services to the public.
No I wouldn't know that they are all part of a society if I had never played the Unfriendly Competition quest and it would be odd of me to assume that they were involved in some society if the quest didn't exist. But since the quest does exist, we have direct evidence that such a society exists and are thus justified in believing in it.
You are advancing the claim that the Mages Guild has total control over who practices magic and in what fashion. The only direct evidence you have provided to suppose that this is true is the Unsactioned Training quest, which PrinceSchroob provided a reasonable alternative explanation for. You also cited the Recruit or Kill Llarar Bereloth quest in which the same woman asks you to kill a member of house Telvanni. But by your own admission, the Telvanni would be exempt from the Mages Guild's control over magic, so even you must admit that her actions in this second quest are illegal, thus adding to PrinceSchroob's claim that she is acting illegally in the first quest as well. The only other evidence you have is that it is never directly stated that the various trainers and mages I listed aren't members of the guild (though the case of Olyn Seran basically does state this).
In short, it requires fewer assumptions to believe that mages are allowed to learn magic on their own rather than make the leap and assume that every mage is either affiliated with the guild or is a criminal.