I think the (Aldmeri) elves would argue that they didn't need guilds or colleges to understand and utilise magic, because they hold that they are more Aetherial than humans and are therefore more attuned to the source of magical energies, the sun (and probably a widdle bit from stars too). In the Mundus, the sun is supposed to be a giant tear in the fabric of the material plane (the Mundus) which was supposedly caused by Magnus, the Divine architect, fleeing from Creation back to Aetherius. Magic, as I understand the legend, is supposed to be Divine energy radiating into the Mundus from the solar hole.
Elves claim they are the descendants of the Old Ehlnofey, the Aedric beings who resented being bound to the Mundus and therefore remained more 'divine' than the Wandering Ehlnofey (said to be the ancestors of humans) and, as such, remained more in tune with the energies of Aetherius, making using magic, energy radiating from Aetherius, more easy for them.
I am inclined to believe that, aside from the sun-leaking source of magic, there is also a more terrestrial source. One that is perhaps easier for humans to manipulate. That being the tapping into the power of the Earth-Bones, the supposed remains of the Ehlnofey which became the foundation upon which Creation was built. I think Earth-Bone magic is used for more primal/earthy/rudimentary magics. Things like alchemy or reagent-based spells. I think it uses the power of divine energy that is an intrinsic and fundamental part of 'nature' and the 'natural world'. I think it's maybe easier for humans to use these types of magics (which we see more 'savage' groups like the Reachmen shamans or High Rock Wyrds utilising), because, as the purported descendants of the Wandering Ehlnofey, those Divines who resigned themselves to their binding to the Mundus, they are more attuned to these mortality-accepting entities and their 'life-force'.