As the topic says is it possible to get into the college without joining? I need to get into there for the main quest and wouold prefer not to join as I am not a mage.

Yes, you can. There's a secret passage somewhere around the college.

Just go into Septimus Signus' cave behind the college. Just like the thieves' guild, you don't have to join CoW to advance main quest.

if you are sent there to get the elder scroll, when asked to prove yourself choose the option about being the dragonborn, that will get you in

oooo a secret cave. I like that it compliments my sneakiness.