So i Went to the Institute with the help from Minuitemen and kinda said "maybe" to father, so that i could get the mutant Serum for that super mutant. Then checked how to progress the quest, and i said i needed to be banned from the Instute or something like that. Is there a way to do this without killing a named character?

Please excuse Spelling, grammar mistakes and such.

You can get banished after the Battle of Bunker Hill. Let the synths go and tell Father you want out when you meet him on the roof of CIT.

If you're with the Railroad you're going to have to tell Des you got banished from the Institute as well.

Thanks.

Doing it this way can cause a glitch at Bunker hill that makes all the residents adopt the cowering position and never get up.

Yes. You can do all the institute quests up to Mass Fusion. (Are you also doing quests with the BOS?). When you are ready, Talk to Allie to get the Mass Fusion quest but THEN INFORM THE BOS. You will be able to do Mass Fusion with them, and this will "banish" you from the institute. You can do all the BOS quests up to and including Blind Betrayal. You can talk to Maxom but DON'T talk to the captain of the Prydwyn. That's the point at which you stop the BOS line from moving forward. You can do all the Railroad quests up to and including all the Dead Drop quests, but don't talk to PAM after you get the ballistic weave perk.