I just left the institute after talking to all the researchers, but the minute men (who built the teleporter for me) only have the same random quests for me and I don't see anything else that looks like a main quest. Do I have to do random tasks for the institute now or what? How do I progress?

You have to either go with the Institute, or make them hostile which triggers the next phase of the minutemen questline.

I think you cand still join the railroad or BoS if you want also.

How do I make the institute hostile? (do I have to just teleport there and start a murder spree or something?) I had already joined the railroad before I went to the institute but Desdemona won't talk to me any more (or let me turn in my latest quest for them) except to say I should switch to the railroad instead of the Minute Men.