I'm stuck in the institute. I talked to doctor Li and convinced her to help the borhterhood again. She installed something in my pip-boy so that I could travel from the institute back to the commonwealth and vice-versa. How do I travel back to the commonwealth?

Just fast travel and you will teleport to where you want to go.

You use fast travel. You have to talk to all of the department heads before you can leave for the first time though. You may have to speak with Father again before leaving too. I don't recall.

The institute is programmed to receive. https://youtu.be/lrfhf1Gv4Tw?t=3m56s

When you do get the ability to leave via fast travel, if my memory serves me correctly, you have to be near the elevator in the centre of the building

Have you used the Network Scanner Holotape? For me, even after talking to Li, I wasn't able to leave until I shoved the Network Scanner Holotape into a terminal. The moment I did that, I was notified that I could use fast travel.