So before I decided to hit The point of no return with the brotherhood, I thought I would go to the Institute and wipe them out. I did this but now I cannot fast travel out. Has anyone else come across this?

Institute FT feature is supposed to become unusable if you become their enemy.

Have you taken the elevator to the Relay and used it to get out?

I've been contemplating this slaughter myself. Will it disrupt the BOH questline?

Elevator up to the Relay Room where you first arrived in the Institute.
Hit the red button on the console then walk into the relay room.

Doing so should also initiate the quests, Burning Cover and/or the kicked out quest, if you are still in the Railroad, and the way to a minute man ending as well.

I'm free! Thanks!