Well, you'll get an explicit notification when you reach the quest saying "By proceeding, ________ faction will be hostile toward you." They make a point of letting you know.
But since you asked, if you progress in the Mass Fusion quest the BoS will be hostile toward you. Completing Mass Fusion doesn't necessarily mean that you have to complete the game with the Institute, but you lose the option to complete it with the BoS.
Here is a http://www.gamesas.com/topic/1581396-how-to-make-friends-with-all-factions-at-the-same-time/ how far you can go before no return. Also for other factions if your planning to make friends with them all.
Cheers,
-Klevs
Mass Fusion - progressing through this quest will lock the BoS questline, reporting to BoS about this quest will get you kicked out of the Institute
End of the Line - since it requires you to destroy the Railroad, completing this quest will lock you out of the Railroad for obvious reasons (accepting it does not lock you out of it)
Airship Down - this quest will lead you to the Institute ending (unless you destroyed the BoS with Railroad, in which case End of the Line leads you into the Institute ending)