Yes, it will give you a warning, if you are trying to start a quest that will make a faction hostile.
Mass Fusion is one such mission.
For the Institute: I just do Battle for Bunker hill and some other minor quests.
The next big quest is Mass Fusion, which you will have to complete in favor for the Institute, for both the RR and the Institute ending (never done an Institute ending though).
With the BoS, I usually go as far as this:
Complete Liberty Reprimed
Complete Blind Betrayal
You have to complete Mass Fusion and some other quests for the BoS ending (which I haven't done so far)
Railroad - the last main quest I receive and maybe start but not finish, is:
Underground Undercover, you have to finish this one and a couple of other quests, if you want to go for a RR ending.
The Minutemen come back into play for me, after I get banished from the Institute, for the MM ending.