I beat the game siding with the institute. I beat the mission Nuclear Family. What happens now? Is there someone in the institute that will give me another mission or is it all over?

You have to forge your own path now. You beat the main quest.

There are various radiant quests for the Institute like helping synths return to the Institute after being captured by the raiders, collecting data for them, go to Diamond City to collect data from McDonalds, etc. Other than that, just wait for the DLC for more things to do.