I have a quest and I don't know how to do it. So I went inside the institute, met Father and he has given me a quest to go get a runaway synth. I left the institute and decided to do a different quest, one that I've had on my pipboy longer. Its a railroad quest where I have to go to University Point and kill the enemies. But when I get there there is a group of synths but they are not hostile. No red dots on my radar.

So do I have to kill the synths to complete the railroad quest? Will it mean I cannot go back to the institute?

You can still do both. The synths will turn aggro on you after you shoot the first few, and then they'll display on your compass as targets. No lasting consequences from killing unnamed NPCs- the factions don't keep score and hold permanent grudges until later on.

It's amusing - right now, I am allied with the Brotherhood of Steel, the Institute, the Railroad and the Minutemen (and the Diamond City Guards, to be complete)... so basically the only foes that are aggro to me are supermutants, raiders, and wild beasts. It can lead to interesting situations like seeing Synths attacking Brotherhood soldiers and ~none~ of them being hostile to ME. Until, of course, i attack one of them. It does make the various missions for the factions easy --- the Railroad sends me to kill a courser, so I go to the location, find the Courser and he's all, like, "Hello, ma'am"... and I nod politely and then attack him at my leisure. Or the Institute sends me after some BoS blueprints... so I find the location, wave hello to the BOS soldiers, pick up the blueprints and head home.