First run through I used it, perhaps too much so. Second run through I haven't used it once. Third run through I'll probably strike a balance between the two and fast travel for selling runs or base management.

I fast travelled from the castle to sanctuary once and felt dirty afterwards.

I do fast travel rarely between two connected settlements when I'm setting them up for the first time.