Save restrictions.
Only because these types of games--FO, TES--are very prone to glitches. That should be a very real consideration. Otherwise this would be ok. There have been a few times just being stuck somewhere in power armor that has caused a 5 ish minute time warp. Imagine something like this setting you 15+ minutes back? No thanks.
Significant Lower Respawn Rates.
This one isn't unfolding in my head very well. Once most of the quests have been completed the only thing left to do is killing things. Hard to say for sure if this is a bad idea before trying it. I hope it's better in practice...
On a side note:
I already never fast travel anywhere except back and forth from you know where... Or via you know where...
So... How is that going to work now?