It can ruin the role-playing elements, exploration, break immersion, and shorten the game.
I don't think players should have to take a fast-travel abstinence vow as in oblivion IE "don't like it don't use it"
I feel it should be "if you want it, you can earn it".
Personally i would like 5. the morrowind system combined with 1. the long quest for a magical item.
That way there's still fast travel and a sweet quest line that adds to the feeling of you becoming more powerful.
Somehow it wouldn't feel bad using it if it was something you earned and not just some cheap time skip gimmick.
I would like other zelda-like item quests too... Some extremely rare and hard to aquire....
Like a faster horse or unique gear like boots of blinding speed.