Here's my own two cents, for what it's worth.
I simply don't have the time to invest in retreading the same areas over and over again. I can dig the whole "added immersion" thing, but it doesn't work for me. With a full-time job and a baby on the way (which will be almost a year old by the time this game comes out) I'm lucky to get in one solid hour of game time, at once, on most days.
Even spending five or ten minutes walking through somewhere I've already fully explored just because there's no practical alternatives, is quality time I could be putting to better use in another game.
Now, I'm not saying to make it exactly like this, but I thought Red Dead Redemption was a good example of the best of both worlds. I almost never used the fast travel in that game, but it was nice for when I just wanted to get on with it.
Physically trekking, however, was not only actively encouraged - but I knew that no matter what, something interesting was going to happen. It was nice to have both options.
To me, it's not about an either/or proposition. One shouldn't be at the expense of the other. They ideally ought to co-exist just fine. It's not about eliminating (or even limiting) fast-travel, but making it so you'd want to use it as little as possible.
RDR is a great example of good use of fast travel - it's there, but you DON'T want to use it unless you are in a rush to complete an objective or quest because the world and quest design is so interesting in of itself, that by fast traveling in RDR you miss out on a lot of good stuff that is interesting. Much of the exposition of the game is also done via travelling outside of fast travel which is a great narrative idea. Instead of you getting a quest and then the guy saying "Meet me there!" and you instantly teleport there (bad design), he tells you to get on your horse and to ride with him. As you are riding, events happen and this character is talking to you. A lot of this contributed to making the world feel more real, alive, and concrete. It wasn't just a platform for gameplay, it was a tangeble world that encouraged you to explore it instead of "skipping it".
Great design right there. It's a game where you can fast travel but the game isn't designed around using it all the time - after all RDR's world is massive and if you fast traveled all the time you'd miss out on the entire point of the game. Which is why it's important for a game like TESV to have that "real" world feeling and to design it to be more dynamic, natural and unpredictable. Oblivion SHOULD have been like this but it failed because the game relied on fast travel from a design standpoint - there simply were not enough incentives to not fast travel. And any incentives there were, were rather weak. Which is why Morrowind is brought up so much in this argument, because Morrowind's incentives for "walking it out" were much much stronger and more well done than Oblivion's (varied environements, no level scaled enemies, interesting dungions,etc).
When there is incentive to not fast travel, then it can be safely used in the game and not be something that ends up being a crutch. In Oblivion, it was one giant massive crutch. And that's what we from the anti-fast travel side don't want. After all, why do we fast travel? Because it's boring doing the same thing over and over again that we find bland and can easily predict. So, make the world not bland, make it alive, and make it so we can't easily predict it. THAT is how you can design a game with fast travel and have it not be a crutch, but a tool when you need it.