Especially in a game series like the Elder Scrolls, it can be a matter of weeks (IRL) between the time that I talk to an NPC about a quest, and actually getting around to trying to do it. In that time I'll certainly have forgotten any directions the NPC may have given me - and often even which NPC it was that I'd talked to about it in the first place.
This is a good point, which could be averted by your journal listing the same directions that the NPC gave you when he or she was telling you where to go, with the addition of your map highlighting exactly where you and the NPC were when you received the quest. Then it would simply be a case of travelling to the spot where you were given the directions, and then following the directions in your journal! That way it wouldn't matter if you couldn't remember who gave you the quest, or where you were when you received it- you would just look at your journal to find the exact spot where the quest was given to you, before travelling there and then using the directions in your journal to find the desired location.