The incentive for the player to go chasing after a fight needs to be removed as well for it to be scary. I think it would rather undermine the point of it if you had to harvest some imp galls and they were 'stalking' you. I'd be looking for them outright.
These horror elements only work when the player is in the vastly weaker position. Why should I be scared of something that I can kill with ease, and then profit from its demise?
Yes, and for a good portion of the game you would be vastly weaker.
Fear of the dark is really fear of the unknown. If you're level 1 and you hear a noise, you'd be best off getting out of there because really anything will handle you.
If you're level 15 and you hear a noise, you've got a decision to make. You are pretty strong, and the thing that just made a noise is PROBABLY weaker than you. But what if it's stronger? What if it is weaker than you, but attacks at the perfect moment and gets a sneak critical? What if there're more than one?
You wouldn't know the answer to any of these questions, so as long is there are a few baddies badder than you, you've always got a reason to be afraid.
At level 35, there's not much to be afraid of anymore, which is the way it should be. But if you become less godly in Skyrim than Morrowind and you're not invincible, you'd still feel the need to be cautious, because sneak criticals change fights. And there could always be more than one of them.