Well how about you tack a percentage onto the enemy detection for this effect. Minus the night vision obviously. =P
You get near certain scary enemies and you would trigger a fear response of xxx enemy by xxx detection.
Each creature has a Scary Level. That plus them detecting you, hidden/caution/detected. Could give this motion blur.
The actual work involved is really tagging every monster/enemy. This could get tricky with a mod like Martigens but it would kinda make sense to add his mod also.
I won't be making this mod by the way. But anyone can use this idea. Its simple enough with FOSE I imagine. Knock yourself out =)