I playing Oblivion through OOO 1.34 with Wrye Bash installed. When I looking a certain NPC like Dremora my FPS get hit badly down to single digits. When I look away my FPS goes up where it was before it had that hit. I thought it was the number of NPC'S, if it was that FPS will be still low so its graphical. The textures that I using are Qarl's Texture Pack 2 512 MB and Insanity's Improved Armoury Compilation.

Is there any way to fix this up or a link?

What are your graphical settings?

Game Screen Resolution is 1680x1050
AA at 4 with large Textures.

Are you using antistropic filtering, playing in 3rd person, or using the wyre bash tweak for all armor to be wearable?
For me all armor wearable always caused some pretty severe fps plummets in oblivion planes. I'm not really sure why