I think we have a winner here...

I downloaded the 9.11 drivers, I had 9.9 I think. BUT I don?t think this did it. Cause I also allowed Oblivion to make a new ini - still remembered to turn all video settings to the max, like they were.
Before I installed the driver, and made the new ini, I saved looking in certain direction, and noticed the FPS was 26 (it was a "good" direction).
After the driver install (didn?t even reboot since they don?t require you to nowadays) and the new ini: 40FPS. Frigging +14 FPS, which means my system runs the game now over +50% faster!
Overall, what used to be dips to 14FPS are now over 20, and so on - now this is acceptable, average travel FPS 25-30 and 17FPS as "bad" fps, not 12.

Thanks for all the suggestions friends, directed me in the right direction. Now I think I actually ffdshow, and it?s active, should probably disable it.
What is it, anyway, all I remember is that I think many movies require it? Is there a "safe" alternative, or do I have to disable it manually everytime I want to play?
EDIT: @shadme: this may not be relevant anymore, but I can?t alt tab without crashing so it?s kinda hard to say..