That's not bad frame drop at all. Most people probably get worse than that.
Oblivion doesn't take much advantage of your multiple cores, and the processor load can get pretty high in populated areas, with lots of actors wandering around. Outside, grass is usually the culprit.
I always get a frame rate hit in the lockpick mini-game.