Not sure what OS you're using, but for Windows 7 (64 bit) what *I* do is to uninstall anything like DLC (e.g., Knights of the Nine) first, then Oblivion itself. Then I also delete the Oblivion directory (where the game was installed), along with My Documents\My Games\Oblivion and Users\[Username Goes Here]\AppData\Local\Oblivion - note that AppData is normally Hidden. Well it is even now, for me, but I can view those files because I chose to be able to do so in the Windows folders options.
That's it. And, I suspect, you would only really need to do all of that except for the AppData bit - probably not necessary, but I'm fussy/odd like that.