if you consider not buying the next console and becoming a PC player yes its worth it.
you need 4 Go of DDR3 RAM (DDR2 is now pricier)
you need 1 Go of video Memory and it needs to be a fast card cause its not all about the memory when we talk about Video Cards
you need either a 4 core processor with at least 2.2 Ghz of frequency or a 2 core with 2.8 or 3 ghz
you can probably find this for 600$
then you need a screen. i would go for 22''' like mine to go about for 160 to 200$
speakers. you can have a 5.1 system at logitech for like 45$ but id go for a quality 2.1 (i got a bose companion 3 wich is about 200$)
keyboard and mouse (depends entirely on the fancyness you want (from 30$ to 400$ for both) (i got a G15 keyboard and a MX revolution mouse that would go for about 240$ for both)
thats all you need your whole thing could range from 835$ to 1400$ with the specification i put up there and all depending on your other things you got with it.
If you intend to install a lot of mods, I'd also suggest buying an SSD or hybrid drive (both of which will likely be much cheaper toward the end of 2011) or, if you can afford two SSDs to set up in Raid0 configuration.
I have a pair of Intel X18 G2 SSDs in Raid0 in my laptop, and read speeds with crystal diskmark can get up above 500MB/sec. With vanilla oblivion it takes a few seconds to load a game and the load time for entering new areas is almost instantaneous.
However, after installing a crapload of mods, it takes 30-40 seconds to load a save game. I can't even imagine how long this would take with a regular hard drive. If you will play on a high difficulty setting and keep dying, that is a lot of time to wait.