Does it still help out the game even if it doesn't have any mods or very few?

Yes. The 4Gb patch will allow you to play for longer periods before Oblivion OOMs and crashes, and the stutter remover will provide a notable improvement to the stuttering which is thanks to Vanilla Oblivion's horrible optimization.

OSR is definitely worth having. Although I had to do a lot of manual tweaking to its settings before I found any improvement. I never saw the slightest bit of change after applying the 4gb patch. :shrug: But maybe my machine is good enough that it doesn't need the improvements if offers. For example, I have never had a problem with crashes.

Nothing to add. Both, the OSR with OBSE and the 4GB should be applied for most fluent gameplay.
Though I know lots of modders find this arguable, I wouldnt consider other mods like streamline essential.

kuertee also does offer a http://www.tesnexus.com/downloads/file.php?id=24606 which makes save games more reliable, though I do not have any statistics on this matter. but I use it too.

How do I improve OSR?

i also recomend http://www.tesnexus.com/downloads/file.php?id=39550. It basically puts the game in a fake full screen mode which actually runs the game smoother. the up side to this utility is it reads all your games and applies the same concept to them as well. of course if you dont want that it is configurable.

Alright, thanks guys, I'm installing all these mods right now! I have a question about Kurtee Clean up. Do I copy both of the ini files (kurteecleanup, kurteecleanuprestoreitems) into the INI tweaks folder or just one of them?

osr has a ini file ( if your using standard install directory c:/program files/ bethesda/ oblivion/ obse/ plugins) in there you will find several tweaks you can make. most people will raise the max fps from 30 i set mine to 60. for some this seams to add more stability.