If a mod introduces entirely new content, it generally can't contaminate the game once its gone. Although you should take care to be in an area not affected by the mod if possible just in case the save game is holding script reference to the mod.
If a mod alters vanilla content, then yes, it can leave behind lasting footprints. This can get especially nasty if the mod screwed around with vanilla scripts and possibly messed up the variable indexing in the process. That can definitely lead to CTDs when the game tries to revert back to the original but your save has a modified version of the script and won't let go of it. It's also possible to corrupt spawn points this way. As BFG99 also points out, if a mod disables vanilla stuff by script, those changes will remain permanent once made. Changes to more mundane things like NPC inventories shouldn't stick though.
Mods using PlaceAtMe calls in scripts that don't do proper cleanup (only possible with OBSE) can leave things behind that could mess things up too. The old version of Real Lights before AN took it over was notorious for this because it handled turning lights on and off with repeated use of PlaceAtMe calls, which is irreversible in the save. It caused bloat due to the number of records it could generate, and it could lead to CTDs if the forms it made contained copies of the scripts from the mod.
DLL based plugins don't normally leave lasting effects behind. OBSE itself is harmless. Oblivion Stutter Remover is harmless. Provided a DLL based plugin doesn't make alterations to the save, they can just be tossed aside with no ill effects. NONE of these things ever touch the original game executeable so that's not a concern. Streamline does not modify the executable, it's just a mod like any other, and if removed it won't leave anything behind.
Thank you for the tutorial
I'm currently using a mod called http://www.tesnexus.com/downloads/file.php?id=19847. And if I recall correctly, an older version of the mod used a resource called http://www.tesnexus.com/downloads/file.php?id=18160, which makes use of the PlaceAtMe function. The mod hasn't used that resource for some time now, but I had the mod back when it did
I'm going to try a new game without it, and see how high I can get Oblivion' memory consumption before crash.