So I have installed Q's high res textures, natural environments, open cites, and a few others. I got it set to Ultra High settings and it's running pretty good. There is some stuttering here and there. However, I thought I read that there was some mod out there that could optimize the engine without lowering visual quality,... something like streamline. Do you guys know what I am talking about? If this mod exist then does it work well?

Streamline and Oblivion Stutter Remover can help. However the engine is still pretty old and terribly optimised. So you'll get some benefit but it won't be life changing.

Are these two things they I can use together or is it one or the other.. if so then which one would you recommend?

They can be used together afaik

It does help to look through both readme's and tweak the ini settings to your liking. Streamline as 4 functions, save, purge, fps settings, and fog. I just use purge and save. It also helps a lot to mess with OSR's heap replacement, can certainly help for many people.

For a system like mine (see my sig) what heap setting would you recommend for OSR?

The fast one that requires an external .dll.