[RELz] Oblivion Stutter Remover (OSR) : IV - Oblivion

» Wed Mar 09, 2011 7:13 am

You can lower uGridDistantCount regardless. It's just more important if you have RAEVWD. And more important if you don't have OSR.

» Wed Mar 09, 2011 2:58 am

You set the heap algorithm to 4 for the new TBBMM, correct? If so, the game ran smooth for me. Is there any feedback you would like?

» Wed Mar 09, 2011 2:04 am

That it worked is good to know. I know it works for me, but there have been a lot of reports of it failing so far.

» Tue Mar 08, 2011 11:50 pm

Yeah, I'm afraid it doesn't work for me either. It gets about 90 - 95% of the way through loading my saved game and then crashes.

Here's the log, if it helps any:

Spoiler

initialize0() running in thread f74
initialize1() running in thread f74
initialize2() running in thread f74
Critical Sections mode 3 (staggered prioritization)
MemoryHeap Optimization Algorithm 4: attempting to replace the Oblivion heap manager with FastMM4 debug / debug.dll
using hashtable size overrides
Hashtable size override: 00418DDB : 37 -> 149
Hashtable size override: 0045A866 : 5039 -> 133123
Hashtable size override: 004A2586 : 523 -> 2711
Hashtable size override: 004E610F : 37 -> 47
Hashtable size override: 004E612C : 37 -> 47
Hashtable size override: 004E8FD7 : 37 -> 739
Hashtable size override: 004F1B44 : 37 -> 127
Hashtable size override: 004F220A : 7001 -> 7001
Hashtable size override: 004F222E : 701 -> 901
Hashtable size override: 004F2B70 : 37 -> 127
Hashtable size override: 004F2A8B : 37 -> 713
Hashtable size override: 004F2AA8 : 37 -> 713
Hashtable size override: 004F2AEF : 37 -> 1301
Hashtable size override: 004F2B12 : 37 -> 1301
Hashtable size override: 006C5396 : 37 -> 83
Hashtable size override: 0067FD35 : 191 -> 3019
Hashtable size override: 0067FE5F : 191 -> 2021
Hashtable size override: 006C5674 : 37 -> 299
Hashtable size override: 00714752 : 59 -> 239
Hashtable size override: 00769BEB : 37 -> 297
Hashtable size override: 009DBF03 : 131213 -> 905671
Hashtable size override: 00B06140 : 131213 -> 905671
Hashtable size override: 009E26F3 : 37 -> 297
Hashtable size override: 00A10DB3 : 37 -> 297
hook point 1: hooking call to main game function
timeBeginPeriod: 0 -> 1
Improved_GetTickCount seems to be working
Heap Initialization (268435456, 0)
CS 0x00B32B80 (caller 0x40100A, path 0, spin1000), by-object, set to mode 3
CS 0x00B32B80 (caller 0x40100A, path 0, spin1000), by-object, set to spin 1500
CS 0x00B33800 (caller 0x40100A, path 0, spin1000), by-object, set to mode 3
CS 0x00B33800 (caller 0x40100A, path 0, spin1000), by-object, set to spin 1500
CS 0xFF244380 (caller 0x701748, path 0, spin1000), by-caller, set to mode 5
initialize3() running in thread de0, renderer at FF244200
thread 0DE0 assigned PT 04597F18, serial# 1
thread 0478 assigned PT 04564FD0, serial# 2
thread 0B3C assigned PT 04565080, serial# 3

» Tue Mar 08, 2011 11:55 pm

I haven't done extensive play testing, but it loaded up fine. I'll post results after I've played for a while.

» Tue Mar 08, 2011 9:37 pm

tbbmalloc crashed and burned for me as well, before it even reached the main menu. It didn't even survive long enough to log anything.

» Tue Mar 08, 2011 8:35 pm

Well, I guess I'm the odd man out in that I can get in-game, but it crashes on fast travel from right out of the tutorial dungeon in a new save, or if I run around long enough from existing saves. Usually 10-15 seconds.

» Wed Mar 09, 2011 8:48 am

Oops! I goofed, SkyRanger. I'm not going to say the mistake I made cuz it's embarrassing. The new heap won't load for me either. Sorry.

» Wed Mar 09, 2011 4:35 am

The new heap works fine for me. I've just played a good two hours with it.

5 still feels smoother, though.

Edit: Actually, 4 is really starting to grow on me. There are several places I can usually count on some stutter to test, and this new build, along with tbbmalloc.dll, have them almost down to no stutter at all.

» Wed Mar 09, 2011 12:35 pm

Just chiming in to say that the new heap replacement TBBMM in OSR 4.1.19 works for me too, on a Windows XP 32-bit.

Skyranger-1, what would be your thoughts regarding this excerpt from the readme contained in the TBBMM zip download from http://www.zachsaw.co.cc/?pg=borlndmm_tbbmm:

tbbmm.dll is from Intel's TBB version 2 commaligned release.
tbbmm.v3.dll is from V3 commaligned release update 2.

If you wish to use v3, replace tbbmm.dll with tbbmm.v3.dll.
Note that v3 is slower but is less likely to fragment.
If you do not notice fragmentation problem with v2, you'd
want to stick to it to benefit from the extra speed.

I notice that the tbbmalloc.dll download is at version 3. Is version 2 indeed faster than version 3? Or would version 3 be the recommended one to use?

» Wed Mar 09, 2011 3:05 am

c0demeist3r: Beats me.

everyone:
From hearing andalaybays earlier reports of problems and motos comments on them, it sounds like both are seeing relatively severe stutter-on-grid-load issues even when using OSR w/ heap replacement. andalaybay even talked about having to take his hands off the keyboard for 5 seconds to let the game catch up.

I see that kind of thing w/o OSR. A ton of it if I enable RAEVWD. Noticably less with uGridDistantCount lowered to 15. A bit of it with OSR but w/o heap replacement. With OSR and heap replacement and uGridDistantCount lowered... I guess it's still there if I go around at speed 330 & athletics 330, but its subtle enough I miss it if I'm not paying attention. And I don't have an SSD or anything special (I think even NCQ is disabled), this is an an old ~~Athlon 64 X2 @ maybe 2.8 GHrz~~ 3 GHrz Core 2 Duo w/ a GF9600. Any heap algorithm other than 2 is sufficient to fix it for me, at least when combined with the CS stuff.

Are other people seeing significant stutter-on-grid-load issues? With or without RAEVWD or equivalents? With or without reducing uGridDistantCount? I've been kind of assuming the problem was completely solved since it went away for me.

» Wed Mar 09, 2011 4:46 am

I haven't lowered my uGridDistantCount yet - I will do so since I couldn't get the new MM working.

» Wed Mar 09, 2011 4:35 am

Re: Stutter for grid loads.

Yes, I still get that. It's nowhere near as bad as it is when OSR isn't loaded though. Obviously I'm using the full setup for RAEVWD. My uGridsDistantCount is set to 18, same for the tree one. I've got AN configured to fog the view past that.

What's NCQ?

» Tue Mar 08, 2011 9:51 pm

NCQ - Native Command Queuing. A quick look at wikipedia:

Native Command Queuing (NCQ) is a technology designed to increase performance of SATA hard disks under certain conditions by allowing the individual hard disk to internally optimize the order in which received read and write commands are executed. This can reduce the amount of unnecessary drive head movement, resulting in increased performance (and slightly decreased wear of the drive) for workloads where multiple simultaneous read/write requests are outstanding, most often occurring in server-type applications.

In essence, it is a technique in hdds that reorders commands passed to the drive for optimum performance. To enable NCQ, the hdd controller must be in the AHCI operating mode (usually set in the bios) and must have proper driver support installed. Windows XP and later only.

Most probably newer installs nowadays don't have to worry about enabling this as this is on by default, provided an updated Chipset driver and drive controller driver, where applicable, is installed. Also, it would help if your SATA drive is not locked to 150 MB/s (some SATA II drives still come with a jumper that has this set, for backwards compatibility with the first SATA spec), so that SATA II mode is used, which is 300 MB/s theoretically.

» Wed Mar 09, 2011 4:54 am

everyone:
From hearing andalaybays earlier reports of problems and motos comments on them, it sounds like both are seeing relatively severe stutter-on-grid-load issues even when using OSR w/ heap replacement. andalaybay even talked about having to take his hands off the keyboard for 5 seconds to let the game catch up.

I see that kind of thing w/o OSR. A ton of it if I enable RAEVWD. Noticably less with uGridDistantCount lowered to 15. A bit of it with OSR but w/o heap replacement. With OSR and heap replacement and uGridDistantCount lowered... I guess it's still there if I go around at speed 330 & athletics 330, but its subtle enough I miss it if I'm not paying attention. And I don't have an SSD or anything special (I think even NCQ is disabled), this is an an old Athlon 64 X2 @ maybe 2.8 GHrz w/ a GF9600. Any heap algorithm other than 2 is sufficient to fix it for me, at least when combined with the CS stuff.

Are other people seeing significant stutter-on-grid-load issues? With or without RAEVWD or equivalents? With or without reducing uGridDistantCount? I've been kind of assuming the problem was completely solved since it went away for me.

I'm currently using heap algorithm 3 because, although it may not remove stuttering as well as 1, it provides the most stability for me. I have my uGridDistantCount lowered as per Streamline's recommendation for Streamsight. I think 7 or 8. I still get stuttering, but it is nowhere near as bad as it is without OSR.

» Wed Mar 09, 2011 4:22 am

TBBMMmalloc works for me. BTW, yes I have reduced my GirdCount and TreeRange to 10. RAEVWD, 5 girds to load. Reducing Gridcount/treerange to 7 gets me more FPS (3-11) and reducing GridstoLoad to 3 and Count/TreeRange to 5 gets me seamless loading.

SR-TBmalloc supposed to be resizing itself?<---wrong log.

SR-old athlon x2, huh? If you have 2.8ghz, you're using an Athlon X2 4800+ (3rd in line behind FX-57 and FX-60, just ahead of my Opteron-same die as yours.) and getting better performance? How low DO you have your resolution set?

» Tue Mar 08, 2011 11:53 pm

I tried replacing the TBBMM with the v3 DLL, and it doesnt load at all. I think its meant to be loaded through the included wrapper or something. Anyways, tbmmalloc is still causing CTD on any attempt to load a save, which isnt convenient to use.

Also, is there any benefit to specifying a certain address for the heap main block through the experimental setting?

» Wed Mar 09, 2011 3:27 am

moto: 2.8 is a 4800? Then I'm confused. Ah, yes, I am very very confused. Jeeze, I must have been tired. Or really confused. I said that already. The Athlon 64 X2 was my previous box, it exploded itself (visible spark-fire coming out of the power supply) after my last move, and it wasn't 2.8 GHrz either it was like 1.8, I remember now. The current box is a 3.0 GHrz Core 2 Duo, almost exactly twice as fast for the CPU-intensive stuff I do sometimes. Though still a bit old.

SkullKnight: In practice, no, I don't think it's useful. In theory... it gives you more control, to attempt to avoid it doing something stupid like needlessly fragmenting address space (useless because you can't actually use that control effectively, lacking any ability to see where anything else gets loaded).

» Wed Mar 09, 2011 2:02 am

Hey... that alternate BorlandMM.dll someone was linking a bit ago... does it actually improve performance any? No one has had any problems with it?

@Skyranger-1
I'm behind on your thread, I'll post (and edit if necessary) as I go along.

Well I can't say that this new BorlandMM.dll actually improves performance over the older version. It's just running well and stable on my system, so no problems for me with heap algorithm 1.

Edit: OK, now finished reading the thread. So people have been moving to TBBMM...

I was wondering if it would help the discussion of what works for whom if we stated our rig details in the signature, at least temporarily. The notion of XP vs. Win7 is an important one, maybe platform too (AMD vs Intel)...

I spent the weekend still playing and testing with 4.1.18 and the FastMM vs. ThreadHeap2. And as much as I would like to love ThreadHeap2, it will just not work as well as FastMM does. With TH2 I got 2 sessions in for close about 60 min each, and then sudden crash out of the blue. "All blocks exhausted" was not the issue. Memory usage by oblivion.exe at that point was only at about 1.2 GB (I have LAA enabled). Also one session may play really smooth, the next one may play with certain stuttering, sometimes even quite badly like andalaybay described (but not quite AS bad).

With FastMM I can essentially play as long as I want and it is generally very smooth.

However, what I noted with some surprise is that with TH2, the game would overall somehow make better use of the two CPU cores. With FastMM they would usually split 50-50 each, but with TH2 it would usually be closer to 130 - 150 overall (where 200 is both cores at 100%). That was with Experimental/LockToThread set to 1 (for both heaps FMM and TH2).

Could this actually be or was I dreaming or caught TH2 in some particularly good moments???

I'm going back to TH2 one more time tonight. I hope to reproduce my stuff again and come to the final conclusion of which heap algorithm to use with my system.

» Wed Mar 09, 2011 7:37 am

Skyranger-1 - I have held back for a while because so many updates with the tripple wip, I kept updating but never really got down to testing (which I still haven't but made a discovery - it may not be new info for yourself but will post it just in case)

Also held back because on the laptop after the last couple of tripple wip versions I have experienced slight stutter on both my laptop and desktop, and yet everyone seems to be experiencing the same performance (mostly unable to discern any improvement from previous versions .. but no worse), which made me believe there was something different in my setup.

There was, a few ini tweaks which I had not re-applied after refreshing the oblivion.ini to optimise setup, here's the relevant ini settings and notes I gathered around the bazaars which are relevant ...

Spoiler

iPreloadSizeLimit=26214400 - This setting appears to determine the maximum amount (in bytes) of RAM allowed for preloading game data. The higher the value, the more chance you have of reducing stuttering. The default value equates to around 25MB (divide the setting by 1024 to get KB, then by 1024 again to get MB).

For those with 1GB of system RAM, try doubling the variable to 52428800.
For those with 2GB, try double again at 104857600 (100MB).

You can raise these further to experiment, however note that raising this to a large amount doesn't force all the game data to sit in RAM, and can actually cause crashes. I suggest the maximum anyone should set this to should be around 262144000 (250MB), even for 2GB of RAM. Make sure to raise your Cell Buffer values accordingly (see above).

*Jaga Telesin's explanation:
"iPreLoadSizeLimit is one of those things people thinks work like a regular "cache", when in fact it does not. Larger values in this setting are WORSE for players since the 1.2.x patch. Martigen and I spent a week straight looking into this and a few other performance related INI values, and in fact it's better to keep it lower if possible. The right number is something you arrive at based on what you are using. The default value of 26214400 can be equated to one "unit" of cache. To find out how many units you need, refer to the following list:
(+1) Vanilla Oblivion + 30 or less mods (all small)
(+1) OOO, Fran's, Warcry (not FCOM)
(+1) MMM (not FCOM)
(+2) FCOM
(+1) QTP1 (or similar normal-size texture pack)
(+2) QTP2 (or similar medium-size texture pack)
(+3-4) QTP3 (or other large-size texture pack)
So for example, someone running Oblivion, a few small mods, OOO and MMM would need a setting of: 1+1+1 (3x default) or 78643200. Generally this floats up and down by 1x, so that person might be able to get away with 2x instead, which is BETTER than 3x. You know you are too low when just after changing it and reloading the game, you start stuttering badly.

A high end machine with 2gb of RAM doesn't need to set their cache at 1gb, that's just insane. They need instead to evaluate what they run and see what setting value they need. If they had Oblivion+mods, FCOM, and QTP3 they would need: 1+2+3, or around 6x default (157286400), which is roughly 153mb of Oblivion cache.

Everyone's experience is unique, so people will have to play with their settings to get it just right. Too large and you will get problems, too small and you'll get different problems with similar symptoms (stutter and lag). Finding the sweet spot is what tweaking this entry is all about. My numbers are just a rough guide, and are by no means the final word on the setting."

Using the ones highlighted changed to the recommended (and tied in with Jaga Telesins recommendation for our load order setup, I could go for 2gig settings but not necessary, and as Jaga says too much can be detrimental also) ....

Using the latest OSR tripple wip, I am now running completely smooth again on both machines, mostly at default settings....

- Only changed to use Threadheap2 on both machines - (previously this would not play well on the laptop, I was using FastMM with the new .dll, but after Cell buffer and IPreload settings changes, no problems with Threadheap2)

I have x32 bit OS on both machines, XP and win 7, I have never needed to change Heapsize, on both machines it remains 450.

Lowest specs here on the laptop - Core2Duo, cores @ 2ghz, 4gb ram, Sata II HD, Geforce 8600 GS (only 256 onboard graphics memory) - This is the Win 7 x32 machine

~~Cell buffers and~~ IPreloadsizelimit as detailed above make a noticeable difference combined with OSR (marvellously smooth in our case).

Edit: One other change I do to default - bFix64Hertz = 0, I dont have any problems with this off and the faster accumulation of save file A-bomb bugs me

» Wed Mar 09, 2011 7:10 am

I've never had any noticeable improvement from running with altered buffer settings and changing them now still makes no difference whatsoever. So they're certainly not a universal solution even with LAA.

Might have to try with the 64Hz thing off. I've never touched it, always left it in the default setting of on. Abomb buildup bugs me too especially when Bash says it's 37% but animation related things clearly have already blown themselves up - like my recent issue with horses.

Rig specs (my sig is too jammed to add this):

Processor: AMD Phenom II X4 955 3.2Ghz
Memory: 8GB Corsair Dominator
Motherboard: Asus M4A79T Deluxe
Graphics card: Sapphire Vapor-X Radeon HD 4870 2GB
Sound card: Sound Blaster X-Fi

» Wed Mar 09, 2011 12:41 pm

@Skyranger-1
I'm behind on your thread, I'll post (and edit if necessary) as I go along.

Well I can't say that this new BorlandMM.dll actually improves performance over the older version. It's just running well and stable on my system, so no problems for me with heap algorithm 1.

Edit: OK, now finished reading the thread. So people have been moving to TBBMM...

I was wondering if it would help the discussion of what works for whom if we stated our rig details in the signature, at least temporarily. The notion of XP vs. Win7 is an important one, maybe platform too (AMD vs Intel)...

I spent the weekend still playing and testing with 4.1.18 and the FastMM vs. ThreadHeap2. And as much as I would like to love ThreadHeap2, it will just not work as well as FastMM does. With TH2 I got 2 sessions in for close about 60 min each, and then sudden crash out of the blue. "All blocks exhausted" was not the issue. Memory usage by oblivion.exe at that point was only at about 1.2 GB (I have LAA enabled). Also one session may play really smooth, the next one may play with certain stuttering, sometimes even quite badly like andalaybay described (but not quite AS bad).

With FastMM I can essentially play as long as I want and it is generally very smooth.

However, what I noted with some surprise is that with TH2, the game would overall somehow make better use of the two CPU cores. With FastMM they would usually split 50-50 each, but with TH2 it would usually be closer to 130 - 150 overall (where 200 is both cores at 100%). That was with Experimental/LockToThread set to 1 (for both heaps FMM and TH2).

Could this actually be or was I dreaming or caught TH2 in some particularly good moments???

I'm going back to TH2 one more time tonight. I hope to reproduce my stuff again and come to the final conclusion of which heap algorithm to use with my system.

iThreadsFixedToCPUs can artificially increase the CPU usage numbers. So can iReduceLongSleep and bRemoveShortSleep. The increase resulting from those three does not necessarily mean anything good.
ThreadHeap2 doesn't do anything special. TBBMM must do something special to get the kind of performance numbers it gets.... TH2 is already close to maximum possible efficiency for the primitives it's built on top of, TBBMM is enough better it must be doing something special.

...snip...

Broadly, alt3rn1ty sees less stutter with increased buffer sizes, Arthmoor doesn't. K.

Assuming my understanding of what's going on is correct, the Abomb buildup thing should only be happening if you've increased or eliminated MaximumFPS. Or are in menu mode. Though... I think someone recently asked me to make MaximumFPS apply in menumode as well, it sounds like a good idea.

» Wed Mar 09, 2011 11:08 am

alt3rn1ty, interior cell buffer and exterior cell buffer in oblivion.ini should definitely be left alone due to how they work..

» Tue Mar 08, 2011 9:50 pm

Forgot to mention not using LAA with my setups.

The A-Bomb faster accumulation is not really a problem (easily rectified via Wrye Bash anyway), its just a personal preference so I do not have to check it so often for all users multiple saves in the house ( 4 players using the same machine wih different user accounts x 20 streamsaves each and their own manual saves = tedious

), I just mentioned that I turn it off in case it may have any bearing.

alt3rn1ty, interior cell buffer and exterior cell buffer in oblivion.ini should definitely be left alone due to how they work..

Uhm, those settings we have used for years, I have a very stable/ctd free game which all users here can play for as many hours as they like - Is there anything wrong with the information Jaga Telesin wrote in the notes I spoilered? (I do realise there are a fair few from the old tweakguides which no longer apply since updated patches to Oblivion were released ... But this is one of the few we still use successfully so long as you find the "sweetspot" for your machine and load order, and purge cell buffers with streamline)

» Wed Mar 09, 2011 9:46 am

@Arthmoor,
I see you have an AMD based rig whereas I have an Intel... wondering if this matters as to what works for you vs me vs anybody...

Probably it should not because we are using Win7 (while Skyranger is still on XP I think) but who knows...

@Skyranger-1
I think I have now officially thrown in the towel on the TH2 heap...

As you may remember I now have overclocked my PC. However, just to level the field again yesterday I reset it all to stock speeds and went for one more round of TH2. Again, after about 80min of gaming it suddenly crashed on me again with RAM usage of about 1,25 GB at that point, like I had on previous occasions. This was with Heap Size 650, but "exhausted all blocks" was not stated in the log file. Up until that point however, performance with TH2 was excellent, smooth and all and without any hickups to note. Again I'm wondering if the TH2 is working better on AMD and FastMM maybe better on Intel systems.... :shrug:

Admittedly it could still be an issue with my installation and modlist...

...so I wanted to test with FastMM again today, but I am doing a round of re-pyffification as there is a new Pyffie version 2.1.7 and took away all my gaming time today. I will come back when I have tested with FastMM again. Should probably grab OSR 4.1.19 then.