Oblivion And Physical Memory Usage

Post » Sat May 28, 2011 12:14 am

At first I was under the impression that the game was just badly designed and leaked memory and that mods just made it more so. Only recently have I heard that mods themselves may be the cause. (is that the consensus now?)
It's not entirely clear, but IIRC everyone I have talked to who sounded like they were leaking memory fast played a heavily modded game. And I believe that some people leak memory MUCH faster than others. That suggests that mods may be related, but no specific mod has ever been identified as a source of memory leaks, so maybe not.
User avatar
Nicole Coucopoulos
 
Posts: 3484
Joined: Fri Feb 23, 2007 4:09 am

Post » Sat May 28, 2011 3:40 am

At first I was under the impression that the game was just badly designed and leaked memory and that mods just made it more so. Only recently have I heard that mods themselves may be the cause. (is that the consensus now?)


Qquix Rock Ships mod was the first I came across that made me suspicious of a memory leak. It was fairly obvious since the Oblivion.exe process memory usage continuously would rise 1mb per second. After talking with Qquix a bit about it, not all memory leaks are as easily apparent.

Thanks for all the effort that went into figuring this out : ). Seeing the results you got from your testing, I can't wait to throw this patch on and start playing! Hopefully that list of mods that leak memory will get started soon too!

Quick question; I'm not all that experienced with modding. Are memory leaks caused by scripts? And if so, is there a way to fix the vanilla OB leaks?


I honestly just started learning about memory leaks myself, so I can't give a ton of insight on the subject. Since the memory leak I tested with was caused by a script, it leads me to think that scripts are the most likely culprit to contain a memory leak. Maybe some of the talented scripters can offer some insight?
User avatar
Soph
 
Posts: 3499
Joined: Fri Oct 13, 2006 8:24 am

Post » Sat May 28, 2011 12:02 am

Thought of a few ways I could improve this topic a bit...

I could run memory ceiling tests for various OS's and provide solid number that players can base from.
For example...

WinXp 32
3gb

Ceiling w/o patch
1.6gb

Ceiling w/ patch
2.6gb


It would take a bit of time to install all of the various OS's, but I think it would really make it easy for players
to better pinpoint if there existing CTD issues are happening because of a memory ceiling.

On another side note, I remember reading that Windows7 had made improvements on how it handles graphics memory intensive programs.
What I remember reading was that once video ram is exhausted, Windows7 would allocate memory from the available memory pool to be
used for video ram!!! I need to find the dumb article again, but if this is true that means that the LLA patch not only adds further physical ram stability, but would also add further video ram stability! Please don't quote me on this yet. I have the Microsoft article bookmarked at home, so I will post it once I have a chance.
User avatar
Enie van Bied
 
Posts: 3350
Joined: Sun Apr 22, 2007 11:47 pm

Post » Sat May 28, 2011 1:19 am

I suppose there is more than one kind of Leak

One that may be caused by modders is the well discussed savegame bloating, where a mod generates new information and does not clean it up when it is not necessary anymore. These are easier to spot, as, sooner or later, players will notice their savegame size is increasing and will warn the modder.

Another one is the 'classical' memory leak that haunts most software developers. This, AFAIK, cannot be directly caused by modders.

I did some investigation after far327 noticed the increasing memory usage when using my Rock Your Ship mod and I did isolate a few situations where, using some specific script functions, seems to cause memory leak. After replacing them, the memory increase was reduced to less than half than before. So far, I could not isolate the cause of the residual leak. Maybe it is not even caused by scripts at all. Who knows?

This [very amateur] investigation left me the impression that some script functions, under some circumstances, may leak a few bytes of memory. Therefore, different players, with different sets of mods, may experience different memory increases, depending on which mods use which functions.

But, as I said, this is an impression, as I am not qualified to anolyze it further. It is not clear if the cause is OBSE or vanilla functions, but I reported it to the OBSE team as they may anolyze it, maybe confirm it and, being the case, fix it. They are quite busy, as usual, and we have to wait until they have a chance to diagnose it.

It is clear from the posts in this thread that Oblivion uses more and more memory over time. As with any software, there may be leaks in any function: graphics processing, havoc processing, weather processing, whatever. Hard for us, outsiders, to find, unless we stumble on one by chance, as was the case with Rock Your Ship extensively using a leaking function and far327 noticing it.
User avatar
YO MAma
 
Posts: 3321
Joined: Thu Dec 21, 2006 8:24 am

Post » Fri May 27, 2011 6:30 pm

Found this article on how to http://msdn.microsoft.com/en-us/library/dd744766%28VS.85%29.aspx
It may have some very useful info for those out there with a bit more knowledge on the
subject. I will look it over too as time permits.

And here is the article I was talking about... Since Windows Vista (Thought it was Win7)
graphics memory reporting was changed. It's actually pretty interesting...
http://www.microsoft.com/whdc/device/display/graphicsmemory.mspx (download the GraphicsMemory.doc file on the upper right corner.)

I'll have to check my total available graphics memory in my device manager when I get home.
In theory, if the total available graphics memory is larger than the dedicated graphics memory,
that should mean that the difference in size would be what is available for additional video memory if you were to exceed your dedicated amount.
My guess is when using the LLA patch, that extra available amount of memory would get
utilized by which ever component needed it first. Be it your graphics card, or your system itself.

Just a theory at this point...

I'll follow up on this once I find a way to actually test it.
User avatar
Genevieve
 
Posts: 3424
Joined: Sun Aug 13, 2006 4:22 pm

Post » Sat May 28, 2011 8:13 am

Just for the lulz of this thread, I tried to open the game, and duplicated myself 400 times. I threw fireballs at my clones, and as they tried to shoot back, they hit some of the other clones, and then the oblivion AI behaved as we know it... I started out with a 300 MB RAM usage. Ran alittle around in the shivering isles.. 500 MB.. duplicated my self alot.... 1000 MB... duplicated myself even more... 1500 MB.... but for some reason, I couldn't get it to use more than 1,5 Gb ram. No matter how much I tried to duplicate myself and cause AI chaos, it simply WOULDN'T go above those 1500 MB.
My mission to get oblivion to crash, was a failure :(
User avatar
Luis Reyma
 
Posts: 3361
Joined: Fri Nov 02, 2007 11:10 am

Post » Fri May 27, 2011 8:07 pm

Just for the lulz of this thread, I tried to open the game, and duplicated myself 400 times. I threw fireballs at my clones, and as they tried to shoot back, they hit some of the other clones, and then the oblivion AI behaved as we know it... I started out with a 300 MB RAM usage. Ran alittle around in the shivering isles.. 500 MB.. duplicated my self alot.... 1000 MB... duplicated myself even more... 1500 MB.... but for some reason, I couldn't get it to use more than 1,5 Gb ram. No matter how much I tried to duplicate myself and cause AI chaos, it simply WOULDN'T go above those 1500 MB.
My mission to get oblivion to crash, was a failure :(
So if I'm understanding correctly, you tried to overload the Oblivion.exe memory ceiling by adding a bunch of on screen A.I. and effects through the console?
It's certainly a different testing method, not a bad attempt though. But the difference is your game doesn't contain a memory leak, or at least not within the borders of the game in which you had loaded. And my guess is you don't have enough resource intense mods loaded to overwhelm your systems memory. However, say you started a new quest that was associated with a mod that contained a leaky script? Then you would notice an apparent leak.

Really the point of my topic is to educate players that have heavy mod loads understand why they are experiencing a CTD after so many minutes or hours of game play. So, your game load may not fit the criteria for this specific type of CTD. (I should probably make that a bit more clear in on the 1st page) Just out of curiousity though, could you post your computer specs and let me know how many mods you are playing with?

EDIT - oh and are you using any purge cell buffer mods or do you currently have the 4gb LLA patch installed?

Thanks
User avatar
Haley Cooper
 
Posts: 3490
Joined: Wed Jun 14, 2006 11:30 am

Post » Sat May 28, 2011 3:31 am

I get weirdness with the LAA enabler. Weird graphical problems, stuff like that.
User avatar
Tanya
 
Posts: 3358
Joined: Fri Feb 16, 2007 6:01 am

Post » Sat May 28, 2011 12:44 am

I get weirdness with the LAA enabler. Weird graphical problems, stuff like that.


Look under the "helpful notes" at the bottom of the first page.
Read "1." and PM me if you have questions.

That is the only thing reported that causes issues.

thanks
User avatar
MR.BIGG
 
Posts: 3373
Joined: Sat Sep 08, 2007 7:51 am

Post » Fri May 27, 2011 11:44 pm

No, check that. I think it's my overclock. Because I reverted back to the default .exe and I still have the same weird graphical problems. Time to up my voltage.
User avatar
SEXY QUEEN
 
Posts: 3417
Joined: Mon Aug 13, 2007 7:54 pm

Post » Sat May 28, 2011 4:11 am

no I play with 100% vanilla game (patched to 1.2.0416) with shivering isles and KoTN.
User avatar
Emma louise Wendelk
 
Posts: 3385
Joined: Sat Dec 09, 2006 9:31 pm

Post » Sat May 28, 2011 9:22 am

no I play with 100% vanilla game (patched to 1.2.0416) with shivering isles and KoTN.


Yeah, good luck crashing that game... lol! Unless you have under 2gb of ram, you probably won't ever experience a physical memory related crash.


User avatar
kelly thomson
 
Posts: 3380
Joined: Thu Jun 22, 2006 12:18 pm

Post » Sat May 28, 2011 2:17 am

1. Do you guys/girls use the "multi-thread tweaks" for the ini file. Such as the background loading options and setting the iThread to 9. I have yet to really notice if those tweaks helped or hurt, and with the presence of such great mods as OSR and Streamline I am wondering if they are even needed? Do those tweaks benefit your game at all?


Don't bother with messing with the multi-thread stuff. The game doesn't utilize it enough to matter, and what little it does is inefficient at best.

Also, I don't remember which thread it was, but IanPatt from the OBSE team said the game doesn't even use the iThreads or OpenMP settings at all so you won't ever see any difference touching those.
User avatar
Kay O'Hara
 
Posts: 3366
Joined: Sun Jan 14, 2007 8:04 pm

Post » Fri May 27, 2011 6:55 pm

Don't bother with messing with the multi-thread stuff. The game doesn't utilize it enough to matter, and what little it does is inefficient at best.

Also, I don't remember which thread it was, but IanPatt from the OBSE team said the game doesn't even use the iThreads or OpenMP settings at all so you won't ever see any difference touching those.
The game doesn't multithread much during normal operation, but on cell transitions, saved game loads, some menus, etc, it multithreads significantly, varying amounts depending upon Oblivion.ini settings.

In the absence of OSR, it performs very badly when multithreading - badly enough that you'd be better off w/ no multithreading. With OSR, it multithreads better - much better if heap replacement is enabled in OSR (disabled by default because it causes stability issues for some people). I typically use default settings.

IanPatt is generally right, but at first glance at [OPENMP] iThreads looks like it is used.
User avatar
teeny
 
Posts: 3423
Joined: Sun Feb 25, 2007 1:51 am

Post » Sat May 28, 2011 7:34 am

Here is a PM conversation I just had which proves another very interesting fact!!!

far327
- It would be a rare case if Vanilla Oblivion contained any memory leaks. But this did make me think of something. Why did your game stop trying to go beyond 1.5gb? How long did you wait when you reached that 1.5gb limit? Oblivion will begin to purge out cached memory over and over until it has no more cached memory left to purge. At which point it really should CTD. Or at least it did that for me under Windows 7. Not sure if it is an OS feature or something within Oblivion to help prevent CTDs
xaxxiron777 - When it was at 1,5 Gb, I tried to increase it for 5-10 min. I have Windows 7 too. And I have no idea how it wouldn't go above 1,5 Gb :S
far327 - Did you see it sit at 1.5gb and then move back down to say... 1.45gb and then increase back to 1.5gb and then back down again to 1.45gb?
xaxxiron777 - Yes, actually!
far327 - That's great news man!! haha, that meas that Oblivon or Windows 7 was actually purging the memory in order to prevent a CTD!!! I personally think it is a Windows 7 feature but I will do another test with XP to verify. My guess is, if you would of waited longer, a CTD would of eventually taken place. Not sure how long it would of taken though.

This indeed means that Windows7 or Oblivion is automatically attempting to purge out cached memory to prevent crashes when pressed up against a processes memory ceiling. But what I noticed when testing the memory leak script on my system is that it can only purge out so much memory from the cache before it finally has to give in to the crash.
For example: This would help explain why we experience a CTD when we enter a highly populated NPC city (Like Anvil Docks) with UL, Better Cities and QTP3 installed. Basically it means that with out LLA enabled, with a heavlily modded game the memory is filling up so fast to it's ceiling that Windows can't keep up to purge the cached data out fast enough. With LLA enabled, this should help remedy that scenario. For XP users, I suppose they would crash faster in this scenario, although I do need to verify if XP is infact purging any cached data when Oblivion.exe reaches it's ceiling point
User avatar
Bereket Fekadu
 
Posts: 3421
Joined: Thu Jul 12, 2007 10:41 pm

Post » Sat May 28, 2011 2:12 am

Oblivion does include some code that attempts to free up some memory under some circumstances, not just PCB. But that code is I believe rather limited in scope, and OSR heap replacement may or may not clobber parts of it.
User avatar
Phoenix Draven
 
Posts: 3443
Joined: Thu Jun 29, 2006 3:50 am

Post » Fri May 27, 2011 11:09 pm

Oblivion does include some code that attempts to free up some memory under some circumstances, not just PCB. But that code is I believe rather limited in scope, and OSR heap replacement may or may not clobber parts of it.


Hmm, I suppose I should try testing with and without OSR then. But either way, it's a nice thought even if it really doesn't do much for a heavily modded game. (oblivion purging cache that is) Just glad LLA more or less allows enough head room to where it shouldn't be an issue anyway.
User avatar
NAkeshIa BENNETT
 
Posts: 3519
Joined: Fri Jun 16, 2006 12:23 pm

Post » Sat May 28, 2011 10:24 am

I have followed this thread until now. I must say it's really educative! Thank you all!
User avatar
Eliza Potter
 
Posts: 3481
Joined: Mon Mar 05, 2007 3:20 am

Post » Sat May 28, 2011 6:15 am

If you're on XP and benefiting from LAA then you probably don't REALLY want to try the game w/o OSR heap replacement. If Oblivion allocates more than 200 MB of small heap allocations on its native heap on Windows XP performance starts plummeting towards unplayably slow. And if you're not using more than 200 MB of small heap allocations, why would you need LAA? On other versions of Windows it's not a big deal though.
User avatar
Elina
 
Posts: 3411
Joined: Wed Jun 21, 2006 10:09 pm

Post » Fri May 27, 2011 10:41 pm

I have followed this thread until now. I must say it's really educative! Thank you all!
You and me both koldunic! Thanks for the positive comment!


If you're on XP and benefiting from LAA then you probably don't REALLY want to try the game w/o OSR heap replacement. If Oblivion allocates more than 200 MB of small heap allocations on its native heap on Windows XP performance starts plummeting towards unplayably slow. And if you're not using more than 200 MB of small heap allocations, why would you need LAA? On other versions of Windows it's not a big deal though.
Right, I remember you mentioned this in a past LLA thread. I'll add your comments to the 1st page of this thread. :goodjob:
User avatar
biiibi
 
Posts: 3384
Joined: Sun Apr 08, 2007 4:39 am

Post » Sat May 28, 2011 8:00 am

side note: I believe it's LAA, not LLA. "Large Address Aware".
User avatar
Loane
 
Posts: 3411
Joined: Wed Apr 04, 2007 6:35 am

Post » Fri May 27, 2011 8:12 pm

side note: I believe it's LAA, not LLA. "Large Address Aware".


:facepalm: D'OH!!
User avatar
Mariana
 
Posts: 3426
Joined: Mon Jun 12, 2006 9:39 pm

Post » Sat May 28, 2011 12:15 am

I tried again, and when I reached 1723 MB ram, the game just exited. Ill tried to make a video to upload to youtube soon!
User avatar
Facebook me
 
Posts: 3442
Joined: Wed Nov 08, 2006 8:05 am

Post » Fri May 27, 2011 10:48 pm

I tried again, and when I reached 1723 MB ram, the game just exited. Ill tried to make a video to upload to youtube soon!


Yup, that's your oblivion.exe memory ceiling point w/out the 4gb LAA patch installed. That was almost exactly the amount of memory I showed before each crash I tested under Windows 7 64bit.

Videos are always great! I started creating some, but
thought it would be more beneficial to just get the info out first and then make it pretty later on. hehe
From your specs of your rig in your sig, My guess is you will hit 3.8gb of memory usage before a CTD after adding the patch.
User avatar
Eoh
 
Posts: 3378
Joined: Sun Mar 18, 2007 6:03 pm

Post » Fri May 27, 2011 7:30 pm

I'm uploading a video to YouTube in 720p showing how it crashes a ~1,75 GB Ram, though it has been processing for quite some time. The url is http://www.youtube.com/watch?v=jq5uJQKMV6s but I would really recommend you to wait until it has processed, as it looks like **** in 360p, and is not worth watching in such low quality.

So wait for teh HD plawkz :)

EDIT:

Seems like the quality has improved slightly, as it is not as blurry anymore. Still, I'd encourage people to wait for it to be fully processed to HD 720p.
User avatar
aisha jamil
 
Posts: 3436
Joined: Sun Jul 02, 2006 11:54 am

PreviousNext

Return to IV - Oblivion