A recent PM exchange with bg2408 pointed out some problems with Oblivion.exe directory thrashing, too which I can add a memory of an old post plus some anecdotal recent evidence.

Inactive ESP/ESM Reading?
First it was observed by DarkOne(?) a couple of years ago that Oblivion seems to be doing something with esps and esms that are in the data directory, but inactive. His experience was that simply having too many esps/esms in Data caused some weird performance/FPS glitches, and that moving these mods to a subdirectory of Data resolved the problem. (He used Bash's File: Hid feature, which moves files to Data\Bash\Hidden.) I think that I've seen some other mentions of this problem.

Related to that I recently had a problem where Oblvion crashed apparently due to a large semi-clone of Oblivion.esm that I had made while testing some bash features. The clone was not active, and so should not have been read, but it's mere presence in the directory caused CTDs. (The clone was also likely not perfect, so perhaps that played a part.) It may have been named "Oblivion something or other.esm" and perhaps Oblivion has special handling for files that start with "Oblivion". (BSA's a definitely treated specially.)

So, if this is true, it's annoying because it means that you simply can't have too many mods in you data directory -- even if they're inactive.

Directory Thrashing
But worse than that is what bg2408 mentioned in his PM...

Some time ago I had problems with random crashes. I've investigated them and found that Oblivion sometimes starts to randomly searches all it's directories, and if this takes too long, it goes haywire. Ever seen the recommendation to not have several hundreds unused .esp files in your data folder? Likely a naive observation based on this behaviour.Now, as seen with procmon, Oblivion does not only search \Data\, but ALL folders in it's install & mygames-directory. After I've seen this, and removed all junk (which were several tenthousand files...) the crashes were gone. Now OBMM stores omods in a subfolder of Oblivion's install folder, therefore omods are affected by this search, too.

I haven't had a chance to investigate this myself, but it sounds even crazier than scanning inactive esps. Why the heck would you scan randome chunks of the install directory AND the savegames directory?? But I wouldn't be totally surprised either -- we've seen the engine do some pretty strange stuff.

So, it would be useful if this were investigated a bit more. Main questions are: 1) How often does it happen? 2) What triggers it? (searching for missing files?), 3) When does it become a problem?

Then there's the question of what to do about it. Possibilities:
1) Add something to OBSE to stop this behavior. This would definitely be good, but the OBSE team have not really addressed these types of problems. (And I've asked about this one and gotten a "Maybe later.") Also, of course, OBSE fix would only help OBSE users.
2) Not put files a lot of extra files in those directories. E.g. I personally have been storing all of my downloads under Oblivion\Downloads. Maybe the advice for folks should be "Don't do that -- put them in the directory above Oblivion."
3) Modify tools to not store files in these directories. As bg2408 points out, OBMM stores files under Oblvion directory. Similarly Bash stores backup and hidden mods files under Oblivion\Data\Bash. And the new Bain feature stores Installers under Oblivion\Installers. Those tools might be modified to use different directories.

For Bash, I could move establish a new directory at the same level as Oblivion directory, and then move installers and mod backups to that directory. But it will be a mild hassle for me to program and for users to use -- so I'd rather not do it unless necessary.

Next Steps:
At this point, the main thing that I'd like to see is a bit more understanding/verification. If it's confirmed, and there's no easy way around it, then I'll go ahead and rig bash to use an alternate directory. But I'd prefer not to do that unless necessary.

OBMM can be configured to move the OMOD storage directory if desired, so that one, at least, can be handled relatively easily if this is all true.

But I feel like a lot of investigation is in order. I mean... I know Oblivion's screwed up, but this is absolutely ridiculous. I'd want to be really certain about the nature and causes of this bug(s) before doing anything drastic...

Not my field of expertise, but http://technet.microsoft.com/en-us/sysinternals/bb896645.aspx. From quick look, at top right of toolbar, deactivate all functions by "Show File System Activity". Then go to filter menu and Filter by process name is oblivion.exe. (I think that will show just file system usage by Oblivion.) Should be able to do filter by Path is xxx and path is not yyy too.

Process Name -> is not -> Oblivion.exe -> exclude
Then it does only log what Oblivion does, which is some pretty weird stuff (not only the searching thing, but this is in my experience the most problematic one).

Interesting. The only related issue I'm aware of is the game refusing to load if you have more than 400 or so files in the Data folder. I've run into this myself once and had to move some stuff out, and I know Vorians encountered this once upon a time as well. There's always been suspicion about leaving inactive stuff lying about but I never gave it much thought beyond exceeding the count. I have quite a bit of junk floating around in my Data folder right now. This makes me wonder if leaving a bunch of junk like screenshots, extra plugins, omod stuff and everything else that's built up isn't why I've started seeing glitchy behavior again recently.

We have all suspected how poorly oblivion was coded, but this is real proof
well done

I assume you don't mean in Data\ and subfolders, but in the folder itself. I have over 32k files in Data\ and subfolders, without any noticeable issues (well, I don't really play enough to know, I guess, though). Currently 300 in Data itself (exactly, oddly enough).

And I had no idea there was a utility like procmon. That's very cool. And it does seem to indicate that this is true... ouch.

It's definitely true. I've repeatedly run into this problem myself and in reports from others.

For me the CTDs start somewhere between 325-375 files in the Data folder IIRC. Might not happen until closer to 400 as others have mentioned. In any case, the CTD usually hits within a few minutes of loading a character and happens even if I'm just standing around in a safe interior cell.

If I hide a bunch of unused plugins to get the count down around 325 (i.e., using Wrye Bash to move unused files into the "bash\hidden" subfolder), the problem goes away completely until the next time I forget and run the game with too many files hanging around (happens fairly often because I use a ton of small temp files during development).

I have OBMM configured to keep all my OMODs on a separate drive partition (partly for this reason and partly to help reduce defrag times on my main games partition).

So here's another thing I just thought of. Does this file amount limit only apply to actual ESM/ESP files or would having BSA files also contribute to it?

Looks like someone will have to make an 'Oblivion Code Patch' to patch directly into the executable as the one recently made for Morrowind.

I fought this problem was known by everyone but it seems not the case!

Every ESM ore ESP in the data folder, active or not, is compiled to run the game when starting the game or the Con set.

The things I noticed are that:

There is a limit of the total amount of active and not active mods in the data folder. I can't remember the number of mods but it was something between 250 and 500. I believe 250 was maximum amount for active mods but when you have even more not active mods the game can or will not start. Game crashes and CTD are unavoidable.

Solution:
Only keep the active ESM and ESP files in the data folder and remove the rest of the files.

Speaking of random thoughts... I run close to 20 different savegames, several of which have reached the 300-500 hour mark ... and although I carry 390 plugins in the data directory, the game is very stable. However, from time to time I have noticed an unusual amount of hard disc activity, enough to hamper the performance of the game (I wasn't using procmon, but I've checked repeatedly and found nothing at all running in the background)--and it is always in the higher-level savegames, games in which any number of mods have come and gone.

I know for sure that saves sometimes hold on to certain records for no apparent reason, resulting in a mod that cannot be removed--but most of the time this is not the case. But it occurs to me that Oblivion may actually be searching for a reference contained in the save ... which is no longer there because its mod has been deleted. In other words, is it the game engine being flaky--or is it just doing what it is supposed to (garbage collecting), and the save structure is at fault?

Interesting. I made my observation (the crash searching thingy) with a ~200 hours character, which also had removed a couple of mods. And hard disc activity is caused by this frantic search I've seen in procmon...

One thing that's unclear so far - does Oblivion search every folder in \Data\, including ones added by the user (i.e., \Data\My Mods) or just the ones relevant to Oblivion (i.e., \Data\meshes)?

No. ALL folders in it's installation and mygames directory. Not only Data, but of course all subfolders of Data.

When Oblivion starts it's searching trip, it goes through all folders associated with Oblivion. Calling this totally freaky is an understatement. As I've seen this behavior the first time in procmon I was like "WTHOMGBBQ?!"

Main question for me is what can trigger this behaviour .



Although the inactive .esp overflow may even be another problem. Sigh...

Welp, I have some files to move
Lost files/references seems to be as good a guess as any.

What makes me really wonder is that it also checks the save game folder(s). Maybe it has something to do with saving - happens whenever the game AutoSaves (or would have AutoSaved if you had them enabled), or, if you're using Streamline, whenever Streamline makes a save.

Maybe related - A couple of times, I've managed to fill up my hard drive and leave only 5 gigs left. Whenever I did this, Oblivion would start crashing at transitions, to the point where it would crash at every transition and eventually wouldn't even let me start up the game. I tried to fix this by freeing up some 50 gigs of space and defragging - didn't work. Uninstalled and reinstalled Oblivion - didn't work. Uninstalled, defragged and reinstalled Oblivion - worked. (Another random note - my save files are on a separate drive from my Oblivion install.)

Thanks for the info.

I have moved the Omod-dir to another hard drive, and additionally moved all unused esp`s and BSA`s to an backup dir.
Not that i had that many ctd`s, but i must have been close to the limit.

It would still help to have someone other the bg2408 confirm the thrashing of all directories. (Just to rule possibility of something weird with his system.)

Re removing esps from a savegame. The only significant thing left would be created items. E.g. suppose that a mod gave you "cool iron sword" and you then enchanted it. In that case, you would have a created record in your save game with same nift and icon refs as the original "cool iron sword" record (plus the enchantment reference). This can also happen when characters get fully cloned (along with their inventory). These records will continue to exist even if you remove the original source mod for the weapon.

Aside from that, removal of a master from a savegame is pretty much complete.

Just tried it and I'm getting 1000s of lines every second from lsass.exe, svchost.exe and rtvscan.exe... time to google...

bg2048 already gave instructions for fixing that - see above.

Thanks, I see it now.

Ran it through - tried to play in windowed mode watching ProcMon. Slowed down Oblivion immensely so I guess I'll just need to wait for my new computer before trying that again Otherwise, we're still talking about 100s of thousands of lines in what was maybe 1-2 minutes of play with minimal interaction (and clearing out the other 100s of thousands of lines before made from the Main Menu and loading up the save game).

So far, the interesting thing I noticed - while in game play it had a bunch of "Close File" lines for each mod, even the ones that weren't loaded. I think this happened at a transition (had a mod which automatically moved me to another room).

The rest of the lines were .ini changes - both Oblivionl.ini and Blend Settings.ini.

On my testing I concur Ob does appear to thrash both the My Games and the Oblivion. I have my Oblivion install on a different HD to my MY games Directory and both go crazy periodically when playing OB

As we all suspected as well there's some very serious data handling and memory handling issues as well. If you start putting mods in with BSAs be extremely careful what is in those BSAs as if you have duplicate meshes/textures in 2 loading BSAs (i.e. in A.bsa you have sword in Meshes\weapons\sword and in B.bsa you also have a sword in Meshes\weapons\sword) then as soon as Ob needs to display that mesh Ob appears to have a fit. If your lucky it will not display the mesh or have the mesh with a missing texture, if you not it will CTD (this seems partly related to memory usage OB is already using) and this is just one of the issues with BSAs (textures with no mipmaps) are definitely a no-no in BSAs even though they display fine when installed in loose directories

I don't have the time to read and post my findings in this thread, but I was the original one who noticed the original issue back about 2 years ago. I will post more when I have time.

I'm poking around a little bit. In my quick test, I did not see save game directory thrashing, but did see it in the install directory. It didn't happen immediately but started later. It looks like it begins a scan of the install directory, scanning through all folders looking for exes and dlls. When it finds a exe or dll, it then looks through it.

I'm wondering if this is related to OBSE? (Since OBSE is much like a debugger, is there some part of it which looks for all exes and dlls in a given directory?)

Running obl in vista64 I found no searching of extra directories within "data" with the singular exception of a very brief excursion into the obmm folder, presumably at the initial game loading screen (so early in the log).

Some new problems (maybe old, I don't know, possibly someone knows about it already):

It looks like "NAME NOT FOUND" errors have to do with just plain poor programming on bethesda or microsoft's part, as lots of programs complain of that. However, oblivion also has loads of "PATH NOT FOUND" errors, and no wonder! Maybe a quarter of the time, oblivion is looking for things that could not *possibly* exist where it is looking, examples include;

textures and meshes in the base oblivion folder
textures and meshes in the data folder(not a subfolder)
ANY sub-data folder in the base folder like "Oblivion\Meshes" or "Oblivion\Textures"

In addition I noticed that it would access music it was not playing several times a second (might be related to my personal hardware or enhanced music control? with music on I get lots of stutter, so I generally keep it off).

I'm wanting to try to figure out a way that I can line up an excel graph of the frequency of these disk writes with a graph from FRAPS of my fps, to see if these episodes of disk thrashing coincide exactly with the dips in FPS that signal visual stutter.

Also, would it be useful for those who know more if I used my second monitor to display the process monitor while running oblivion? (will it work with oblivion in fullscreen or would I have to window it? I think my comp is powerful enough to run ok windowed).

This was a bit stream-of-consciousness and I apologize. I'm quite willing to help test to figure this thing out as needed.