Re Esp Thrashing:
It looks like it opens ALL esps in alphatical order and scans them briefly (probalby just headers?). Then it reads all active esps in load order. Then closes all active (and inactive) esps. Of course, more reasonably, you would only expect it to open the active esps. I suspect that it's sharing common code with tescs.exe which does need to open/scan all esps.

Some interesting looking queries:


Cobl Main.esm and 300_White Stallion 4.esp stand out because I'm not actively using them. They are both there for convenience. Cobl because some mods I need to make patches for hav dependency on it, White Stallion because I was planning to look through it to see if I need to patch it.

During startup a whole bunch of registry keys were accessed. The only one that stood out as particulary strange was the query against Terminal Server. Why exactly does Oblivion need to care?

The specific version of DX9 it wants is interesting. It seems OB wants that specific DLL. It trekked off to Windows\System32 to find the one it wanted. It may explain some weirdness with OBGE because the OBGE dll file wants whatever is installed on the system - which in my case as of a few minutes ago means d3dx9_40.dll. Is it a bad thing for two different DX9 dll files to be involved like this?

The even more interesting part is that Oblivion did 4 scans at startup of all of the plugins in the Data folder. Once by alphabetical order for ESMs, one by alphabetical order for ESPs, one for every plugin file by file date, and one more which ended up producing the ordered by date list of active plugins. Odd indeed, but definitely means it's probably best to keep strays out of there. It made two similar checks against every ESP file looking to see if they had BSA files that mathed generally against their name.


This one in particular alerts me to a lurking potential problem, at least for myself. I have temporarily moved plugins to the folder *ABOVE* Data as a way to take them out of the way of being loaded while testing something. I never noticed if any errors resulted from this sort of thing. Why I had a copy of oblivion.esm sitting up there is beyond me though. It doesn't look as though the game chose to use it, but it's still somewhat bothersome to me that it noted it as a valid entry.

I haven't noticed any evidence of random thrashing about, but the log I generated in a few minutes time was nearly 1 million filtered entries just for all the wackiness oblivion.exe is up to while running. I have been noticing though that at least for me it's always looking into the loose files in the Data folder to find stuff before moving on to the BSA files. And of course in those cases where it first looks in the top level Oblivion folder and then in subfolders it thinks should be under that I've seen instances where it asks 4 times and only goes to the BSA after they all fail. Repeat this sort of thing enough and is it any wonder the game lags on disk I/O?

Ok I promised a more thorough writeup of my own investigation into this issue that I did a little over 2 years ago.

Note I'm rather non-technical and this was done with the sheer purpose of getting my game to start working again. But it might provide some useful insight into the issue.


Originally I started having problems where Oblivion would seem to randomly "misplace" certain textures at random. I had a savegame that was saved near Bravil, and after about 5 minutes of playing the game would CTD.

Leading up to this CTD random textures would be displayed as black and/or pink (indicating a missing texture). The thing was, it was always different what textures were "misplaced". The textures WERE there, as they would appear one moment, then on cell change (or sometimes even just going out of the player's visual range for an instant) they would suddenly be displayed as missing (pink/black).

For example, in one test I remember quite vividly I approached bravil and first noticed one of the wayshrines textures appearing all black, except for the altar. Next, the bay horses outside bravil were pink, except for their eyes. Next, once I was inside bravil I was greeted by a very frightening guard whose entire face was pink as well. Then, upon leaving bravil again, the grass was pink (it was fine earlier in the test). Then, as I turned to go back into the city the entire city wall turned black and the game CTD'd.

I struggled with this problem for well over a week. It even occurred with no mods loaded. So I then turned to my system. I tried new drivers, system restore, checked power supply and heat levels, all seemed fine. I even went so far as to copy my Oblivion install to my friend's machine to see what would happen (expecting that the issue would not be present, since I thought that by unchecking all mods it had eliminated the install as a possible cause) - this was mostly to cement the fact that something in my own machine was dying and I needed new hardware.

Much to my surprise, with no mods loaded and the only similarity being the files I had copied to his machine, my friend's machine experienced the same problems as mine, randomly missing textures and CTDs after 5-10 minutes of playing.

At some point along the line I just started deleting stuff out of my data folder. I didn't know what else to try. I used Wrye Bash to move a large number of inactive ESP files to the "hidden" directory as well.

I think I had somewhere around 450 esp files in my data folder, of which only about 150 were active. The rest were junk I had downloaded to test or inactive optional esp files for mods.

After I had trimmed this down to a more reasonable level, the problem went away and has not returned.

I posted my findings on these forums, but I think that thread has since been trimmed. I also believe I PMed Wrye about the problem, but at the time everyone passed it off as a strange, but random occurence.

On several occasions I have helped people on these forums who have been having similar issues, and on at least one occurence I can remember another user reported his problem also went away after cleaning out his data folder.


Hopefully this info helps trace the problem down.

I was using OBSE at the time, but the problem persisted even when I ran Oblivion using the default launcher, as remember, I thought as well it was mod related, but I couldn't even get the game to run properly using JUST oblivion.esm and the normal .exe w/o OBSE. My conclusion was that the large # of inactive plugins were the problem.

DarkDragon: Yep, that was definitely the first mention of esp thrashing that I heard of. (And with the reminder, I think that may be the problem that I've been having recently.) But esp thrasthing is clearly separate from directory thrashing.

It would be possible for Bash to temporarily hide inactive esps. Annoying, but possible.

This is completely anecdotal evidence, but I've bsa'ed up everything in my game and seem to have less thrashing afterwards.

I also took some rather extreme steps that I don't expect most users to do like extracting all base vanilla meshes, overwriting them with Operation Optimization and UOP meshes, PYFFI'fying everything, adding in the RAEVWD _far meshes, Roberts male and female body mods and vanilla clothes and armor mesh replaces for them, AWLS + a couple other mods I know I definitely want, then packing everything back up uncompressed. I also extracted textures, overwrote some and repacked as an uncompressed bsa.

When I add mods now I try to do it as a bsa once I decide I want to keep the mod around. If nothing else it makes the file organization easier to deal with, but I do seem to notice like thrashing/stuttering.

As for esps, I generally tend to move inactive esps to another directory outside of Bethesda Softworks/Oblivion or any of it's subdirectories. I could never prove it but I always felt like it was messing with inactive mods and causing issues (guess I was right ) so I just moved those files way out of there to be on the safe side.

I'd imagine it's much harder to move meshes and textures out though since it's pretty difficult to figure out what mod used what sometimes. This is why I prefer to bsa everything. Bethesda did it with all their DLC mods and those are easy as pie to remove.

True, but nonetheless I think the issue is related. If it's directory thrashing it's probably doing it for a similar reason to why it thrashes the .esp files.

I'll second the "less thrashing" from super-custom BSAs. I've gone to more or less the same extremes you did with packaging up all sorts of mods in with the vanilla stuff, PyFFI'd as much as possible and any _far.nifs introduced getting thoroughly cleaned. Even with all that dilligence, I still get occasional lag and have lately been unable to fast travel to Chorrol's north gate. But I rarely use fast travel so I haven't bothered to look for the cause.

In running procmon I wasn't able to find any suspicious activity other than registry calls to look up info on Terminal Server. I still can't see why the game would need that at startup.

Using custom BSAs does appear to reduce the directory thrashing, that's what I do as well. However you do have to be very careful about what you put into the BSAs .

It appears that having textures that have no mipmaps (e.g. the new imperial Guard helms that OOO introduced) in the BSA results for some reason in them not displaying the texture whereas they work fine when plced in loose directories.

Also you have to be extremely careful with mesh replacers , you should only have unique meshes in BSAs i.e. you can't have the vanilla female body meshes in Oblivion - meshes.bsa and then have same the exnem female body meshes in Oblivion - meshesixnem.bsa. . For some reason even having the BSAs have different dates still causes issues in OB opposed to a single unique mesh even though you would think it would look in the more recent BSA first , find the mesh and then stop searching

This is really concerning, and some totally silly behavior.

I've been trying to track down some crashes I've had recently, but I never expected OB to be digging around in the folders I created within /Data for anything, since it's so picky about all the stuff going into the right folders...
This is uh, some unoptimized looking stuff.

I've been storing all the downloads of my mods in the /Data/Backup folder, because I do a lot of mod-modding to make them all behave... this, and the number of inactive ESPs in my Data folder could be causing all sorts of havoc. Off I go to remove them and then test OB.

EDIT: Wow, now that I think about this, if it's digging around in all the folders for random things, are the mods that have folders for ini commands and whatnot contributors too?

EDIT EDIT: Also, Bash does not want me to move the docs directory out of /Data.

For an alternative directory, I suggest "Oblivion Mods" (at the same level as Oblivion, so for a normal install, it would be \program files\bethesda softworks\oblivion mods.) (As an option) I'll be making use of such a directory in next release of Bash (it will be possible to move the Installers directory from Bain and the Data\Bash directory to the Oblivion Mods directory.)

I would be very interested in any clues you find about the problem near Chorrol's north gate. I've had this problem myself and other players have reported the same thing. Never found the cause.


Nice! For ages now my setup has been almost just like that. I have D:\Games\Oblivion\ and mod installers/archives in D:\Games\Oblivion_mods\ (with subfolders in there to group mods by type, such as quests, textures, combat, homes, etc.). Sounds like your new structure will work pretty well for me.

My computer knowledge is poor, but regarding searching files...when CTD occurs there is a note about crash like offset 00001bcd or 0x003d22d5 or smtg similar. Are maybe this hex codes related to an item Oblivion is searching for in other folders?

I've got Oblivion installed in C:\Oblivion and now my mods are all in C:\OblivionMods.

That would be a no. Those offsets are concerning Oblivion.exe's internal structure, and what command (well... what assembly instruction, I think) made it go boom.

Is it possible to find out which command is problematic and thus solve the problem or only Bethesda can solve this?

my directory structure is similar to Iceciros F:\Oblivion (for game) then F:\OBbackup ( back up meshes textures I'm playing with) and F:\OBMods for esps\esms

So...

This could explain why my game became far less stable after I started making Item Interchange... Even though I kept a separate data folder for modding when I try to play the game it reads through all the junk that's collected in the data folder for making mods and eventually throws a fit. Item Interchange has require me to keep track of so many mods, patches and optional files the data folder is a complete mess (24 esms, 226 esps and I have no clue how many meshes and textures from how many mods that II does and doesn't support).

I'll try switching to having a duplicate installation folder instead then? Like: C:\Program Files\Games\Oblivion\ and C:\Program Files\Games\Oblivion Mods\ switching between them when I want to play and when I want to mod? Will this likely help? This is both really dissapointing that the problem exists and also very heartening that people have found it.

Thankyou very much for the heads-up: maybe this will solve (or at least reduce) my otherwise apparently unsolveable CTDs.

Vac

Just reading about several people saying that packing stuff into BSAs helps reduce the 'thrashing' got me thinking...

Would compressing the Oblivion/Data directory, using Windows' file/drive compression tool, achieve the same effect, or would it cause more problems? Looking at the effects this has on files makes me think that what Windows is actually doing is zipping the files into an archive, only transparently to the user. And a BSA is basically similar to a zip archive, isn't it?

Eloise

Packing in BSA files help, but only if they're uncompressed. As soon as they're compressed, Oblivion needs to decompress them first, which is causes a brick wall effect - the BSAs pile up, at one point your memory gets full and then Oblivion goes buggy / crashy / laggy.

Also putting stuff in virtual zips can have all kind of side effects. Not recommended imho.

about the saves folder scanning...
I tried the figure out how the game sets the save number, and I believe that when making a save the game scans all files in the saves folder to determine the number for the next save.

Note that bash/bain will look specifcally for "..\Oblivion Mods" (note, space but no underscore). Under that, the structure will be
Oblivion Mods\
--Bash Installers\ <<--If you're currently using BAIN, DO NOT DO THIS YET!
--Bash Mod Data\ << Do NOT DO THIS YET!

shadowborn: No. That would slow things down not speed them up.

Esp Thrashing: I expect that I'll add some feature to bash to auto-hide esps. But this will be a pain in the butt to code since it breaks some pretty basic assumptions. What I would have to do is to change inactive esps/esms somehow so that they are no longer is subject to esp thrashing (e.g. by changing the extension). However, the feature should otherwise be transparent -- e.g. to the rest of bash, it looks like the mod is still there. This of course would be a real pain because there are lots of things in bash that are dependent on the name of the esp itself. And naturally, it would confuse other mod utilities, which would no longer see the mod, so I probably need to be able to flag certain mods to NOT autohide even if they're inactive.

Thanks alot for this important info. That explains why as soon as I installed a bsa mod all of sudden I started getting disk thrashing with OBLIVION. The same for unactivated esps I have gotten into the habit of having lately.

Actually... *prepare to flee* ... I'd wanted a "bash file" (maybe extension .esb?) for a long time now - for those kind of stuff that really ought to be used only as imported (or maybe merged) file. Saves the trouble with people activating them .

Good idea -- I'll keep that in mind. There are some complexities associated with trying to do it though. E.g. suppose that I have xxx.esp already in the data directory, and the user installs xxx.esb (shadow(?) version). Then Bash has to figure out which of those two trumps the other. (Vs. that suppose that user already has file existing as xxx.esb, but then user installs xxx.esp.) If the installation was done purely by bain, I could sort it out, but users might install manually instead.

Anyway, I'll keep it in mind. I'm not sure how soon I'll be able to get to esp hiding. Maybe next weekend. (I'm booked with the bain directory/build handling stuff right now.)

This is probably gonna sound paranoid but would it be possible to install hidden esps that could do malicious things without the users knowledge? Or does it have to be knowingly activated before its possibe for it to become hidden?