Oblivion scripts are broken into sections; each one named Begin **** - common ones are Begin GameMode, MenuMode, OnActivate, OnDeath;
in the compiled section of the scripts I'll just search for the codes for begin **** and then insert a chunk of precompiled code giving the mod/script name (into the bashed patch so just rebuild without that selected and it is gone)
(though I did just now realize I was forgetting one small thing... I'll have to figure out how to correctly "compile" a single string during patch build for each script... If I can't get it to work correctly it is workaroundable; just would require a bit more work on the users part... hopefully won't have to go there but if so I'm sure that users will still be able to do so).
Pacific Morrowind

Oeh, oeh, like shadeMe and his upcoming CSE?

What resources (programming tools) are available to you at the time you want to create the strings (i.e. Python, OBSE, external dlls)? My thought would be to create a line that calls an OBSE extension function that is wrapped in that critical section I've been harping on. If we can link the logging function as a separate persistent thread with a static string buffer we can avoid the overhead of creating all those unique strings and bloating the oblivion memory space.

Call could look like:

SomeOBSEFunction(String modName, String scriptName, int beginBlockCount);

Then the runtime string allocation would happen without raising the memory footprint beyond the added lines in the scripts (provided OBSE's function calling method doesn't copy the strings it gets as arguments... in which case nevermind)

How about CTDHunter as the module name?

EDIT: Had another thought. Memory use could be reduced even further by not using strings at all, just pass integers (Load order instead of modName, Script number within a mod, etc). With some shifting we could probably represent each call in a 32bit integer. Then we can do all the string processing to create the human readable log in a post processor.

2 (okay make it 3) cents:
1. Why modify the scripts instead of hooking the codepath that executes them? Bear in mind that modifying a compiled script requires reallocating, modifying, and copying its data buffer. A hook would simply print the formID of each script that passes through.
2. Most scripts run every frame. As mentioned by others you are going to see massive I/O traffic.
3. The assumption that scripts cause most CTDs is dubious. In any case, CTDs that are caused by scripts are very likely to occur after the script (and many other scripts) has executed rather than at the precise moment a particular script or script command is invoked, making debug output consisting of a sequence of script executions more or less useless.

You'd probably be far more successful debugging a CTD in a real debugger, particularly if you attach to the game process using VS with OBSE's source code loaded.

Just for the sake of clarity, VS == Visual Studio? Interesting... I think I actually have a copy of it laying around somewhere.

Thank you for the insights.


From my point of reference, mostly because I wasn't sure how easy it would be to get the hook in place and PacificMorrowind thought he could do it easily through the bashed patch.


I don't have a great answer here. Although, there are ways to mitigate some of the hit, this is clearly a concern.


Now that just svcks. Is it the case that there is enough multi-threading going on, that you need to follow each one to find the problem. Can you explain the execution model used by Oblivion to run its scripts? Please.


Is the OBSE source available? And how important is VS to the mix? Would gdb or valgrind or another open source debugger be an option?

Or better yet, how would you go about identifying CTD causes? And is there a way to make it available to the general player?

Script processing isn't multi-threaded - No two scripts run concurrently.

You could use any other debugger of course, but with VS, you can make use of the code exposed by OBSE. You should find the source in the main download package.

Given these two statements I am confused. Taken together this implies, that script processing will continue after a script performs an illegal act (divide by zero, null reference, etc) or that script related bugs that cause crashes are the result of tweaking some other weakness in the implementation.

Do any of our experts have a working list of what typically causes crashes laying around somewhere? And would they be willing to share such a list?

Scruggsywuggsy, I am in the process of getting VS up and running and relearning how to use it (I've been in the open source world and mostly doing java for the last 7 or 8 years). I expect it will be another day or two before I can follow your advice to attach the game to a debugger. It the meantime, if you would indulge a question or two:

In your experience are CTDs the result of the game throwing untrapped errors?

After my first quick read through of the OBSE source, it looks like OBSE wraps itself around the Oblivion executable. Is this correct?

Thank you so much for helping to educate me on how all this works.

One of the more common crashes that occur on my rig is during the load game routine, specifically while loading extra data from the save file. The game crashes when an object's script extra data can no longer be constructed (like when you remove the mod that contains that script). More precisely, it happens (AFAICT) when it attempts to access the parent script's local variable list. When the parent script isn't loaded, the operation becomes illegal (protected memory gets accessed) and the game crashes.

I wonder if you came upon Windom Earle's Oblivion Crash Prevention System plugin ? It directly patches the executable to handle specific crashes.

Yes, I ran into OCPS pretty early on. It beeps at me with some regularity. It looks like OCPS treats symptoms and not causes. I don't think it's under active development any more either. What I'm hoping to be able to do is pinpoint why it blows up and maybe help to do something to prevent it. I realize video/sound card or >2GB images (4 with patch) are all common reasons for CTDs. It's the rest of them that are so frustrating. I have some time and some experience with software development, so I'm hoping I can make a contribution toward making the game more stable. To do that effectively, I'm hoping to be able to pick some of your minds as I haven't been neck deep in the innards of Oblivion for the last few years. I just started playing again a few weeks ago (after my first go round with the initial release). I found that with all the ULs, Better Cities, MMM, some quest mods and a number of smaller tweaks my game has become unstable. Reading the boards It seems like I'm not the only one. Maybe I'm being naive, but it seems there ought to be a way to make the whole process easier.

RB

I don't think you're being naive at all, and I relate to the frustration. I am a software developer myself and, all modesty aside, a pretty darn good one... for what I do. Unfortunately, what I do is extremely far removed from Oblivion, and truthfully gaming software in general.

I write server-side Java code, and if you need a data-driven solution to your Web application needs, I am your guy. But truthfully, when I get down into the depths of the CS, I quickly find that I am really no better suited than your average mod user for messing with this stuff. (OK, maybe not your average mod user!) A BS in Computer Science and several years experience... and suddenly I feel like an idiot. Hell, I couldn't even figure out why I had screwy helmets in the game... or even the right place to ask for help with it! (finally got that one right, but in retrospect I should have been able to figure out what was going on)

Now, backing up a little, I actually have done some reading on modding and using the CS... I am not totally incompetent there anymore. But I have also realized that if I ever want to do anything really sophisticated, I have a LOT more learning to do... and probably lots of mistakes to make. And then I think about the folks here that have been modding TES4 for years now... and still struggle sometimes to get the OB engine to bow to thier will... can I really do better? Or even as well? And how long will that take? I have a fulltime job and a family... TES5 will be out first! lol!

My approach has been to follow the guidelines of the folks that really seem to know what they're doing... for starters. That's how I got this build together (and really, it could be better). Then I tweak things the best I can to get the best performance and fewest CTDs. That has meant making some sacrifices. I, too, love Better Cities, but I simply can not run it if I want FCOM, too. I tried tweaking everything I could but in the end, my machine just can't handle it. Taking it out reduced stutter/lag related CTDs immediately. There have been other mods I have wanted to run... some I can do, but others just won't work with my current set up.

I am sorry I don't have anything more in the way of help for the proposed crash identification system. Perhaps this can go somewhere in the end... I hope so. But even if it doesn't, you may still be able to make your build more stable. If you have not seen this already, allow me to point you to a great (and growing) resource: Tomlong's http://sites.google.com/site/oblivionpoinfo/ site. It is incredibly useful, with tons of stabilization and optimization ideas and suggestions. I recommend it highly.

Good luck, man. I know this wasn't particularly helpful, but maybe something can be extracted from it. If nothing else, Tomlong's site should be helpful!
veg

Thanks for the thoughts veg. And yes I found Tomlong's very useful site. I've managed to get a debug build of OBSE using VS 2010 express, but am still struggling with getting the symbols (pdb) for the obse dll recognized by the debugger (I miss Eclipse). In other news, I finally ran across a bunch of posts about the hard limit on the total number of esp/esm/bsa files you can have in your data directory. A quick test leads me to believe that turning on auto-ghosting has dramatically reduced my own CTD issues. That said, it still seems to me, debugging a modded Oblivion build is an arcane art. One post mentioned if you are seeing a crash at 0x0003962b then you should turn on auto-ghosting. I am still convinced improvements can be made.

One thing I still am not clear about is whether or not the script thread will continue after a null reference or other illegal instruction. If it doesn't then a tracing utility would have some use.

Another thing I'm wondering about is would it be possible to wrap the main game loop in a catch statement to trap currently uncaught errors and maybe dump some in-game information before exiting. I don't have enough experience with DLL injection methods to know if this is possible.

RB

This was some good discussion - don't let this die... hopefully something comes out of it - most of us are sick of troubleshooting.

At this point in my game, I troubleshoot more and play much less, enough for my wife to quip that I like tweaking things more than actually enjoying them :swear: