Haama,

The wiki description of this bug, documented as http://cs.elderscrolls.com/constwiki/index.php/Common_Bugs#Adding_Multiple_Items_with_the_Same_Script is overly vague. Can we improve this description with any more details?

Sounds to me like we're talking about the duplicate script continuing from the same point but missing any previous data? So, if you had an earlier GetContainer or similar then you'd have a great recipe for CTDs, right?

I'll try to gather up some details about the Dueling Inventory Changes theory and post here to follow up.

Ok, here's a recap of recent conversations with Reneer and Shadowborn about possible issues with what I've dubbed "Dueling Inventory Changes".

What I have come to suspect after many problem reports and much testing is that it seems very likely there is a generic problem with having too many scripts trying to alter NPC and/or player inventory at the same time. Specifically, I think there's a strong possibility that you run the risk of getting a CTD when more than one script tries to add and remove items from inventory in a single frame.

We already know this is true if you add an item and then try to remove it too quickly. Anyone doing a lot of inventory manipulation has probably encountered that problem at least a few times.

The specific cases for me involved FCOM/MMM/RenGuardOverhaul after we switched MMM to using token-based NPC scripts for wounding, fleeing, feigning, etc.

The problem is almost certainly not specific to RGO/FCOM, but it definitely happens most often when users run several different token-based mods at the same time.

In trying to help people track down where many unexplained CTDs were coming from, we repeatedly discovered that removing some or all token-based script mods stopped the CTDs every single time. This could mean disabling RGO, Deadly Reflexes 4, MMM - Extra Wounding, or even just disabling Streampurge. Any one of these things was as likely to help as the others. Often the user was able to make a clean save, re-enable the "problem" mod, and then have no problems for a long time.

Now, many players have never encountered this problem, even with FCOM, RGO, and other heavy scripting mods together. I believe many more players have never seen the problem when using MMM by itself or using OOO by itself either. But FCOM places a lot of extra load on the average computer setup and we've definitely noticed it being more susceptible to issues like this for whatever reason.

Reneer suggested that we might be able to create a test case for the problem. Something along the lines of:

1. Create a test object that is scripted with an OnAdd block that adds another instance of the item to the player's inventory.

2. Have a Pluggy script running that checks the number of test objects in the players inventory and immediately writes that number to a file (perhaps once a second or longer between writes.)

3. Multiple instances of this test could be run on several computers, and if the numbers match up, it can be assumed that the issue is multiple AddItem / RemoveItems being called in the same frame. Of course the obvious caveat to all this is that another reason for the CTD might be that the game simply runs out of memory / reference ids and chokes.

He also suggested another test could be that several ESP files are created that link to a master file. Once a variable in the master file is set to 1, each mod will add items to the players inventory. In this way we could determine roughly the number of mods adding items in the same frame that causes CTD issues.

If we can prove that this does in fact crash the game for some users, then he suggested that we devise a system that separates item addition / removal from the player and NPC inventories. Likely this would require the creation of an ESM file that each of the mods links to as a master. If an item addition or removal needs to be made, a variable in the ESM can be changed so that no other mods will try to add or remove items during that particular frame. The problem with this is that it could potentially slow down mods and would require that a large number of mods be changed to allow for the new feature, as each AddItem / RemoveItem would need to be conditionalized to check to see if the variable is set to 0 or 1.

On the other hand, both Haama and Shadowborn are skeptical that the problem is as simple as I've described. Maybe it's just a general overload issue, which would svck since there's probably no way to get around that, or maybe it's a duplicate item scripts problem where RemoveMe gets called too quickly because a duplicate script got its delay timer hosed.

Here are the related wiki entries:

http://cs.elderscrolls.com/constwiki/index.php/Crashes#Dueling_Inventory_Changes

http://cs.elderscrolls.com/constwiki/index.php/Common_Bugs#Adding_Multiple_Items_with_the_Same_Script

Shadowborn has created some test cases, but has not been able to replicate the crashes yet. However, she may not be the best test case since she's able to use RGO and FCOM with no problems. I'm going to try running the same tests and I'll report my findings here.

Here's what Shadowborn reported about her test results:



I wonder if perhaps the tests should focus more on inventory changes to nearby NPCs than to the player. Seems like we might have a better chance of replicating the problem that way, but I'm far from certain about any of this.

If anyone has thoughts, suggestions, a more detailed testing scenario, or hard data (like existing test cases!), please let us know. We'd very much like to arrive at a definitive answer if at all possible.

Thanks!

horse.GetLOS player seems to CTD the game. I just spent about two hours figuring out why my script was crashing the game and finally found that the problem was related to GetLOS running on horses. I don't know if this is something I should have already known or not, but maybe this information can save someone else some grief. I tested mostly at the Cheydinhal Stables, but crashed at the other stables I visited as well.

I was curious about this as I have done tests with GetLOS recently.
I did a few tests and didn't have CTDs, even when mounted. There must be some additional detail in your case that may be causing GetLOS to CTD. Would you give us some more details on what you are doing?

And, quite by accident, I found a most unexpected behavior from GetLOS while doing this test and I think it belongs here in the Gotchas thread:

When called on the player (Player.GetLOS xTarget) GetLOS returns different values depending on whether the player is in first or third person view.

It seems it returns true if the target is rendered in the screen, visible by the human player.

Scenario:
PC is facing north and there is an NPC behind her/him to the south. In first person view, GetLOS returns false.
Turn to third person view and start moving the camera further back.
As soon as the NPC shows up on the screen, GetLOS starts returning true.
Move the camera closer to the PC and GetLOS starts returning false just after the NPC disappears from the screen.
All of this without ever changing the PC or NOC position or heading, of course.

Oy, good find!

So this means you're a lot better off doing:

xTarget.GetLOS Player

Right? Or does that way suffer from the same problem?

You're talking about running myhorse.GetLOS player while mounted right? You're right there are some more details.

set target to Apple
set target to (GetFirstRef 36 1) ;creatures
Label 1
if target ;if target is 0 that means there are no targets left.
set type to target.IsCreature
if ( (type == 1) && (target.GetDead == 0) )
set level to target.GetLevel
set d to target.GetDistance Player
set ch to target.GetAV Health
set mh to target.GetBaseAV Health
if mh == 0
set percent to 0
else
set percent to (ch / mh)
endif
If ( (level <= capLevel) && (level > tempHighestLevel) && (d < 300) && ( percent >= .75) && (target.getLOS player) && ( target.GetCreatureType != 4 ) ) ;<------------------Crashes game when in proximity
set tempHighestLevel to level ; of horse until I took out the getLOS
set topdog to target
set numTokens to GetItemCount miAnimalCompanion
elseif ( (level >= tempHighestLevel ) && (numTokens > 0 ) )
set topdog to target
endif
endif
set target to Apple
set target to GetNextRef
Goto 1
endif

Edit: This script is attached to a spell that runs on the player every couple of seconds.

[semi-rant]
I can place the examples, that are linked in the article, up on the wiki. However, as of now, the problem is still too vague to really comment further. The end of the first paragraph (that I should make a test file for) really demonstrates the issue. Setup - I had fake "Button" items that I would use in container menus. Select them to move them to the player's inventory, an onAdd block would run the "Button" bit of code and then, after 5 frames, remove itself. I always had 2 of each button in there (to get around another oddity). Bug - If you double-click the item, so that you take two of them very quickly (in the same frame?) one of them will process and remove itself correctly. The other seemed to process, but it's variables remained at 0 so it never removed itself.

Now that I think about it, it might be another issue - part of the script was to add back the button to the container. Maybe the error came from adding the item to one container, then immediately removing it to the player?

I ran into an issue with that problem just yesterday. I was getting constant CTDs when I added an ingredient with a scripted effect (scripted effect had a script, but the ingredient itself did not) to one container and then moved it to the player in the same frame. I was able to remedy the problem by waiting 5 frames, though in one test it seemed like it only needed 1 frame (did 5 to play it safe). I'm not sure what to make of this, though, as I also got CTDs if I directly added the item to the player. I tested both the Poisoned Apple (Dark10Apple) and Rat Poison (DASheograthPoison?) and got CTDs, but I didn't when using non-script effect ingredients like Corn, Wheat Grain, just about everything I could buy from the pub owner. One other note, it wasn't just added and removed in the same frame - one script activated another, the second script added the item to the container, then ended returning to the first script, and the first script RemoveAllItems over to the player.

I played with several variations to narrow it down - Adding the item to the container alone didn't cause a crash. Adding the item and then removing it on the next line didn't cause a CTD. Adding the item and later using RemoveAllItems (but not to the player) didn't cause a CTD. Adding the '1' flag to RemoveAllItems and changing the container to be owned by the player seemed to change the crash (it took a few seconds instead of the blink it did before), but it still crashed.

I wonder if it might be oddities with the Extra Data. Maybe we can get the OBSE team to write a function to spit out Extra Data for us?
[/semi-rant]

Ignoring the above results, thinking just about dev-akm's post, and focusing as much as possible on one aspect...
Maybe the scripts themselves will shed some light - what's on these token scripts, they're handler script, and (at least for RGO), what does the pluggy print-out look like right before the crashes?

Otherwise, the number of possibilities is going to drive us insane. For instance, in Shadowborn's tests - are those 200 tokens of the same item or different items, are they scripted or not scripted, what's on the script... So let's talk about how token systems are set up.

Most token systems use an AoE spell that adds the tokens to NPCs. Is there an error when two AoE spells occur in the same frame? Is the spell working exactly as expected - only adds one token, tests for player correctly? What happens when the spell is cast at a cell border, especially if one of those cells hasn't been loaded yet?

Overload - the real question is what type of overload? Is it just adding/removing items, to one NPC or multiple NPCs?


...
I think we need some more general information about how Oblivion does what it does. From QQuix's PlaceAtMe Bloat tests and from talking with behippo, it seems that each NPC has a linked list with 3 parts - the item's base record, the count, and extra data (owner, script variables, etc.). Also, I remember hearing somewhere that each item creates a temporary reference of itself, outside of an inventory, to run its script. I think, if we start there, we can answer the question a bit better.

That's a very real possibility. We definitely know the Extra Data can cause very strange issues in inventory, especially for some item types (esp. alchemy gear). I think a function like that would be extremely useful.



Very good questions and suggestions. We need to find a way to reliably recreate the crash, but this is hard to do if none of our guesses about the cause are on target. Neither Reneer nor Shadowborn are seeing the crash happen themselves, and that makes it very hard to track down (though they've both heard plenty of reports about it).

I agree we need further anolysis of the test cases to see if we're pushing the limits in the right direction (i.e., really trying to make it crash).

Hopefully Shadowborn and Reneer can chime in with a few more details.

Thanks, man!

For testing could someone

activate the MMM increased spawn rates plugin that would increase the amount of token's that are get handed out to MMM npcs..Or use alot of mods that alot of npcs to world, aka travellers + crowded roads + tie..extra wounding token would get added to those randmon npc.




Shadowborn can answer better about the MMM token system that hand out token that handle wounding/fatique damage for the MMM side..Reneer knows best about RGO of course.

But, I will answer what I can..

200 tokens of the same item is totally possible with MMM..if you ran into nothing but certain type of npc..

But is is most likely either npc, boss, creature, token objects. Since there is different token object that are assigned to npcs based on what type of class they are in, lvl, etc in MMM.

Extrawounding is also a possiblity that it could be handing out 200 of same token object to different non mmm npcs or non mmm creatures..




Me too, I am not the scripter but I will help with what I can and answer with what I know.

Thanks, man.

Your suggestion about running the test with increased spawn rates, additional enemy variants, crowded roads, etc. is a good idea. Anything we can do to push the limits as far as possible will help.

The 200 number was from a separate test setup Shadowborn created just to see if she could force a crash with a lot of overlapping additem/removeitem activity. It's not at all representative of "normal" MMM token usage, just a simulation to push the limits until things start breaking. It didn't crash, so we need to dig further into it and make sure we're accurately simulating a realistic test, or we need to try it on a weaker system than she has to see if that makes a difference, etc.

This is my understanding of the token/CTD problem:

When a stack of items is involved, calling removeme in a script on one of those items simply removes the top stacked item. RemoveItem will remove the top X items of the type passed to the command. The only times when I have run into CTDs with these two functions are under the following circumstances:

1) have 1 container, and one token base object with a script attached to it, and in which the onadd block calls removeme at the end.

2) set up two scripts to add the same token to the 1 container in the same frame. to be clear, i mean script A does container.additem scripteditem 1, and Script B does the exact same thing.

3) observe CTD

Taking note of how exactly removeme works, we can see exactly what is going on:

1) two tokens are added to the container in the same frame with on add blocks. They go on the item stack as A-B

2) oblivion schedules both of those snippets of code to run (the same snippet, it just runs twice.

3) scripts run asynchronously, so if B is on the bottom, and B's code executes first, the when RemoveMe is called, it removes the top item in the stack and here's the problem: Script B is removing the item attached to script A, while Script A still has a queued OnAdd block to run.

I don't know if the CTD occurs then, or if it occurs when script A tries to run on a nonexistent item.

Now I know object stacks don't quite work like how I've described -- there's one item and the extra data is attached -- except the extra data is 'attached' to individual virtual items. In the above example, the same script is being run twice by the two tokens (which are on real item/two virtual items), but each script will keep track of it's own variables. So Script B removes the extra data for Virtual Item A which has the queued script, etc.


I have to say again, I've never run into removeme/removeitem CTDs under any other circumstances. The code for my Q-Core mod (where I encountered this problem) tosses tokens back and forth over like 40 containers, and I never ran into CTDs except in the exact above situation.

xTarget.GetLOS Player does not have this problem. It returns the same result in both first and third person mode.

But notice that "xTarget.GetLOS Player" is not the same "Player.GetLOS xTarget" as the result depends on which way they are facing.

I came across a new problem: under some conditions (*) GetLOS randomly returns 0 for 1-4 frames, even when the player is face to face with the target. It is as if the target was blinking (now you see me, now you don't). When it should return 1s all the time, it returns something like this: 1110011111011111000011101000111111111011100111 . . .

The test site has a horse and an NPC facing each other, 500 units apart.
The test script has four GetLOSs and prints the results every frame:
Player.GetLOS Horse - - Horse.GetLOS Player - - NPC.GetLOS Horse - - NPC.GetLOS Player
In a configuration where all four should return 1, the Player.GetLOS Horse returns 1 as expected, but the other three follow the pattern mentioned above. And ALL three fail at the same time. It is as if the engine got hiccups and decides to calculate LOS only for the Player during that frame.

(*) It seems the hiccups are related to the number of actors in the cell. All worked fine up to 8-9 actors in the cell. Above that the more actors in the cell the more frames GetLOS will fail.

After all, this is not that bad, as I don't think there are many areas in the game (if any) with so many actors.


Anthrop, I still did not look at your script. I will do that tomorrow, as it is getting late around here (2AM)

I tried to reproduce this but was unable to. http://www.tesnexus.com/downloads/file.php?id=18334 so you can see what might be different. Setup - a Start Game Enabled quest moves the container to the player, sets 2 quests' fQuestDelayTime to .01, starts them, and prepares 2 pluggy strings.

scn TestRemMeCTDAInitScriptlong  numTokenslong  spTxtFilelong spTempbegin GameMode	TestRemMeCTDACont.MoveTo player	set TestRemMeCTDAAdder01.fQuestDelayTime to .01	StartQuest TestRemMeCTDAAdder01	set TestRemMeCTDAAdder02.fQuestDelayTime to .01	StartQuest TestRemMeCTDAAdder02	set spTxtFile to (CreateString -1 "\Tests\RemMeCTDA\Test.txt")	DelFile spTxtFile	set spTemp to CreateString	StopQuest TestRemMeCTDAInitend

The quests both have the same script which add the tokens to the container. They also print out the GameHour - if it's the same GameHour then it's the same frame. (note we can also use OBSE v16's GetCurrentFrameID, but that would require v16 and I didn't want to dig through my PMs for the exact name).

scn TestRemMeCTDAAdderScriptlong  Addedfloat fQuestDelayTimebegin GameMode	if (Added == 0)		set Added to 1		FmtString TestRemMeCTDAInit.spTemp 0 "Token added, Gamehour %.5f" GameHour		StringToTxtFile TestRemMeCTDAInit.spTxtFile TestRemMeCTDAInit.spTemp		PrintC "Token added, Gamehour %.5f" GameHour		TestRemMeCTDACont.AddItem TestRemMeCTDAToken 1	endifend

Finally, the token prints out the GameHour from an onAdd block and removes itself

scn TestRemMeCTDATokenScriptbegin onAdd	set TestRemMeCTDAInit.numTokens to (TestRemMeCTDAInit.numTokens + 1)	FmtString TestRemMeCTDAInit.spTemp 0 "Token script: Token %g, Gamehour %.5f" TestRemMeCTDAInit.numTokens GameHour	StringToTxtFile TestRemMeCTDAInit.spTxtFile TestRemMeCTDAInit.spTemp	PrintC "Token script: Token %g, Gamehour %.5f" TestRemMeCTDAInit.numTokens GameHour	RemoveMeend

Interestingly, I don't think I could repeat the crash because the tokens aren't running in the same frame! The two quests run in the same frame, and the first token runs in the same frame as the quests, but the second one runs in a subsequent frame.


Will play around with RemoveMe and see if I can get them to run in the same frame.

That's fine. I'm impressed with the checking you've already done. So it appears that GetLOS doesn't crash the game from the horse in every situation. My script used GetLOS player on every creature in the area, at nearly the same time so maybe the problem has to do with running it too frequently. On the other hand, it really does seem to have to do with horses as I wasn't crashing in areas with zillions of imps but with a few horses around my game would crash every time.

Hm, i don't remember exactly how i got things to run in the same frame but I guess the two quests thing doesn't work. It's been a while since I worked on that particular issue.

Found it.
Problem is that stables have horses for sale initially disabled, and they are returned by GetFirst/NextRef.

Calling GetLOS on a disabled actor crashes the game most of the time (for some mysterious reason, I don't get CTDs at the Cheydinhal Stables). "Player.getLOS xDisabledActor" did not cause CTDs.

You will be fine if you change your second if to: if type == 1 && target.GetDead == 0 && target.GetDisabled == 0

I also noticed a "set NumTokens to GetItemCount xxx". I know nothing about spells, but shouldn't it be Player.GetItemCount? Or do spells assume its target as the reference?

Great work! So I can use GetLOS, I just need to use it after a GetDisabled to make sure I'm not using it on a disabled reference.

Spells do assume their target as the reference.

Good stuff, man. Very curious to hear what you discovered, if anything.

I did some testing with Shadowborn's test case (running multiple copies of a plugin that constantly adds/removes tokens from the player) and I couldn't replicate the crash that way. It's not going to be easy to track this down.

I'll see if I can devise a more complete test using NPCs.

Thanks, man.

Dev pointed me in the direction of this discussion. Please excuse me while I catch up...

This may not be related to the issue under discussion, but one thing I noticed when modifying various scripts in MMM is that, in the scripts that are attached directly to NPCs, if you run code in the 'OnLoad' section, for the first frame only, many variables, settings, and even the attributes of the NPC in question, are returned as zero. Most notable in my testing was that GetRandomPercent often returned zero in the first frame. Adding a one frame delay (or more correctly, a one script cycle delay) solved the problem. I wonder if this is at play in this situation?


Again, this might not be related but mentioning the Extra Data reminds me of another oddity. In the Hunting & Crafting plugin for MMM, you can manufacture a number of things, including making arrow flights from feathers (quills). You do this by clicking on the feather, which gives you a 'Do you want to make arrow flights?' confirmation box. Clicking 'Yes' removes one feather and adds five arrow flights. If you have both stolen and non-stolen feathers in your inventory, it will often remove more than one feather, and sometimes it will remove multiple feathers even if you just have a single stack. Now I don't know an awful lot about the way Oblivion handles stacks of items, but I'm wondering if this is indicative of it getting muddled with the Extra Data and the number of items in the stack?


In that particular test, there were initially two tokens, each with their own script. Each one had an 'OnAdd' block that added the other token, then removed itself (so token 1 added token 2, then removed token 1; and token 2 added token 1, and removed token 2). If I did the RemoveMe immediately after the AddItem, I got a CTD 100% of the time. After adding a delay of 10, and running the RemoveMe in the GameMode block, I couldn't get a CTD. I put a limit of 100 on the number of tokens added (so you got 50 of each being added), and set up a quest script that would add one copy of token 1 every time it ran (on the default delay). I then duplicated this plugin, renamed everything, and ran both at the same time. So plugin 1 was adding a total of 50 copies of token 1 and 50 copies of token 2, while plugin 2 was adding 50 copies of token 3 and 50 copies of token 4.

If you want to take a look, you can download the pair of plugins http://www.shadowborn-manor.org.uk/Files/CrashTestDummies.zip.


Good question. I can't speak for RGO or any other plugin, but MMM (mainly the Extra Wounding plugin) runs one AoE spell with a radius of 4000 (if I remember correctly), which is cast by an activator on the player. The spell is 'on touch', rather than 'on target', to avoid having to mess with the script effect cast sound. It checks all NPCs and creatures it hits, adds one or more tokens depending on the kind of wounding process that actor should have, then adds a 'process' token that acts to indicate that that actor has been processed - the first thing the script effect does is check if the 'process' token is present, and terminates if it is.

In another test, I copied the Extra Wounding plugin and had both copies running at the same time, so you had two AoE spells being cast, but only one set of tokens. I went to a number of heavily-populated areas - both towns and lairs - and couldn't get it to CTD.


I think I'm fortunate in that I've got a fairly stable system. I'm even running RAEVWD and all of the UL mods, in addition to a full FCOM install, and while I do get a certain amount of slowdown outside, I don't tend to get more than the occasional random CTD - although it does often CTD on exit.


Now that's a good thought. But if we're thinking that the problem lies with multiple copies of the same token being added at the same time, I can't see how that can occur with MMM, and if it was something that happens in any other mod, I'd expect people to be reporting issues when running that mod on its own. But the problem only seems to occur when running multiple different mods, each of which use their own tokens. So rather than it being a case of two items running the same script and removing themselves at the same time, I think it's more likely to be when two items (from different mods) each run their own scripts and remove themselves at the same time.

I don't think we can narrow down the circumstances in which the CTD occurs any more at the moment; it's not like it mostly happens in particular situations, for example when the player kills an NPC (which could point to MMM), or when a guard decides to do something (which could point to RGO).

Completely left-field thought: Does anybody know if the CTDs tend to occur at any particular in-game time? If it turns out that the CTDs tend to occur at or around the hour mark, it may point to an issue with tokens affecting NPC decisions when they do their regular package evaluation.

Eloise

Some very good thoughts there, Shadowborn. I don't have any answers for you, but all the suggestions so far have given me a few ideas. I'm digging through Extra Wounding and RGO for hints about how to devise another test. I'll post details if I get anywhere with it.

As requested by dev_akm I post here a possible solution for the "enemy-NPC-freezes" in the "Dwemer Skyship" plugin:

Awesome! Now we just need to figure out what that means exactly so we can warn people to avoid it in the future.

Thanks, man!

You should add this little problem to the troubleshooting section. The problem I'm talking about starts at the second post.

http://www.gamesas.com/bgsforums/index.php?showtopic=911650&st=0&gopid=13245911&#entry13245911


Pretty common sense I guess, but just in case someone else has the issue.

dev_akm: The troubleshooting section is a GREAT idea. If every problem that was solved could be posted in that section and well ordered that would be amazing, and you would be my hero a thousand times over. The trick is finding all the solutions. The easiest way would be for people to post links to their discussion threads in this thread and provide a quick summary of the problem and solution.

the search function on these forums is pretty useless. I have more luck searching on google. That's another reason why this troubleshooting section that you're making is so valuable. If only we could get everyone involved, and so it doesn't come down to just you searching through the forums to find stuff and people randomly posting their help questions everywhere, so that the same questions get asked a dozen times and people waste their time trying to answer them.

that's what the wiki is for. but it could be better.

One suggestion: An alternative way to order the solutions would be to have on the first page, the links: Building and Editing, Textures, Models, Quests, Scripts like on the main page of the wiki, and basically match the paths.

for example, if I'm having an issue with LOD not showing up, on the main page of the Troubleshooting section I would click building and editing, then on the landscape LOD generation and then the read the section about moving the worldspace index or whatever using TES4Gecko, along with all the other problems that have to do with LOD. These solutions would also appear as a link on any tutorials or topics dealing with LOD or new worldspace building/landscape editing. some of the pages have this information, but some do not, so they should have a link to the troubleshooting section that contains the solutions associated with that particular topic. This would make it a lot easier to find these solutions, which you put the work into and are quite helpful if you can find them!

just my thoughts.

Rowan, that's a good suggestion. I will give it some thought. Thanks!

I've added a new blurb on http://cs.elderscrolls.com/constwiki/index.php/Common_Bugs#Touchy_Cells_in_Tamriel.