I've got a rather huge script that manipulates a lot of data. It's currently got probably around 300-400 arrays and variables related to different potions/ingredients and their effects, user settings related to those effects, etcetera. But this script also has a lot of functions and events that manipulate that data, and sometimes the data needs to be accessed by other scripts too.

I'm looking for a good design method to use which will make accessing all this data as quick & efficient as possible. One of the main reasons I'm asking is that I sometimes notice in my MCM menu a lag of 2-3 seconds before an option will respond to user clicks, which I think is caused because the MCM script is asking for the value of a boolean property from my main data-storage script, which is also probably already bogged down in a dozen or more other multiple competing threads relating to its own functions and events, so I think it takes awhile before the MCM script is given access to the info it is asking for. The MCM is just one example, there are other scripts that also try to access this storehouse of data that need to be able to get the data as fast as possible.

I'm wondering what a better way to manage my data would be.

One thing I've been considering: moving all my data into a script with no functions whatsoever, and just have all the other scripts grab the data from this functionless script (which will then hopefully never be busy to prevent the data being accessed). A question I have related to this option, though, is whether threads lock a whole object or just a script (i.e., would I need this data storage script to be attached to an entirely different quest to avoid being affected by threads working on other scripts on the same quest?)

The other thing I considered is, since 80% of my data is in arrays already, maybe I could just create local array variables in each script that point to the same array. And I could convert some of my other variables that aren't currently arrays into arrays (for instance, my booleans into boolean arrays with only one member - which would then allow me to access and manipulate the same data directly through different locally defined arrays in the various scripts).

Maybe there are other options I haven't thought of too, so I was hoping someone with more experience with this kind of thing could offer some suggestions. Thanks for listening!

That is some script.

I don't know the intricacies of object locking but I'll throw a few things out there. First are you able to cache results? That could cut down a lot of access time without too much mucking about. Second remember the third rule of optimization: Profile first. Really figure out where all the time is spent before mucking around too much. Finally would it be possible to make it into an skse dll mod? C would almost certainly be a lot faster than papyrus stuff, although it would get a lot more complicated.

I always cache the config values in my MCM script to avoid that lag time, then commit them back to the globals and/or scripts they actually need to be stored in when the menu is closed. This avoids the lag problem entirely at the cost of a little extra code in the MCM script.

I also use passive update checking when updating a script that already uses OnUpdate or that otherwise fires an event fairly regularly. My flying mount script, for example, updates on footsteps. That's too rapid for repeated check, but it does check a single GlobalVariable called UpdateFlightScript. When the user changes something, UpdateFlightscript gets set to 1. This tells FlightScript that it needs to pull new values from the metaquest script, then set the GV back to 0. Otherwise FlightScript assumes its own copy of the variables is good and it doesn't check anything other than the GV.

Like DienesToo, I'd say find out exactly where the time is going before you do anything else. Calling functions in remote scripts is inherently slow and should be avoided wherever possible. Writing some of the costlier functions into multiple scripts may offend your sense of elegance but it can really improve performance vs. trying to centralize everything. It's nearly always usually faster (in Papyrus) to make ten scripts do the same thing simultaneously than it is to make the same script do the same thing ten times.

Hmm, that's some helpful advice to start. I could probably do a better job caching variables as you suggest, for instance the MCM could gather the bools on load and keep track of some things locally. So I can try that. But it would still need to be able to set many of those variables on the other script(s) in order for me to update things while the user is still in the menu (necessary in order to update everything as quick as possible), so I think I'll still run into similar issues.

I also didn't do a formal profile the last time around, though I did have some very extensive trace output that I was monitoring which gave me a sense of what was going on and in what order it was happening - but I'll try profiling after I finish this new build. I'm actually rewriting the entire mod from scratch more or less to improve upon a lot of these kinds of data issues, so have some things I'll have to finish first.

The dll would probably be the best option as you suggest, but I have zero experience in C or SKSE dlls, so it would be a lot more work than the script I've already mostly finished. I'm sure I'll want to tinker around with making a dll eventually, but for now I'm going to try and do the best I can with papyrus (been fun to try and orchestrate this all efficiently anyhow )

This is actually my current plan, more or less. I have a collection of events for each potion type, which I am going to spread across 10 or 15 "satellite" scripts, and then I can call them via a SendModEvent. One of the issues I ran into though is that during certain high-volume periods of script activity (like during installation, when I scan all the potion and ingredient forms that exist) - or during game load, when I have to update everything back to the user settings (since this data isn't stored by the save), sending too many events led to papyrus throwing out stacks and things stopped working right. So some of the activities I am doing I have converted over to functions instead of events - these are housed in the main "data" script, which initializes everything during install, and updates everything onPlayerLoadGame. But this data needs to be accessible to this main script and to all the satellites, as well as the MCM.

Furthermore, the issue with MCM caching is that I can't wait until the menu closes to update things because trying to reset the magnitude and duration of hundreds of potions and ingredients takes awhile, and happens much much faster while in menu mode than if I wait until after (which was my old strategy in version 1 & 2). And while in the MCM, the satellite scripts need to be able to access all their data as it is updated by the MCM in order to accomplish this.

Edit: Okay, so by passive updating, maybe I could use globalvariables as you suggest instead of the bool and int properties I'm currently using, and then just go into "passive update" mode while the MCM menu is open, instead of calling events directly as I do now? That might work... hmm.