(Info: This is supposed to be a tutorial aimed at no particular audience. My plan is to just write it down in one swoop, submit it, maybe double-check it later ).

Yesterday, a request in the SKSE thread got me thinking about how to design of a fully redistributable Papyrus library (again). The purpose of that library would be to provide some generic, UI-related features, but that doesn't really matter here. What I’m going to describe are some common considerations that should apply to any kind of library or framework.

To clarify, by redistributable I mean that the library is not required to be installed and loaded as an external dependency by the user, but can instead be included directly with any mod that wants to use it.

As this post is probably going to get long, I won’t waste any time and get right to it

What is a Papyrus library? It’s one or multiple scripts that provide functions, which can be used by other mods. Example:

scriptname MyLibraryint function AddNumbers(int a, int b) global	return a + aendFunction

To use it, another mod has to have access to the MyLibrary.psc and the compiled MyLibrary.pex. Then it can call the functions it provides:

; some codeint result = MyLibrary.AddNumber(1,2)

Let’s ignore the fact, that this function is terribly useless and pretend others started using it. Because other modders don’t like having extra external dependencies, we are nice and allow them to include a copy of MyLibrary.psc/pex.

But we made a mistake… return a+a should be return a+b. Fixed:

scriptname MyLibraryint function AddNumbers(int a, int b) global	return a + bendFunction

Now there are two mods out there using our library. One uses the fixed version, the other one is no longer actively developed and still uses the old version. Unfortunately, both scripts have the same filename, one of them is going to override (if loaded from a BSA) or overwrite (if distributed as a loose file) the other. Which one of the two is going to win depends on the order of installation and not on the fact that version 2 is clearly the better one.

So that presents the first problem. If our library was an external download and nobody was allowed to include it, we could just update the dependency and instruct people to upgrade. But other mods include our files, we have to account for outdated versions of these files.

The solution: There can never be two versions of the same script with the same name. Each script has to include some kind of version identifier in its filename:

scriptname MyLibrary_1; Brokenint function AddNumbers(int a, int b) global	return a + aendFunction

scriptname MyLibrary_2; Fixedint function AddNumbers(int a, int b) global	return a + bendFunction

Any new mod using our fixed version includes and calls

MyLibrary_2.AddNumbers(...)

Unfortunately, this also means the old mods keep using the broken MyLibrary_1. Maybe we can do something about that later.

Anyway, so each script version gets its own file. What else is there to do. Our function is declared as a global function. What if we want to do something like this:

scriptname MyLibrary_1 int	currentIndex = 0int[]	listOfNumbersevent OnInit()	listOfNumbers = new int[128]endEventfunction AppendNumber(int theNumber) global	listOfNumbers[currentIndex] = theNumber	currentIndex += 1endFunction

That is not going to work (or compile). Why? Speaking in OO programming terms, a global function does not have member variables, because it does not run on any object. currentIndex and listOfNumbers would have to be declared as global variables, too, but that’s not possible.

So the things we can do with global functions are limited. We can’t access variables that persist beyond the scope of the function call. We can’t do things like RegisterForUpdate(n), because that is equivalent to self.RegisterForUpdate(n), and self is not defined for a global function (because it does not run on an object, as mentioned before). Not to forget that this is only defined for subtypes of Form anyway (just like OnInit()), and our script is not a subtype of Form.

The alternative is using a quest script:

scriptname MyLibrary_1 extends Questint	currentIndex = 0int[]	listOfNumbersevent OnInit()	listOfNumbers = new int[128]	RegisterForUpdate(10)endEventfunction AppendNumber(int theNumber)	listOfNumbers[currentIndex] = theNumber	currentIndex += 1endFunctionevent OnUpdate()	Debug.Trace(“Current number count: “ + (currentIndex +1))endEvent

Here’s the next problem: A quest script has to be associated with an actual quest. That quest is just a dummy so we can attach our script to it - it doesn’t define any objectives or the like for the player. Nevertheless, the quest form has to be defined somewhere in an actual plugin so we can attach our script to it. We could do this in a separate plugin: MyLibrary.esp. Or rather, MyLibrary_1.esp, which contains a single quest for the script. Just like for the scripts, each mod using the library also includes this .esp.
The problems with this approach:
- Each esp takes up one spot in the load order. If we release 10 versions, that would mean up to 10 slots used by our library alone.
- Steam workshop only supports one esp per mod.

Overall, it seems like a better idea to instruct modders that want to use our library to create the dummy quest themselves, in their mod, and attach the script to it.

This is where things are starting to get a little more interesting. Let’s assume there’s only a single version our library yet, which is used by N different mods. This means, we have N different instances of our script running on N different quests. If we would, for example, provide a common, external .esp for our library, we would have a single, shared state:

MyLibrary.esp
MyLibraryQuest, FormID 0x02000001

ModA.esp (depends on MyLibrary.esp)

scriptname AScript extends Quest…MyLibrary_1 property MyLibrary auto ; filled with MyLibraryQuest from MyLibrary.esp…event SomeEvent()MyLibary.AppendNumber(10)endEvent()

ModB.esp (depends on MyLibrary.esp)

scriptname BScript extends Quest…MyLibrary_1 property MyLibrary auto ; filled with MyLibraryQuest from MyLibrary.esp…event SomeOtherEvent()MyLibary.AppendNumber(20)endEvent()

Both numbers appended by A and B will be added to the same list (associated to the form 0x02000001).

But that would require a common MyLibrary.esp. As mentioned before, we can either achieve that by just letting each mod ship that extra esp. That will lead to different versions of the esp overwriting each either, so we need one per script version. That will lead to too many esps, SW incompatiblity, and furthermore we’ll still have one quest per library version instead of one shared state for all versions.

Let’s check how it looks like when both A and B contain their own quests for the library quest:
ModA.esp
MyLibraryQuest, FormID 0x03000002

scriptname AScript extends Quest…MyLibrary_1 property MyLibrary auto ; filled with MyLibraryQuest from ModA.esp…event SomeEvent()MyLibary.AppendNumber(10)endEvent()

ModB.esp
MyLibraryQuest, FormID 0x04000003

scriptname BScript extends Quest…MyLibrary_1 property MyLibrary auto ; filled with MyLibraryQuest from ModB.esp…event SomeOtherEvent()MyLibary.AppendNumber(20)endEvent()

We’ll have 2 different quests running the script now.
A.MyLibraryQuest.listOfNumbers contains 10.
B.MyLibraryQuest.listOfNumbers contains 20.

That’s no good if we want our library to act as some kind of middleware, that has a single, shared state for all its clients (i.e., a list containing both 10 and 20 in this case).

The approach we’ll follow here: Both A and B contain the full library infrastructure, so they can stand on their own without having any external dependencies. Now, if two or more mods are loaded, they should decide on a common quest to use. For example, A and B can decide that they are going to share A’s listOfNumbers. B’s list will just remain unused.

Assuming A and B have decided that A should be the leader, there are two ways that could work:
1. BScript’s MyLibrary property has to be re-filled with MyLibraryQuest(0x03000002) from ModA.
2. Any calls to B’s AppendNumber have to be redirected to A.

The latter seems more feasible, so that would look something like this:
A.SomeScript.SomeEvent() -> A.MyLibrary.AppendNumber() -> A.MyLibrary.AppendImpl()
B.SomeScript.SomeEvent() -> B.MyLibrary.AppendNumber() ->A.MyLibrary.AppendImpl()

Now, let’s see how we could elect a leader (= which quest to use) among all clients of the library.
It doesn’t matter, who is the leader, as long as all agree on it. Fortunately, with the formId, we can establish a globally consistent ordering over all all quests. Because we can assume one quest per mod, let’s just say, the quest with the highest load index is the leader. We can calculate that with

int myIndex = Math.RightShift(GetFormID(), 24) ; Note: that’s an SKSE function

To communicate these indices, we use SKSE’s SendModEvent:

scriptname MyLibrary_1 extends Questbool		foundLeader = falseint		maxIndexMyLibrary_1 	leaderint	currentListIndex = 0int[]	listOfNumbersevent OnInit()	listOfNumbers = new int[128]	int myIndex = Math.RightShift(GetFormID(), 24)	foundLeader = false	maxIndex = myIndex	leader = none	RegisterForModEvent("MYLIB_hello", "OnPeerDiscovery")	Utility.Wait(2)	SendModEvent(MYLIB_hello, “”, myIndex)	RegisterForSingleUpdate(3)endEventevent OnPeerDiscovery(string a_eventName, string a_strArg, float a_numArg, Form a_sender)	MyLibrary_1 other = a_sender as MyLibrary_1 	int otherIndex = a_numArg as int	if (otherIndex > maxIndex)		foundLeader = true		maxIndex = otherIndex		leader = other		UnregisterForUpdate()	else		RegisterForSingleUpdate(3)	endIfendEvent; Triggered 3 seconds after no other event has been received anymore; Let’s assume that’s safe event OnUpdate()	if (!foundLeader)		; Found no other with higher index, so I am the leader		foundLeader = true		leader = self	endIfendEventfunction AppendNumber(int theNumber)	if (leader)		leader.AppendNumberImpl(theNumber)	endIfendFunctionfunction AppendNumberImpl(int theNumber)	listOfNumbers[currentIndex] = theNumber	currentListIndex += 1endFunction

As we can see, in MyLibrary_1 script, there’s a pointer to another MyLibrary_1 instance (leader).
The AppendNumber call is forwarded to leader.AppendNumberImpl.
For our A and B example, B would have the highest load order index, so both
A.leader and B.leader point to B.

Obviously, there are still a lot of details and special cases to be handled, but the concept should be clear.

To be continued. Next issue: How to support the same thing for multiple library versions (this is going to get ugly).

Oh, and feel free to post comments, suggestions, corrections, etc.

(Reserved)

(Reserved #2)
(No more reservations after this one )