SmartMerger : III - Morrowind

» Fri Oct 01, 2010 6:04 pm

After fixing the "sound_waterfall" ID clash, SmartMerger ran to completion and produced an output file. Now I just have to figure out what it is that I have. If it merged everything, does this mean that I can run just a single plug-in?

That's the goal. Although if it didn't do clash detection for some reason.

I'd like a copy of exactly what you had that was clashing so I can find and fix my own code. It SHOULD detect and fix those clashes, but it seems I'm missing something. Probably something simple.

The command I ran is:

SmartMerger --debug --load_ini "Output.esp"

Output.esp is 261 MB in size. The CS identifies Output.esp as a Master File. I selected all the Master Files that I use and Output.esp. It successfully loaded into the CS. Summarizing Warnings.txt, 567343 duplicate references removed mostly from Output.esp, trap spell 'burden of sin' is an invalid trap (it's from Tombs Expanded) and any unique place names used to filter dialogue were lost because the places couldn't be found. The implication to me is that Output.esp still needs at least some of the merged mods to be useful.

The first file that's given will be the one that determines type & name. You can override it, or if you reference data the data after the referencing is the one that determines it. Example, if I merge the three masters together, It will still look like Morrowind.esm, except be much bigger and hold all the data.

The merged files shouldn't need to rely on anything unless it was outside the scope of the input files. I had it programmed where if it required a module and that is included as part of the merging, it would remove it from the final list. Ones lacking should remain present.

» Fri Oct 01, 2010 6:44 am

Hmmm... The inventory not working sounds like something to look into. However give this a try. I've added all the missing fields for AI's. I hoped to keep these grouped as a whole. We'll take away once we determine which ones shouldn't be separate for merging.

I hope there wasn't some sort of confusion. I didn't test merging with your program, this was just a test with regular .esp files. If SmartMerger handles NPCs already, then my request is useless. I didn't see it in the readme though, so I asked. I would imagine that keeping all AI as one merge would be easier, but it would seem more beneficial to be able to merge services, fight, flee, wander, etc., separately.

» Fri Oct 01, 2010 8:20 pm

I hope there wasn't some sort of confusion. I didn't test merging with your program...

Then yes then there was confusion, I had the impression there were bugs in that part of the program.

This is why I built the program to begin with, because the CS and game only takes the last entry and not changes. I thought I covered that as part of an example, both here on the fourm and in my documentation.

I would imagine that keeping all AI as one merge would be easier, but it would seem more beneficial to be able to merge services, fight, flee, wander, etc., separately.

True. I just considered the AI as a whole and left it as that. Right now during my DEV version I'm leaving them all as separate fields. If this causes problems we will remove them or merge the fields as appropriate.

» Fri Oct 01, 2010 4:39 pm

Pathgrid ESP record:

Interesting and useful. There was a question before about moving the pathgrids with the landmass. With the information present and a quick check, it seems they are already moving along with the land (since it's just a Cell X,Y ID). Hmmm most curious.

This means soon I'll be adding a routine to merge pathgrids... If anyone else can confirm it moves with the land that would be good. (Tested with my own island mod..)

Also to note, I noticed I have a bug in 1.7 that the sm_disabled_fix entry adds a useless subrecord that is garbage (Bad length in declaration). That is fixed as of DEV 18_2.

» Fri Oct 01, 2010 11:54 am

Then yes then there was confusion, I had the impression there were bugs in that part of the program.

This is why I built the program to begin with, because the CS and game only takes the last entry and not changes. I thought I covered that as part of an example, both here on the fourm and in my documentation..

Indeed you did, I just meant that I found nothing specifically about merging NPCs. I know the behavior of the game and how it overwrites, so I was pretty sure it would do the same for NPCs, but I wanted to test to be sure. Sorry for the confusion.

Glad to hear about the pathgrids, by the way.

» Fri Oct 01, 2010 1:54 pm

Indeed you did, I just meant that I found nothing specifically about merging NPCs.

That's because it merges records and sub-records. EVERYTHING is records and sub-records

The biggest part is dividing up a sub-record into fields, and that is mostly already provided. There's only a few record specific functions, although that's probably going to grow a bit soon.

EDIT: Working on Pathgrid merging, got the function written and just working on debugging. I'll hope to upload a working beta later today for testing.

I'm also planning on changing how it saves the merged new item id's (FRMR), perhaps giving it some leeway, say 100-200k per file/plugin. The reasoning behind this is later if one of the plugins get's updated then it won't effect the id's of other plugins when it's merged.

EDIT: Well I got the pathgrid merging working, however not all the points are always connecting the way they are suppose to. The CS also complains if the two data structures are too big. Just means there's more work before the release is ready.

» Fri Oct 01, 2010 10:06 pm

That's because it merges records and sub-records. EVERYTHING is records and sub-records

The biggest part is dividing up a sub-record into fields, and that is mostly already provided. There's only a few record specific functions, although that's probably going to grow a bit soon.

Oh I see. I completed the edit to my last post just a moment after you replied. Just a heads up in case you missed it.

» Fri Oct 01, 2010 11:01 am

First, a comprehensive anolysis of how Dialogue works: (alot of this is going into my guide I started in my PMR-Sea of Destiny readme)
I hope this mostly makes sense. If anyone has contrary or supplemental information, please let me know.

Spoiler

Dialogue Structure

Dialogue is organized into "topics" and each topic has it's own unique dialogue chain of responses for that topic.
Dialogue within a given topic is linked with a Next ID and a Prev ID (these are what need changes to relink dialogue)
The first entry doesn't have a PrevID and the last entry doesn't have a Next ID (naturally).
Greetings, Persuasions, and Voices are really just fancy "topics" and work in the same way as if they were an ordinary topic (filtering etc).

Filtering

The game engines read from the top each dialogue entry in that topic until if finds one that meets the current conditions (i.e. filtered for X, possess X item, variable z=x etc)
Each dialogue entry can be filtered by many different options. This prevents the result from showing up unless certain condition are met
Improper order in filtering can "break" topics lower in the chain (e.g. dialogue filtered for everyone above all others will prevent any other dialogue entries from ever appearing)
The best filtering should be from the most restrictive to the least
- Filtered for one person with multiple conditional statements, or a very specific "unique" conditions should be near the top since they will only appear on rare occasions
- Filtered for factions, and other bigger groups should come after the more specific ones, that way they don't override a specific persions unique dialogue
- Filtered for everyone (or nearly everyone) should be at the bottom of the chain

Hyperlinking

For each NPC, any topics that are filtered for them will be highlight in blue and "hyperlinked" in any dialogue they say.
"Hyperlinking" will correctly find combinations of words. If there are two topics that overlap ( i.e. RAT and RAT BURGER), the engine will find the longer one first.
Topics won't show until the PC "knows" them, will is usually done by simply including the word in the greeting, rumors, or advice text.
While you can select to "hyperlink" dialogue in the Construction Set (CS), the game engine will do it automatically provided the dialogue is linked and filtered in a proper manner
It was once believed that if two mods alter the same (or similar topic), the second would break the first mods topic. Based on my research (multiple mods with criss crossing dialogue), this isn't true. All of the dialogue worked correctly and hyperlinked as expected. Each person had only his/her own dialogue. It is my belief that these "broken" topics are actually just a result of bad filtering or conflicting filtered dialogue entries between the two mods.

Pitfalls

To have clean dialogue (many of these won't apply to Smartmerger since you're looking to relink the entire topic)
- If using the "copy" command to make a new dialogue entry, a modder should make sure to only alter the new "copied" entry. So that the original keeps its original ID number. Prevents issues with new mods that affect the same topic.
- For the same reason, no mod should alter the first or last entry in an existing topic. Especially greeting 1 since it will break the associated quest.
- New entries should be place at a minimum second entry down (so as not to overide the first entry that has no Prev ID) and no farther than second to last entry (which has no Prev ID).
- The engine doesn't like multiple entries in a topic not having a Prev ID or Next ID and it can't properly link the dialogue chain (similar to the problems Smartmerger was having)
- a mod actually needs to have the previous (non-mod) entry and the next non-mod entry. This is to preserve the linking for that mod (otherwise you get those errors the the CS gives for Tribunal).
- all new entries should be place at minimum second entry down (so as not to overide the first entry that has no Prev ID) and no farther than second to last entry. This causes a problem when more mods edit the same topic's dialogue.
If the above conditions are met, the engine seems to be able to relink dialogue from multiple mods. As long as their are multiple missing Prev ID or Next IDs.
If the modder "preserved" the ID numbers of included original dialogue, this would reduce broken original dialogue.
Improperly filtered dialogue or dialogue placed too early in the chain could permantly override entries lower down the chain.
Same applies if a modder places its own dialogue (or moves original dialogue) too low in the chain. It may itself be overriden.
If two mods alter the same dialogue chain and filter with the same conditions (i.e. for the same NPC or faction), the newer will appear above the older and override it.

How to clean a single mod manually

Spoiler

Individual mod cleaning

A mod can be cleaned enough to minimize incapatability and mitigate some of the problems above.

When cleaning an individual mod, you can disregard all new "unique" topics that have their own dialogue train (since it can't break anything but itself ...if it's written poorly).
For each modified "original" topic:
- Check if the topic has altered the first or last entry (look for the "*" entry that indicates new, changed, or moved dialogue).
- If so, "move" the first "original" dialogue line back up to the top of the dialogue chain and/or "move" the last "original" dialogue entry back down to the bottom. To move a dialogue entry, highlight it and then press left arrow (for up) and the right arrow (for down).
- Now to get rid of the "original" dialogue that didn't get changed and fix the sequence numbers back to the original sequence. The only "original" dialogue entries that will be in the mod are any right before and right after so it'll link correctly.
- Hightlight and delete EVERY original dialogue entry (press delete). Once you have marked all original dialogue as "Deleted", save the mod.
- Click Data File, click on the mod (and set to "active") but don't load it. Click DETAILS. This will bring up a list of every record in the mod.
- Hightlight every dialogue entry marked as "D" and press DELETE. (You can Ctrl click for multiple selection). This will mark those entries as "I" for Ignore.
- Now Load the mod then immediately save it. All the dialogue is still there but now not modified and with it's original ID numbers.
Capping dialogue
- Reload the mod. CS will list a bunch of "Strings don't match" warnings because now the CS doesn't see a working dialogue chain for the new entries ( Prev ID and/or Next ID doesn't point to existing dialogue.
- To put them back in line, each new dialogue entry (or block of new dialogue entries) needs to be "capped" off with an "original" entry.
- This is done by moving the new dialogue up then right back down. Then move it down and right back up. Now the "original" dialogue entries before and after have been altered to list the new dialogue as part of the chain.
- (If it's a block of new dialogue, move the first one up and down. Then move the last one down and then back up.)
- Save...and done.
It was once believed that to clean dialogue, one must remove ALL original dialogue. But by "capping" the dialogue with original entries, the engine can intergrate the new dialogue more accurately. Some older "clean" mods stripped all original dialogue out.
- Fix by following the steps for "Capping dialogue"
Check to see that each entry is filtered properly. Most narrowly filtered entries near the top (i.e. a single person) and the most broadly filtered entries near the bottom. Move as necessary.

Much of it was gleaned from forums (thanks to Emma, Jog and others), personal experimentation, and anolysis. It isn't a cure-all. It can't tell when one mod "filters" a dialogue in a way that will override another. But I believe broken dialogue can be greatly reduce with this method. Unfortunately it's bulky and the CS is daunting enough without having to dive into "dialogue" editing. Nevermind that the average mod user probably has at least 20-30 mods and many have 100's (thanks to merging).

SUGGESTED METHODS FOR SMARTMERGER

Spoiler

Method 1: MOD by MOD
This would be an "automated" version of the manual cleaning method (which is why it's included) that Smartmerger (SM) could do on each mod before it merges them. That way there are less chance of conflict and less [-1] breaks in Smartmerger relinking.

This applies for every modified "original" topic. Basically making each mods dialogue as "safe" as possible before (or during) the merge process so that it will be less likely to break anything.
Smartmerger would have to make sure that "original" dialogue maintains it's original ID number and if a new entry "hijacked" a dialogues ID number (by using copy but then editing the original dialogue and leaving the "copy"), it would need to reassign that entry a new ID #.
If a mod, changes any "original" dialogue that is the First or Last link in the chain, SM would need to restore the orginal Prev ID or Next ID to it's original state.
Lastly, it would need to generate a new Prev ID and Next IDs for any modified dialogue.

Now I think this method would be tricky without referencing the originating ESMs (assuming those aren't being merged too). To get the proper links and ID numbers, the program would need to look back at the masters ESMs and possibly include entries from them too ( to "cap" the dialogue entries).

Method 2: WHOLE CHAIN (I believe this is the best method)
The main idea of this method would be to include the entire dialogue chain from any topic that is in the mods to be merged. This method presumes that the master files aren't being merged too. If so then, every topic would be included anyways.

This method would need to reference ESM files
If a mod has a dialogue that alters an "original" topics dialogue chain from an ESM, Smartmerger would need to "include" that entire dialogue chain in the new mod. That way it will completely override the topic.
It still would be ideal to make sure that all "original" ID numbers are retained and that the top and last link aren't changed. Just for those inevitable "do not merge" mods that will be out there.

Method 3: THE FULL MONTY
This is simply including all dialogue in the merge. All ESM dialogue and all mod dialogue whether a mod changes it or not. I don't see this as a valid method.

Optional functions (but very handy functions):

Check all IDs to make sure that none overwrite another. Although not likely due to the length of the numbers.
Check for empty dialogue entries, again not common but possible.
Check for dialogue (within the same chain) that has the same filter. The only way to "fix" this automatically would be to either "force" a filter on one or add "Rand" condition on both so that there's a percentage change it would be selected.
Enforce a priority sorting method. Automatically make narrowly filtered tops at the top and widely filtered topic near the bottom.
- Possible filter priotrity list could be:
  - a single person with additional conditions
  - a single person
  - a faction with conditions
  - a faction
  - a cell with conditions
  - a cell
  - a race
  - a race with condtions
  - a trade
  - trade with condtions
  - and so on...getting broader and broader

» Fri Oct 01, 2010 1:25 pm

First, a comprehensive anolysis of how Dialogue works: (alot of this is going into my guide I started in my PMR-Sea of Destiny readme)
I hope this mostly makes sense. If anyone has contrary or supplemental information, please let me know.

A note on the pitfalls, not in the dialog but in the MW engine, if the Records (although ID's may be correct) are not in order in the file, the engine and CS will not only complain but it may break the order and append at the end.

EDIT: Also to note that I forgot to mention. Regarding topic/hyperlinking does try to go by the largest match first, however with the current MW Engine it is heavily dependent by record order. I've sorted it alphabetically originally and certain topics don't show up since it matches a shorter name first. The CS by default sorts it to prevent this so it's not normally noticed.

SUGGESTED METHODS FOR SMARTMERGER
Method 1: MOD by MOD
Method 2: WHOLE CHAIN (I believe this is the best method)
Method 3: THE FULL MONTY

1) Good suggestions, I'll see about adding those in before attempting to merge.

2) Likely the best choice. But honestly the biggest issue I have had is still with the same problem. It's noticed in mods where for some reason the top/bottom parts of the dialog are swapped. so if the dialog is 2, and in one mod it's 1->2->3, it may be 3->2->1 in another. I got something I want to try to see if that will fix it.

3) Yeah... maybe if you combine 300 mods you won't mind a huge file with all the topics, but for smaller groups of mods or cleaning, or distributing your own this one isn't a good option.

EDIT: Well, I'm not getting anywhere with the pathgrids. There's something about the flags that is preventing the final merge from working quite right.
Anyways, try it out if you want.
http://rtcvb32.001webs.com/MW_SmartMerge_v18_4.zip

» Fri Oct 01, 2010 8:29 pm

Version: 1.8

Fixed memory management bug causing crashes.

Included is a compiled ELF GNU/Linux executable. I haven't done extensive testing with it.

At this point I am going to stop this project; at least for the C version. As I've mentioned before, I am migrating to another language. Since to keep going as it is will only make the code more and more unmanageable and harder to add features. (So is the limitations with C.)

I've left off the pathgrid merging for now, so this is essentially V18_3, to those that were following.