Hey all,

There is a new mod merging tool on the scene. An Era_Scarecrow on Planet Elderscrolls has been making a new tool called Smartmerger. It seems that he may tackle "cleaning" mods too (and if we're really lucky "moving" landmasses like Tesfaith). It is progressing very very rapidly. I just posting this to raise awareness of his efforts. Seems very few people have comment on that forum thread.

Check it out.
http://forumplanet.gamespy.com/morrowind_technical_problems/b49680/20464481/r20487638/

Definitely interesting work.

So far I've tested/used http://planetelderscrolls.gamespy.com/View.php?view=Utilities.Detail&id=440748&id=73 till version 1.2, and I can say that small utility is useful so I have great hope for http://planetelderscrolls.gamespy.com/View.php?view=Utilities.Detail&id=440748&id=78, I will give SmartMerger a try ASAP (by the way, if you are reading Era, please reupload the 1.3 version of Nif Texture Stripper at PES, there must have been some upload problem).

Sounds almost too good to be true, but I will definitely be giving this thing a shot!

Cool. More power to the modders!!

It would be nice. I have long hoped to see an integrated utility with the merge function.

I downloaded the SmartMerger but I haven't used it yet. I also wanted to download the new version of Nif Texture Stripper, but the download doesn't seem to work from PES where abot's link goes. Does anyone know anyplace else where it can be downloaded?

There's a link to 1.2 http://www.gamesas.com/index.php?/topic/1109144-mod-detectives-xx/page__view__findpost__p__16480874. The 1.3 PES link seems to be busted, it's been reported in the download comments and when I posted in his SmartMerger thread I mentioned it.

Thanks for the link... I have versions 1 and 1.1. I may download w.2 but I think I'll try to hold out for 1. 3 for a bit. I'm pretty certain that there is at least one or two people who do admin at PES who hang around here occasionally, maybe they'll see this thread, eh?

Thank you for informing me on the other fourm, i have that fixed now. (or at least another link, not sure what the problem is...)
Here's the alternate link. http://planetelderscrolls.gamespy.com/fms/Download.php?id=99546

I'm also glad someone came and posted here, and they posted at my fourm. It hadn't occurred to me to post here. I started this project when i was fed up trying to find a good merger, not finding one, and was trying to merge 100+ plugins together to make a great enhanced version of MW, which ended up taking hours and hours of manual updating from plugins, and never did get done. Obviously i got fed up, and spent about 2-3 weeks working on this to the state it is in now.

I'll be posting updates and try to answer questions on the other thread, so stop by if you have anything to comment. I'll check this fourm every so often too.

Thanks for providing the alternate link (not to mention posting it in separate places)! I have it downloaded now, but I rally hate having to use Internet Exploder to get a download...

Ditto. I've just accepted that to get most of the stuff from elder-planet i have to use IE. But i don't use it anywhere else. Oh well. BTW, as long as we're on this topic and it seems to get more hits. What functions/features do people want to see from both tools? I eventually do run out of ideas of what to add to make it better, or fix issues that are present in it.

As far as Nif Texture Stripper is concerned, I wanted to get rid of absolute paths for textures, and I wanted the recursive directory stepping. Getting away from the command line is good too... I can use command line, but it's clunky for me.

You might remember a few months ago I had asked for instruction for the 1.1 version, and you were kind enough to email me with help. I never did really get it working, but that was more because I lost my internet and my basic interest for a while. I'll be trying the new version over the next few days.

I haven't yet used SmartMerger, so I really don't know what it has, let alone what I want it to have. I'll thank you in advance though for all the hard work I'm sure went into these.

Regarding Nif Texture Stripper, I'd like an option to automatically strip the Textures\ prefix from path containing only the texture filename and no other subdirs, for instance
Textures\tx_name.dds could be simply tx_name.dds, as the Textures folder is the default one.
This way there would not be unneeded changes to nifs referencing textures in the default directory, and some bytes spared. By the way, thanks for the added option to keep original file dates

Regarding SmartMerger, as a player continuously modding/changing my playing list , I am mainly interested about merging a few mods at a time than making a handy for playing but hard to re-mod unique 1GByte mega-mod, so I'd like a option to generate only merged objects similar to TESTools, and /or eventually merged leveled lists similar to Mash /tes3cmd.exe multipatch.

So far I have mainly tested how to interface it with Mash, to be able to select multiple files in Mash and feed them to Data Files\SmartMerger.exe, it makes things easier.
As I'm not Python savvy, I'm mainly copying and pasting, here are changes to masher.py, Python experts feel free to fix!
Code tag sorry, spoiler tag spoils indentation

#-- ... ( other Mash code here )class File_Sort(Link):    """Sort the selected files."""    def AppendToMenu(self,menu,window,data):        Link.AppendToMenu(self,menu,window,data)        menuItem = wx.MenuItem(menu,self.id,_('Sort'))        menu.AppendItem(menuItem)        if len(data) < 2: menuItem.Enable(False)    def Execute(self,event):        """Handle menu selection."""        message = _("This command will sort the selected files alphabetically, assigning them dates one minute after each other. It cannot be undone.\n\nNote that some mods need to be in a specific order to work correctly, and this sort operation may break that order.")        if ContinueQuery(self.window,message,'mash.sortMods.continue',_('Sort Mods')) != wx.ID_OK:            return        #--Scan for earliest date        fileInfos = self.window.data        newTime = min(fileInfos[fileName].mtime for fileName in self.data)        #--Do it        for fileName in sorted(self.data,key=lambda a: a[:-4].lower()):            fileInfos[fileName].setMTime(newTime)            newTime += 60        #--Refresh        fileInfos.refreshDoubleTime()        self.window.Refresh()#--begin changes by abot#------------------------------------------------------------------------------class File_UpdateTime(Link):    def AppendToMenu(self,menu,window,data):        Link.AppendToMenu(self,menu,window,data)        menuItem = wx.MenuItem(menu,self.id,_('Update time from current system time'))        menu.AppendItem(menuItem)        if len(data) < 1: menuItem.Enable(False)    def Execute(self,event):        message = _("This command will update the time of selected files, starting from current system time, keeping the same plugin order. It cannot be undone (at least, not in a single step).\nIt is meant to be used with files displayed in loading order. If you know what you are doing, this function is great to sort/move entire groups of files that must be loaded last, keeping the relative loading order (just ctrl+click or shift+click the files to organize).")        if ContinueQuery(self.window,message,'masher.File_UpdateTimeFromCurrentSystemTime.continue',_('Update time from current system time')) != wx.ID_OK:            return        #--Scan        fileInfos = self.window.data        newTime = time.time()        #--Do it        for fileName in self.data:            fileInfos[fileName].setMTime(newTime)            newTime += 60        #--Refresh        fileInfos.refreshDoubleTime()        self.window.Refresh()#------------------------------------------------------------------------------class File_UpdateTimeFromCurrentTime(Link):    """Sort the selected files."""    def AppendToMenu(self,menu,window,data):        Link.AppendToMenu(self,menu,window,data)        menuItem = wx.MenuItem(menu,self.id,_('Update Time from first selected file time'))        menu.AppendItem(menuItem)        if len(data) < 1: menuItem.Enable(False)    def Execute(self,event):        message = _("This command will update the time of selected files, starting from the time of first selected file, keeping the same file order. It cannot be undone (at least, not in a single step).\nIt is meant to be used with files displayed in loading order. If you know what you are doing, this function is great to sort/move entire groups of files keeping the relative loading order (just ctrl+click or shift+click the files to organize).")        if ContinueQuery(self.window,message,'masher.File_UpdateTimeFromCurrentFileTime.continue',_('Update Time from first selected file time')) != wx.ID_OK:            return        #--Scan for earliest date        fileInfos = self.window.data        newTime = fileInfos[self.data[0]].mtime        for fileName in self.data:            newTime = min(newTime,fileInfos[fileName].mtime)        #--Do it        for fileName in self.data:            fileInfos[fileName].setMTime(newTime)            newTime += 60        #--Refresh        fileInfos.refreshDoubleTime()        self.window.Refresh()# tes3cmd.exe clean --replace [selected files]#------------------------------------------------------------------------------class File_tes3cmd(Link):    def AppendToMenu(self,menu,window,data):        Link.AppendToMenu(self,menu,window,data)        menuItem = wx.MenuItem(menu,self.id,_('tes3cmd.exe clean --replace [selected files] > out.txt'))        menu.AppendItem(menuItem)        if len(data) < 1: menuItem.Enable(False)    def Execute(self,event):        message = _("This command will call Data Files\\tes3cmd.exe clean --replace passing selected files.")        if ContinueQuery(self.window,message,'masher.File_tes3cmd.continue',_('tes3cmd.exe clean --replace [selected files]')) != wx.ID_OK:            return        #--Scan        fileInfos = self.window.data        path = os.path.join(settings['mwDir'],'Data Files')        exe = os.path.join(path,'tes3cmd.exe')        args = ['tes3cmd.exe']        args.append('clean')        for fileName in self.data:            args.append('"' + fileName + '"')        args.append('--replace')        #args.append('>')        #args.append('out.txt')        #--debug        #print exe        #for x in args:            #print x                cwd = os.getcwd()        os.chdir(path)        os.spawnv(os.P_NOWAIT,exe,args)        os.chdir(cwd)                #--Refresh        fileInfos.refreshDoubleTime()        self.window.Refresh()#------------------------------------------------------------------------------class File_SmartMerge(Link):    def AppendToMenu(self,menu,window,data):        Link.AppendToMenu(self,menu,window,data)        menuItem = wx.MenuItem(menu,self.id,_('SmartMerge selected files to smartmerged.esp'))        menu.AppendItem(menuItem)        if len(data) < 2: menuItem.Enable(False)    def Execute(self,event):        message = _("This command will call Data Files\\SmartMerger.exe passing selected files, creating smartmerged.esp.")        if ContinueQuery(self.window,message,'masher.File_SmartMerge.continue',_('SmartMerge selected files to smartmerged.esp')) != wx.ID_OK:            return        #--Scan        fileInfos = self.window.data        path = os.path.join(settings['mwDir'],'Data Files')        exe = os.path.join(path,'SmartMerger.exe')        args = ['SmartMerger.exe']        for fileName in self.data:            args.append('"' + fileName + '"')        #args.append('--reference')        args.append('smartmerged.esp')        #--debug        #print exe        #for x in args:            #print x                cwd = os.getcwd()        os.chdir(path)        os.spawnv(os.P_NOWAIT,exe,args)        os.chdir(cwd)                #--Refresh        fileInfos.refreshDoubleTime()        self.window.Refresh()#------------------------------------------------------------------------------class File_SmartMerge2(Link):    def AppendToMenu(self,menu,window,data):        Link.AppendToMenu(self,menu,window,data)        menuItem = wx.MenuItem(menu,self.id,_('SmartMerge selected files to smartmerged.esp with --no_disabled_fix option'))        menu.AppendItem(menuItem)        if len(data) < 2: menuItem.Enable(False)    def Execute(self,event):        message = _("This command will call Data Files\\SmartMerger.exe passing selected files with --no_disabled_fix option, creating smartmerged.esp.")        if ContinueQuery(self.window,message,'masher.File_SmartMerge2.continue',_('SmartMerge selected files to smartmerged.esp with --no_disabled_fix option')) != wx.ID_OK:            return        #--Scan        fileInfos = self.window.data        path = os.path.join(settings['mwDir'],'Data Files')        exe = os.path.join(path,'SmartMerger.exe')        args = ['SmartMerger.exe']        for fileName in self.data:            args.append('"' + fileName + '"')        args.append('--no_disabled_fix')        args.append('smartmerged.esp')        #--debug        #for x in args:            #print x                cwd = os.getcwd()        os.chdir(path)        os.spawnv(os.P_NOWAIT,exe,args)        os.chdir(cwd)                #--Refresh        fileInfos.refreshDoubleTime()        self.window.Refresh()#--end changes by abot#-- ... ( other Mash code here )        fileMenu.links.append(File_Sort())            #--begin changes by abot        fileMenu.links.append(File_UpdateTime())        fileMenu.links.append(File_UpdateTimeFromCurrentTime())        fileMenu.links.append(File_tes3cmd())        fileMenu.links.append(File_SmartMerge())        fileMenu.links.append(File_SmartMerge2())    #--end changes by abot

[EDIT]fixed double quote
[EDIT2]changed from args = [exe] to args = ['SmartMerger.exe'], seems to work better with path stripped
[EDIT3]updated; options to sort files like TESTool, to call tes3cmd, to call smartmerged with or without --no_disabled_fix option

Very good idea!


+1
A merging tool (eventually with the ability to merge ESM & ESP into a new ESM or new ESP).
Ability to merge object list and levelled list.
Ability to apply several fix (same fix as tes3cmd.exe)
Support for non-US version (i.e. for cells names, objects and dialogues with non-US characters)

A question though: sometimes compatible plugins require a specific load order. Does merging mods with that tool will allow to solve the load order issue?

Exemple of complex load order necessity:




Hi Abot, it's sounds very good. Is it already working as described and can you merge several mods from Mash this may?

Well, it should work exactly as giving the DOS command line.
So far I have just tried merging a few small mods (9 quest of the week serie), and comparing with merging the same mods with TESCS combine loaded plugin options. Major difference seems to be the injected scripting code to avoid the potential bug with uninitialized spawned/moved NPCs (which can be skipped if desired using the --no_disabled_fix option, I'm not sure I want it on each and every NPC)
I was hoping the --reference option could work in a way similar to TESTool merged objects, but it seems another thing.
So, for now I have replaced the --reference option with the --no_disabled_fix option, it is just a matter of changing a line of code, also I'm testing similar code to use tes3cmd.exe clean --replace [selected files], seems to work well , I will update previous post with my current complete code

For a wish list....too tempting...

It would be great to somehow be able to batch "cap" certain variables that tend to be very unbalanced (i.e. wgt, gold, hit damage).

For example...The BillyJoeMegaDungeon.mod has awesome dungeons and quest but for what ever reason all the weapons are uber godlike and worth more gold than you could ever spend. Having a command line argument that would automatically set a ceiling (or min?). So if i didn't want anything costing over 200 gold, i could process the mod with a ( -costmax 200 ) variable. Another use for this would be so you could automatically make every merchant in the mod(s) not have unlimited stock items by changing the "quantity" from -1 to a positive number.

My ideas actually are quite more lofty but there's a limit to time and energy

Actually as an alternate that might be reasonable (might).....how about output to a text file of whats in a mod (sorta like Testools output but with out the comparision to other mods...just the mods in order with whats in them). Of course skipping the crazy complicated stuff like cell mapping and landscapes or dialogues.

Example output(ish)

BillyJoeMegaDungeon.esp

Cell Billy 28,39
Cell Joe 29,39
Cell MegaDungeon (interior)
Weap (or whatever) (and then what ever variables that item entry has...i.e wgt,gold,chop,thrust..etc seperated by a space

That way people can do a easy eyeball check for unbalancing items so they could see what they need to go in and edit (if they so choose).

If its plain text and fairly formatted then it could be easily imported into excel or access if the user liked.

Enough reaching for stars for now...

I added that? Hmm.. Much more likely it's a side-effect that keeps the original creation date. I'm not sure. I'll try and add the prefix dropping on the next release, nice idea hadn't occured to me, just getting it to strip the main paths was the important part, extras are just plain nice/fun.



Unfortunately i don't know python yet so i can't really comment much on that. I'll look into what TESTool does and come back to this later, if i can remember...



It should support those already. Currently it doesn't care if it's esm, esp, or ess, it treats all equally. Guess it just depends on what you give it and what flags you enable. If it doesn't, give me something to work with so i can fix it



I would say yes it would. Generally the order you feed it to SmartMerger doesn't matter. UNLESS two mods happen to modify the same item's Subrecord, and within that subrecord the same group. IE: Two mods modify iron bolt to do 2-5 damage, while most other changes would go through it would post this out to you. Then you could change the order based on that.

There might be a slight difference in how it determines which changes and preferred if you don't put the master's first, since it can't then detect what is already the defaulted values for comparison. Becomes an all or nothing kinda deal.

BTW, all the 'damages' are one group, so all or nothing on that one for now, unless everyone wants me to tell it to split the 6 bytes apart and label them.



I agree, every NPC seems overkill, however i'd rather not worry about accidentally crashing the game that way than playing and having it crash all the sudden cause i talked to someone. Too annoying from past experience. It may just be those with scripts might not initialize right, so i might remove the forced script adding. Actually that might be right... I'll add an option to remove that. Kinda 'play at your own risk' at that point.

BTW, accidently having the fix on the player NPC was kinda funny, start a new game and you can see the snout of an argonian, and look down into his neck Safe to say i added a ignore for that npc.

Although interesting to note. I've been playing a 100+ mod merged version, and so far it hasn't crashed once, i was playing 2-3 hours at a time mostly saving every half hour or before doing something questionable. (I'm not counting the infinite loop in the dialog, I'm currently trying to fix this).

Regarding TESTool, i'll need to look at what it does before i decide what to do. --reference is intended so you have previous data (masters i recommend) to reference against when finding duplicates or defaults, so everything previously loaded is ignored for output unless something in it changes. It's sorta the option to allow basic GMST and duplicates cleaning.



Sounds easy enough, i got the framework of this all already built, adding 1 function to fix this won't be hard. However rather than a cap, i'd recommend a formula, that way you don't suddenly have every enchanted sword in the game as 200 gold when it needs to be higher. But sometimes you just want to cap one thing...

Just an example, i haven't tested this, but for armor it could work too, once the community decides an appropriate formula to use. If used with an appropriate economy plan, it would balance a lot. Might remove the need for those balancing mods.

dagger minweight = .5
dagger maxweight = 5
dagger normalweight = 1.5
per maxdamage = 5%
per mindamage = 2%
enchantment = 500%
base gold = 15
max gold = 2000
price = weight / minweight * (1 + (chopmin + thrustmin + slashmin) * mindamage) + ((chopmax + thrustmax + slashmax) * maxdamage) * enchantment * basegold

With the formula, heavier daggers become cheaper, and lighter ones more expensive, and the more they do, the more powerful they are. Enchantment? Add X5 (or something) to it's price. I wonder if the enchantment should be considered, and if so actually read it's contents and make a decision on that. That's getting complex, and would require deciding what's the ratio for each item. IE Strength is higher than feather, so feather is 1/8th the value of 1 strength. (i say 8th since you don't get the damage or any other bonuses, otherwise it's 1/5 or 1/6)



Currently already supported. Although you have to do a round-about way to do it. Do this.

>SmartMerger.exe myMod.esp --query 2>output.txt
then asterisk and enter twice. All done That outputs everything in the file, in detail. (if it references an item, only the item, not the item's properties)

If you wanted just the list of what's in it (and not full blown details) then this should work.

>SmartMerger.exe myMod.esp

EDIT: Correction on that last part. Re-reading it i see what you are meaning. It wouldn't be hard to add, but would take a little work. Right now it does give detailed output, i'll see about adding that in.

When i was designing a similar tool up (devoted way way too many hours for what was just an intellectual exercise) one of the things i looked at was how to handle this situation. One idea was to set a threshold for every variable (in a txt file or something) then when the program runs into something that exceeds (or is less than) a given variable, then the program would process with a given algorithm. Since the ideas wouldn't be to make the threshold the max or min, there would be a percentage chance of going outside the threshold norms by a preset range (random) and thus retain an element of unpredicability.

Example:
(ThresholdMINMAX.txt)
(VariableName min max %chancetoexceedThreshold CanExceedGiven%)
Dagger
Weight min1 max20 chance5 range20
Gold min0 max 200 chance10 range10
Chop min0 max 5 chance20 range20
Thrust min1 max 15 chance 20 range10
Slash min2 max5
Durability min1 max100
(etc)

Then the program comes across a dagger that is outside the norms...say Gold value is set to 50000 (seen it) then it would apply a formula. So the program would tweak the value according to the threshold rules and 5% of the time allow the value to exceed the max by up to 10%. The rest of the time it would make the value something less than 201 (but not always 200 cuz that would be dull).

Hope that kind of makes sense. It's a little rough and my little one is insisting in "battling" me while I type.

This way if you had the program read the variables in from a file, then everyone can tweak as they prefer or even they could use prebalanced files provided by the community. You formula looked better by the way.

Also, thanks for the output.txt thing, i didn't realize it did that. Hopefully I can try it soon.

Actually, for me, a replacement for TESTool's Merged Object function is what I'm most interested in. The current implementation in TESTool is broken (too many records busts a buffer or something) and esper just doesn't function correctly. Sadly there's no source for TESTool available but esper's is.

I guess I saw "merge" and imagined it was the TESTool functionality this was duplicating rather than the Combine Loaded Plugins of the Construction Set. Teach me not to read things correctly...

Oh, and @abot: Nice integration into Mash. It'd be good if, when this is finalised, a replacement mash.py could be bundled with it. Shouldn't be too hard to set-up the paths. This would mean that people could install this and not go near a command line (which a lot of people hate) and it spares Era from writing a GUI for the program's main uses.

Anyway, they're my thoughts. Take as much or as little as you like from them.

@ Abot: thanks! I am not aware of the bug with uninitialized spawned/moved NPCs, will try to be careful.


May I had another wish?
- in such case, could the merger tool have an option to show the conflicting values? For example: the first mod modifies iron bolt weight and damage to 2-5, the second one modifies only damage to 3-8. I would like to have the choice during the merging process to modify weight according to the first mod, and damage according to second... but in some other case, I may want to keep (or not) weight and damage from the first mod only (2-5).
- can we hope to have a simple but friendly use interface? (merging several mod may lead to very long command lines :blush: ...)


Testool merge function has also a bug with NPC and "auto calculate stats"
And Esper does not work with non-English mods

Hmm. I have instructions on how unicode works, so i could forcibly use that; Or use wide characters for a alternate build. However i still need an example file. Can someone provide me with an example esp/esm? Preferably with something like 50+ records spanning from creatures and static objects, to NPC's and cells. And at least 1 dialog entry. With that I'll be able to make an intelligent choice on how to implement it.

I really do hope it's UTF-8, since that will be easiest to implement, since then i only modify what string compare function i use

BTW - On my ID matching i have it case-insensitive. Does anyone think this is unwise? Afterall if i get into unicode (not currently using regex), it will become a lot more strict by default.

The http://translate.google.co.uk/translate?u=http%3A%2F%2Fwww.sureai.de%2Findex.php%3Foption%3Dcom_content%26task%3Dview%26id%3D43%26Itemid%3D64&sl=de&tl=en&hl=&ie=UTF-8 (Google translation) has some umlauts in some of the dialogue topics. Does that help?

Not sure. I've downloaded it, i'll take a look at it soon. I finally got 1.1 done, but i don't feel like uploading it right now, i need a few hours of gametime after my 2 days of working on something that shouldn't have taken nearly as long.

BTW, someone commented about ID clash detection, i've added that, and it will also spit out notes on duplicate ID's with dialogs (which were silent before). It's incredible the number of duplicates in the Morrowind.esm file, i'll post them here just for fun. Most of these have 3-5 each. In my list there's 211 duplicates. I didn't think it was anywhere near this many honestly.

confirm, don't, It's up to you.

I believe ID matching should always be insensitive to case. Anyway, that's how I've been doing it.

Are you referring to the WEAP.WPDT subrecord? This is what I currently have in http://code.google.com/p/mlox/source/browse/trunk/util/tes3cmd:

[WPDT =>
[["weight", "f"],
["value", "L"],
["type", "S"],
["health", "S"],
["speed", "f"],
["reach", "f"],
["enchantment", "S"],
["chop_min", "C"],
["chop_max", "C"],
["slash_min", "C"],
["slash_max", "C"],
["thrust_min", "C"],
["thrust_max", "C"],
["flags", "L"],

"f" - 32bit float
"S" - 16bit unsigned short
"C" - 8bit unsigned char
"L" - 32bit unsigned long

Let me know if you have questions about decoding, I've got some stuff worked out but not documented yet (particularly for savegames).