Announcing the "beta" release of Esper.

For some time it has puzzled me that the only way to merge objects in Morrowind is by using "TESTool". This is because TESTool is notoriously prone to crashing during a merge -- the only way most people can get it to behave is to ensure no more than 10-20 mods are enabled when it runs. I suspect this means most people don't bother merging objects.

Because I haven't been able to find "merge objects" functionality anywhere else, I have started to write it myself. Esper is the result. Esper is written in Common Lisp.

The latest version is available at: http://bitbucket.org/eeeickythump/esper/

See the README there for the most up to date description of esper's capabilities and command line options (more up to date than this post).

Zipped Windows XP executable: http://bitbucket.org/eeeickythump/esper/downloads/

FallenWizard has kindly compiled a native executable for 64-bit machines: http://www.4shared.com/file/258967438/967b2769/esper64_Beta16.html

A discussion thread is http://www.gamesas.com/bgsforums/index.php?showtopic=1067443&st=20. It is probably best to discuss bugs and glitches there.

Currently, Esper successfully parses Morrowind.esm and the two addon esm's, and all the plugin files I've tried. It also writes Morrowind.esm to, and reads it from, ESTXT format (see below for what this means). The TES construction set seems to load the merged plugin files OK. Several people have reported that the plugin produced by esper works in-game.

Esper can also read and write files in a format called "ESTXT". This is a plain-text file containing lisp expressions, one for each record, slightly simplified to be more readable. If you specify an output file whose name ends in ".estxt", output will be in this format rather than binary. Similarly if any of the input filenames end in .estxt, they will be treated as these text files.

Example of an ESTXT record (currently does not expunge null characters from strings):

(CREA 486 (CREA NAME 23 (STRING-VALUE "winged twilight_summon"))  (CREA MODL 21 (STRING-VALUE "r\\WingedTwilight.NIF"))  (CREA FNAM 16 (STRING-VALUE "Winged Twilight"))  (CREA NPDT 96 (CREATURE-TYPE AEDRA) (CREATURE-LEVEL 15)  (CREATURE-STRENGTH 50) (CREATURE-INTELLIGENCE 50) (CREATURE-WILLPOWER 100)  (CREATURE-AGILITY 50) (CREATURE-SPEED 8) (CREATURE-ENDURANCE 50)  (CREATURE-PERSONALITY 50) (CREATURE-LUCK 50) (CREATURE-HEALTH 220)  (CREATURE-SPELL-POINTS 210) (CREATURE-FATIGUE 800) (CREATURE-SOUL 300)  (CREATURE-COMBAT 70) (CREATURE-MAGIC 90) (CREATURE-STEALTH 20)  (CREATURE-ATTACKMIN1 15) (CREATURE-ATTACKMAX1 45) (CREATURE-ATTACKMIN2 15)  (CREATURE-ATTACKMAX2 45) (CREATURE-ATTACKMIN3 15) (CREATURE-ATTACKMAX3 45)  (CREATURE-GOLD 0))  (CREA FLAG 4 (CREATURE-FLAGS (:NONE? :WALKS)))  (CREA NPCO 36 (ITEM-COUNT 3) (ITEM-NAME "random_void_salts"))  (CREA NPCS 32 (STRING-VALUE "resist fire_50"))  (CREA NPCS 32 (STRING-VALUE "resist frost_50"))  (CREA NPCS 32 (STRING-VALUE "resist poison_75"))  (CREA NPCS 32 (STRING-VALUE "resist shock_50"))  (CREA NPCS 32 (STRING-VALUE "reflect_20"))  (CREA AIDT 12 (AI-HELLO 0) (AI-UNKNOWN1 0) (AI-FIGHT 50) (AI-FLEE 20)  (AI-ALARM 0) (AI-UNKNOWN2 0) (AI-UNKNOWN3 0) (AI-UNKNOWN4 0) (AI-FLAGS NIL))  (CREA AI_W 14 (AI-DISTANCE 1000) (AI-DURATION 5) (AI-TIME-OF-DAY 0)  (AI-IDLE (0 50 50 0 0 0 0 0 0 1))))

Usage:

ESPER can be used from any directory, but it's easiest to put it in your "Data Files" directory; then you won't need to give paths to esp/esm files.

Basic usage:

esper [OPTIONS] [FILE...]

Options:

--help, -h
Show help

--output , -o
Save output to . By default this is 'output.esp' in the current directory.

--merge-plugins, -p
Mode = merge plugins

--merge-records, -r
Mode = merge records from plugins

--list-active, -l
Mode = list active plugins and masters in Morrowind.ini

--check-resources
Scan plugins for names of texture/icon/sound/model files, then look in appropriate directories and active BSA files. Report if the files cannot be found.

--find-conflicts
Reports names of all records which are modified by more than one plugin (along with names of plugins).

--mwdir , -d
Morrowind.ini directory is

--all, -a
Use *ALL* active plugins when merging (!)

--debug
Usually esper just exits if an error occurs. Using --debug makes esper enter the lisp debugger instead.

--batchfile , -B
Where is a text file containing one plugin file name per line. Uses the plugins listed in this file as if they had been specified on the command line.
Note: you must use -B or --batchfile; -b does not work.

Notes

"PGRD" records are not parsed at all, simply treated as binary "blobs".

Record types that are currently treated as mergeable:

	"CLAS" "RACE" "FACT" "SOUN" "SKIL" "MGEF" "REGN" "BSGN" "STAT" "DOOR" "MISC"	"WEAP" "CONT" "SPEL" "CREA" "BODY" "LIGH" "ENCH" "NPC_" "ARMO" "CLOT"	"REPA" "ACTI" "APPA" "LOCK" "PROB" "INGR" "BOOK" "ALCH"

Note that esper does not currently merge levelled lists.

Examples of usage:

See usage information with:

	esper --help

List all active plugins in order of loading with:

	esper -l

Merge plugin1 and plugin2.esp into `output.esp': (this works like the TES
construction set facility of the same name)

	esper -p plugin1.esp plugin2.esp

The wildcard '*' can be used, but if you do this you must surround the 'wildcard expression' in quotes:

	esper -p "LGNPC_*.esp"

Merge RECORDS (objects, NPCS, etc) from plugin1 and plugin2 into merged-objects.esp:

	esper -r plugin1.esp plugin2.esp -o merged-objects.esp

Merge records from all plugins into an ESTXT file:

	esper -a -r -o bigmerge.estxt

Please test Esper out! It has been tested by at least half a dozen people and the resulting plugin seems to work for most people. Still, the occasional bug does pop up. The first thing to try if a particular Morrowind plugin causes problems is to try re-downloading the plugin -- several people have reported that this has solved their problem. It seems that some other Morrowind utilities can sometimes corrupt plugins (Enchanted Editor has been involved more than once).

1. Am I wasting my time - does Wrye Mash or another utility already do this without crashing?

Mash only merges Leveled Lists.

2. Apart from the obvious (dialogues, "LAND" etc), which records should be merged and which should not? Is there a reason no-one talks about merging creatures or NPCs?

TesTool merges creatures and NPCs as far as I know but it can break scripts during the operation. (That's why mlox suggests to disable CreaturesX while merging objects)

3. Is TESTool's source code publically available? I have not been able to find it.

No

4. How should I deal with container/inventory records (records that contain an arbitrary number of "contents" subrecords)? Should I replace the entire inventory, or add the two inventories together?

The second one please.

5. Is there any other record-manipulating functionality that would be useful in a command-line program? I have implemented "merge plugins" -- just call the program with the esps to be merged on the command line, and it outputs a single plugin. (Hmm, this would make it easy to merge ALL active plugins into one_big_blob.esp ... might make loading much faster)

Export/Import texts and cell names to make it easier to translate english mods to another language versions. That would make it a TON easier to localize mods. Also please make it possible to update headers like TesTool does. MW will not complain about master versions at the start then.

How about if the program could read and write mods to a relatively "human readable" file format? You could then load the file in a text editor, search for all strings, or for fields named "TEXT", "DESC" etc, edit them, and then turn the changed file back into ESP format.

eg the file might contain records that look like:

(CREA
(CREA NAME "Clannfear")
(CREA DATA ........)
(CREA DESC "Description of clannfear"))

Of course creatures don't have descriptions, but you get the idea.

I would welcome a new merge object tool, however, what would be cool is if someone could write a merge landscape utility. Right now if two mods alter the same cell the last loaded mod wins. A merge landscape tool would merge the landscape data. However if the landscape is changed in the exact location then the last loaded mod would win. The merge utility would allow two mods to alter the landscape in the same cell and work together provided they did not change the exact same area.

It would be great to have a good object merger tool.

I've seen a few discussion of TESTOOL's object merging in the past, along with some anolysis of its algorithm, like in http://www.gamesas.com/bgsforums/index.php?showtopic=919790. I wish I'd bookmarked the others, but I didn't. If I find them I'll link them.

I'd also look at how Bash does its Bashed Patch, as it has some very nice features. For example, IIRC you can tell Bash you just want to merge head/hairs from a plugin, this allows you to easily add an NPC face replacer, but without merging any of the other NPC properties from that plugin.

Being able to include or exclude plugins from the merge would be pretty crucial too.

I think it would be awesome to have a new, reliable tool for merging objects. I'm eager to see how progress on this goes. :^)

I'm afraid I can't be much use, but I do remember http://www.gamesas.com/bgsforums/index.php?showtopic=919790 from awhile ago that may have some helpful information for you.

@john.moonsugar and Bycote: thanks for that thread, it looks useful.

@DesertRat: would this involve merging CELL records or LAND records? The structure of LAND records seems to be a mystery in the MW modding community (please correct me if wrong). Probably involves heightmaps and such. The data in CELL records includes such things as whether the cell is interior or exterior, lighting, location of exists etc.

More info about the project: it's called "esper" and is hosted at

http://bitbucket.org/eeeickythump/esper/

It's written in Common Lisp. Unless you know about lisp, it won't be usable to you at the moment as the command-line functionality is not in place yet. The version there at the moment doesn't parse CELL records correctly (have fixed this, but corrections not yet uploaded).

Dude, that's really cool. I've always thought Lisp was a neat language. I'll have to check out what's available for Windows these days.

I don't know if this will help, but my http://code.google.com/p/mlox/wiki/Tes3cmd can print out plugins as text. Sadly, the docs are out of date but it does a lot of stuff. I had to figure out some things that weren't in available docs on the esp format.

:dance: You are definitely not wasting your time. An object merger that doesn't have the index-related (?) crash that TESTool has would be terrific.
If I generate a Merged Objects and peek at it in Enchanted Editor then I see these record types:
Activators
Alchemy
Alchemy Apparatus
Armour Types
Books
Clothing
Containers
Creatures (as to Creatures X and not including it in the Merge, this was never definitive. It seemed to work for people who had the huge Guar herd problem. But not everyone who used Merged Objects had the problem. So, my belief is there is something screwy in the way Creature records are merged by TESTool)
Enchantments
Factions
Ingredients
Items Miscellaneous
Lights
Lock Picking Items
Magic Effects
NPCs
Races
Regions
Repair Equipment
Skills
Spells
Weapons
:( No
Add the two together
Simply doing a Merged Objects replacer would be a god send.

Although I know what Merged Objects does, I can't really add any technical help on this matter, but I'd love to see a new tool for Merging!
I've had troubles with TESTools' Merge Obj. in the past.
Full support from me.

What they said.

I am not sure what is currently known about the land records. However Yacoby made a Cell to Nif convertor ( http://www.tesnexus.com/downloads/file.php?id=20928) There is the grass generator program that has to read the terrain map to place the grass for MGE.

Well anyway I look forward to just a new merger. Hopefully someone will figure out how to merge the land data

Oops, didn't notice that john.moonsugar already posted a link to that thread before me. Sorry john, didn't mean to repeat you, I feel embarrassed now.

Besides that, I just want to repeat how excited I am for this utility to actually come about. I'm just setting up a new Morrowind installation now so this is perfect timing! :^)

Thing about merging inventories (of containers, creatures, NPCs, etc)... An issue with TESTool's method is that changing the count from negative to positive isn't registered as a change to be merged, so negative counts end up reinserted in Merged Objects.esp. This is a problem for, e.g., PTE (or any other mod that wants to turn an infinite inventory item into a finite one). Would be nice if you could handle that more neatly.

Otherwise, all I have to say is: Good luck, looking forward to using it!

Clothing mods are one of those that can go screwy when merged. For example Better Clothes merged with PTE, which changes vanilla clothing, messes things up. (Thankfully there is a patch included for that.) I think the problem stems from the more complex parts of an article of clothing. It would be nice to have more control over that process. For example, a setting in the program that when selected wouldn't merge the entries that cause this type of problem but will still merge other changes like gold value, weight, etc. Not sure if that all makes sense but that's the best way I can put it as a non-modder. Maybe someone more knowledgeable could expound on that.

If you want to merge land and cell records you should check out TESFaith at http://projectmanager.f2s.com/morrowind/TESfaith.html which comes with the source. TESFaith moves landmasses if you don't already know so probably has some useful data on merging land and cell records.

Shouldn't be any problem with the negative/positive value issue; as it works now, if two integer values are different then the newer one overwrites the older.

Thanks for the pointer to TESFaith - although I don't have plans to deal with "LAND" records at present (maybe later).

Clothes records don't look especially complicated to me -- I'm not sure what causes problems when TESTool merges them, it may be an issue with TESTool.

The more tools, the better!
A little unrelated, I'd love an automated tool to fix interior cells with 0 fog settings, many big mods still have this problem and fixing by hand can be tedious.
TES3lint can signal the problem, but not fix, I think something able to scan all mods and autofix would be nice and not so difficult to implement/add to an existing tool (I think adding this to Mash or TES3lint might be the faster way for someone knowing Python/Perl well)

[EDIT]TES3lint

http://code.google.com/p/mlox/wiki/Tes3cmd#Patch_Fogbugged_CELLs will do it!

Ah, how I missed it!, great! ... back refreshing my dos batch FOR command memories, I'll see if I am able to cook a dos batch to scan & fix all mods! :twirl:

Terrific news, eeeickythump!

Will you eventually build on a gui interface?


Almost expected you to have written this using Python.
"We are no longer the knights who say nee..."

No need. Tes3cmd figures out where your Data Files is from where it is installed, scans all plugins, and produces a patch which fixes all fogbugs for all current plugins.

Edit: P.S. I apologize for hijacking this thread!

I apologize too! Just realized and tested, it works well John! :goodjob: ( only thing I did not realize at first is that script needs to be in Data Files directory).
To try to remain more in-thread, as a mod author, I'd like more to directly be able to fix the original mod(s)... I think this could still be a suggestion for a possible new tool option

Tes3cmd can do that, but the solution isn't obvious to anyone but me:). I moved my answer to http://www.gamesas.com/bgsforums/index.php?s=&showtopic=934107&view=findpost&p=15514532.

Unlikely -- though if someone else wants to they are welcome.



Python is a good programming language, but once you've programmed with lisp's macros, you can never go back...

Anyway -- I have now released beta 1 of esper. Source code and windows EXE are available as links from the first post in this thread. I assume almost everyone who plays Morrowind can run winXP executables.

Look forward to reports of horrible crashes and misbehaviour...