[WIP] New merge objects program

Post » Tue Mar 15, 2011 5:52 am

eeeickythump: I thank you. I was able to run esper inside CCL and I was able to compile it.

To help you a little bit out, I will compile unofficial 64 bit versions of esper from now on since you are not able to.

I hope you don't mind.



http://www.4shared.com/file/233935719/778dd1a2/esper64_Beta13.html
User avatar
alicia hillier
 
Posts: 3387
Joined: Tue Feb 06, 2007 2:57 am

Post » Mon Mar 14, 2011 1:52 pm

OK, took beta 13 out for a spin in my modded-to-hell-and-back install.

Couple of things in warnings.txt:
Region 'Bitter Coast Region' Weather Chances do not total 100 percent.
Here's the final part of esper's Region merging:
..Merged records:Original: (I:/Bethesda Softworks/Morrowind/Data Files/Morrowind.esm)#   #   #   #   #   #   #   #   #   #   #   #   #   #   #   #   #>New:#   #   #   #   #   #   #   #   #   #   #   #   #   #   #   #   #>Merged:#   #   #   #   #   #   #   #   #   #   #   #   #   #   #   #   #>
Quite a number of other plugins change the Bitter Coast region, not included.
Unable to find script '_BME_A_Ice minions_Shield2' on object '_BME_Ice_minion_Shield2'.
This is a weird one as there is no script defined on this object.
Merged records:Original: (I:/Bethesda Softworks/Morrowind/Data Files/BT_Whitewolf_2_0.esp)#   #   #   #   #   #>New:#   #   #   #   #   #   #>Merged:#   #   #   #   #   #   #>
The displacement of the male part ID to be the first subrecord seems to be causing this problem. Also, esper's moving of a (fe)male part ID to be the first subrecord, before ID and Modelpath, seems to also be messing up icons. Somehow. F'rinstance, this one shows the old smiley face:
Merged records:Original: (I:/Bethesda Softworks/Morrowind/Data Files/Morrowind.esm)#   #   #   #   #   #   #   #   #   #   #   #   #>New:#   #   #   #   #   #   #   #   #   #   #   #>Merged:#   #   #   #   #   #   #   #   #   #   #   #   #   #>
Actually walking around in game I saw a lot of missing bodyparts (see my earlier post and guenthar's), I think this whole bnam / cnam displacement might be a problem, as well as the mulitple bnam / cnam subrecords. As an example (dump of output.esp from Enchanted Editor):
CLOT	CNAM	BC_common_shirt_02_hh_FwCLOT	CNAM	BC_common_shirt_02_hh_FfCLOT	CNAM	BC_common_shirt_02_hh_FuaCLOT	CNAM	BC_common_shirt_02_hh_FwCLOT	CNAM	BC_common_shirt_02_hh_FfCLOT	CNAM	BC_common_shirt_02_hh_FuaCLOT	CNAM	BC_common_shirt_02_hh_FCLOT	CNAM	c_f_shirt_common_2hh_cCLOT	NAME	common_shirt_02_hhCLOT	MODL	c\C_M_Shirt_Common_2hh_GND.NIFCLOT	FNAM	Common ShirtCLOT	CTDT	2CLOT	CTDT	2CLOT	CTDT	4CLOT	CTDT	20CLOT	ITEX	c\tx_shirtcomm_2hh.tgaCLOT	INDX	CLOT	BNAM	BC_common_shirt_02_hh_MwCLOT	INDX	CLOT	BNAM	c_m_shirt_common_2hh_uaCLOT	INDX	CLOT	BNAM	c_m_shirt_common_2hh_uaCLOT	INDX	CLOT	BNAM	c_m_shirt_common_2hh_faCLOT	INDX	CLOT	BNAM	c_m_shirt_common_2hh_faCLOT	INDX	CLOT	BNAM	c_m_shirt_common_2hh_wCLOT	INDX	CLOT	BNAM	c_m_shirt_common_2hh_w
I use the All Silt Strider Ports and All Boat Ports mods. These were created (I think) with Enchanted Editor and add all possible Silt Strider / Boat destinations to the travel NPCs, rather than just the four one has with the vanilla game. Anyway, on merge I lose all those extra destinations:
Merged records:Original: (I:/Bethesda Softworks/Morrowind/Data Files/Morrowind.esm)#   #   #   #   #   #   #   #   #   #   #   #   #   #   #   #   #   #>New:#   #   #   #   #   #   #   #   #   #   #   #   #   #   #   #   #   #   #   #   #   #>Merged:#   #   #   #   #   #   #   #   #   #   #   #   #   #   #   #   #   #   #>
It'd be nice to keep them :)

[Edit: Oh, something else. I have a mod that adds Hurdy Gurdy's robes to members of the Mages Guild. But for some of them it doesn't seem to be surviving the esperisation, see Galbedir here:
Merged records:Original: (I:/Bethesda Softworks/Morrowind/Data Files/Morrowind.esm)#   #   #   #   #   #   #   #   #   #   #   #   #   #   #   #   #   #   #   #   #   #   #   #   #   #   #   #   #   #   #   #   #   #   #   #   #   #   #   #   #   #   #   #   #   #   #>New:#   #   #   #   #   #   #   #   #   #   #   #   #   #   #   #   #   #   #   #   #   #   #   #   #   #   #   #   #   #   #   #   #   #   #   #   #   #   #   #   #   #   #   #   #   #   #   #>Merged:#   #   #   #   #   #   #   #   #   #   #   #   #   #   #   #   #   #   #   #   #   #   #   #   #   #   #   #   #   #   #   #   #   #   #   #   #   #   #   #   #   #   #   #   #   #   #>
]
User avatar
Neil
 
Posts: 3357
Joined: Sat Jul 14, 2007 5:08 am

Post » Mon Mar 14, 2011 9:32 pm

I forgot to mention before that the problem with the missing body parts didn't start happening until version 12 or 13. Last time I built my merged objects patch was with version 11 and I didn't have this problem.
User avatar
Tha King o Geekz
 
Posts: 3556
Joined: Mon May 07, 2007 9:14 pm

Post » Mon Mar 14, 2011 11:11 pm

Hi guys

Esper beta 14 is up. Bug reports in this thread allowed me to identify two very important bugs. I hope that fixing them has solved most of the problems recently described. I suggest everybody redoes their "merge records". The caches should not need to be redone.

@FallenWizard:
eeeickythump: I thank you. I was able to run esper inside CCL and I was able to compile it.

To help you a little bit out, I will compile unofficial 64 bit versions of esper from now on since you are not able to.

I hope you don't mind.


Not at all, I'm very impressed! Common lisp is often quite difficult to get set up correctly.

You now have the advantage that you don't need to download the zip files, you can just clone the repository and recompile.

@Dragon32:
Assuming I'm right (and I haven't checked this against the full text output to check which plugins are modifying this object) these multiple BNAMs (male body part) and CNAMs (female body part) are what is causing the problem here. I'd suggest that for CLOT and ARMO a "last loaded mod wins" approach is taken for which BNAM or CNAM takes precedent. Assuming that (when including the Bethesda ESMs) the BNAM or CNAM isn't the default, Bethesda value.


I looked at esper's output myself and all the armour/clothing items with multiple BNAMs/CNAMs seem to be correct. Each one is for a different body part slot. Eg "pants" will have BNAMS & CNAMs for "groin", "right leg", "left leg", "right foot" and "left foot".

The "last loaded mod wins" approach is what WAS used, but it was not correct for armour and clothing -- see earlier posts in the thread.

Oh, and as to the plugin name the new, default output file being called "esper_output.esp" is fine. What I was getting at was that if we could ensure that when someone does a "esper -a -r" for merged objects then that name is common to as many as possible, of course people can just rename it. By having a common name for this output, which I think will be the main use of esper, then I can amend the mlox rules to match this default name.


Got it. OK, next version I will change the name again.

Oh, and btw what's the reason behind Linux users' antipathy to DOS text (CR + LF)? I know Unix-format is linefeed only but I'm intrigued... Both you and John.Moonsugar seem to suffer from it smile.gif


Not active antipathy, just that all unix tools naturally save text files in unix format, so I'd have to specifically re-save them in Windows to get the CR+LF line endings.
Also text files from DOS/Windows look ugly when loaded in unix as each line ends in a ^M.

I can't test with a cache as I get this error when using --build-cache


Fixed.

OK, took beta 13 out for a spin in my modded-to-hell-and-back install.

Couple of things in warnings.txt:
Here's the final part of esper's Region merging:
..Merged records:Original: (I:/Bethesda Softworks/Morrowind/Data Files/Morrowind.esm)#	REGION-SOUND-NAME: "wind calm1?$?Xm??Ud?g"


Doesn't look like it's reading region-sound-name correctly, though that may not actually be causing any problems.

Looks like esper may need to manually recalculate the "sound chances" for regions itself to make them add up to 100%.

The displacement of the male part ID to be the first subrecord seems to be causing this problem. Also, esper's moving of a (fe)male part ID to be the first subrecord, before ID and Modelpath, seems to also be messing up icons. Somehow.


You are right, it was adding "new" subrecords to the start rather than the end of the record. Fixed.

[Edit: Oh, something else. I have a mod that adds Hurdy Gurdy's robes to members of the Mages Guild. But for some of them it doesn't seem to be surviving the esperisation, see Galbedir here:


Try it with the new version: the fix to subrecord comparisons may have fixed your problem.

@quenthar:
I forgot to mention before that the problem with the missing body parts didn't start happening until version 12 or 13. Last time I built my merged objects patch was with version 11 and I didn't have this problem.


Yup, hopefully the bugfixes mentioned above have fixed your problem. Give it another try and see.

Next tasks:
- option to force autocalc for NPCs with "short" data subrecords.
- change default name to esper_merged_records.esp
User avatar
Sunnii Bebiieh
 
Posts: 3454
Joined: Wed Apr 11, 2007 7:57 pm

Post » Mon Mar 14, 2011 2:07 pm

Not at all, I'm very impressed! Common lisp is often quite difficult to get set up correctly.

You now have the advantage that you don't need to download the zip files, you can just clone the repository and recompile.


I am using Arch Linux with dwm/irssi/mutt/vim etc.... I am used to configure everything by hand and I couldn't do it without your help.

Thanks for the update! I uploaded a new version of esper64.

http://www.4shared.com/file/235229076/ca862ec1/esper64_Beta14.html
User avatar
Andrew
 
Posts: 3521
Joined: Tue May 08, 2007 1:44 am

Post » Mon Mar 14, 2011 2:39 pm

I don't have the missing body parts anymore and don't see any other problems.

PS. Just to let you know my name starts with a g.
User avatar
Ross
 
Posts: 3384
Joined: Thu Aug 10, 2006 7:22 pm

Post » Mon Mar 14, 2011 8:06 pm

Thanks for the new beta, eeeickythump. I'll let you know how I get on.

[Edit - Quick info on timings with Beta14:

ESPER -A -R
09'05"
09'50"

ESPER --build-cache (works :twirl:)
05'17"
05'18"

ESPER -A -R --use-cache
06'13"
06'13"]
User avatar
Life long Observer
 
Posts: 3476
Joined: Fri Sep 08, 2006 7:07 pm

Post » Mon Mar 14, 2011 5:27 pm

Played the game for a few days with Esper instead of Testool and it runs VERY stable. I haven't seen a single problem yet.


Good work eeeickythump. There are still some annoyances left (For example the 1234 bytes header) but they are easily fixed with Mash and Testool.


My opinion: The next version should be a 1.0 version.
User avatar
Laura Cartwright
 
Posts: 3483
Joined: Mon Sep 25, 2006 6:12 pm

Post » Mon Mar 14, 2011 6:40 pm

Hi!

Will you make a GUI for this program?
User avatar
Kill Bill
 
Posts: 3355
Joined: Wed Aug 30, 2006 2:22 am

Post » Tue Mar 15, 2011 3:24 am

Tried esper14 with -a -p -o espered.esp and got this error:
E:/Games/Morrowind/Data Files/DN_AshVampires.esp In file "E:/Games/Morrowind/Data Files/DN_AshVampires.esp", record type "CREA", last subrecord "NPCS":
SIMPLE-ERROR: Illegal record name "ndus" at position 2321 in "E:/Games/Morrowind/Data Files/DN_AshVampires.esp"
Exiting...

The offending file is http://planetelderscrolls.gamespy.com/View.php?view=Mods.Detail&id=6043.
User avatar
Horror- Puppe
 
Posts: 3376
Joined: Fri Apr 13, 2007 11:09 am

Post » Mon Mar 14, 2011 10:18 pm

Feature request: when you do a merge all records (the equivalent TESTool's Merge Objects For Active Plugins) could you give the output.esp a special default file name that the MLOX load order utility can use to recognize that it's a replacement for Merged_Objects.esp?
User avatar
Leah
 
Posts: 3358
Joined: Wed Nov 01, 2006 3:11 pm

Post » Mon Mar 14, 2011 11:41 pm

Tried esper14 with -a -p -o espered.esp and got this error:
E:/Games/Morrowind/Data Files/DN_AshVampires.esp In file "E:/Games/Morrowind/Data Files/DN_AshVampires.esp", record type "CREA", last subrecord "NPCS":
SIMPLE-ERROR: Illegal record name "ndus" at position 2321 in "E:/Games/Morrowind/Data Files/DN_AshVampires.esp"
Exiting...

The offending file is http://planetelderscrolls.gamespy.com/View.php?view=Mods.Detail&id=6043.


Hi,
I downloaded that esp (from the "elric's mods" site, as the file has actually been deleted from planetelderscrolls). It went through merge-records and merge-plugins without a problem.
Could your plugin have been modified? (Dragon32 had a similar problem earleir in the thread where some plugins had been corrupted by another program).

To the person who asked about a GUI: no I won't be making one. There will be a batch file to "merge all records". If you really want a GUI perhaps talk to the maintainers of Wrye Mash, who might be able to make their program call esper.
User avatar
Casey
 
Posts: 3376
Joined: Mon Nov 12, 2007 8:38 am

Post » Tue Mar 15, 2011 2:51 am

Hi,
I downloaded that esp (from the "elric's mods" site, as the file has actually been deleted from planetelderscrolls). It went through merge-records and merge-plugins without a problem.
Could your plugin have been modified? (Dragon32 had a similar problem earleir in the thread where some plugins had been corrupted by another program).


Ok, thanks for reply. I'll check the things on my side...

Yep! Redownloading the original file solved the problem.
User avatar
Kristina Campbell
 
Posts: 3512
Joined: Sun Oct 15, 2006 7:08 am

Post » Mon Mar 14, 2011 6:48 pm

That's amazing work!!! For game it is a lot of plugins, and a limit is only 255. Big thanks for your program.
And I have such question:
For example, I have a plugin which adds NPC with new hairs and faces. Can I merge plugin with Morrowind.esm so that values Meshes were replaced with the standard ones? (Body Part->Part-> Hair and Head)
Or it is better that a hairs and faces were replaced on new of other plugin (For example, Ren's Oblivion Hair for Morrowind...)
Is it possible?
User avatar
Margarita Diaz
 
Posts: 3511
Joined: Sun Aug 12, 2007 2:01 pm

Post » Tue Mar 15, 2011 4:09 am

Hi eeeickythump,

At first, many thanks for what you are doing, it would be fantastic to have an efficient merging objects tool with a merge plugins function.

I've just tried to merge a few plugins, two by two, and each time I first checked if there was any conflicts (using find-conflicts command, and there was not). The merging operation seemed to be successful each time, but then I got the followind errors:

Land (X, Y) unable to load texture idx N


The game cannot launch and crashes.

I reverted to an older save (a plugin merged using esper), without error, but I noticed that many land textures do not load correctly and disappeared (they were replaced by the default land texture). It is like the landscape coordinate were corrupted. The plugins I tried to merge with another was a landscape plugin (altering the land textures, similar to Texture fix mod) and adding trees, rocks, etc. with another small plugin, just adding a few item to the landscape.

Just reporting for further fix.
User avatar
Manny(BAKE)
 
Posts: 3407
Joined: Thu Oct 25, 2007 9:14 am

Post » Tue Mar 15, 2011 1:14 am

That's amazing work!!! For game it is a lot of plugins, and a limit is only 255. Big thanks for your program.
And I have such question:
For example, I have a plugin which adds NPC with new hairs and faces. Can I merge plugin with Morrowind.esm so that values Meshes were replaced with the standard ones? (Body Part->Part-> Hair and Head)
Or it is better that a hairs and faces were replaced on new of other plugin (For example, Ren's Oblivion Hair for Morrowind...)
Is it possible?

Do you mean overwrite the plugin's hairs with those from Morrowind.esm? I don't think so, masters (ESMs) are all loaded before plugins (ESPs) so your plugins would always override previously loaded records.

@Alexir: glad it solved the problem.

@Mordicus: not sure what is causing that, sounds like a problem with cell records.

Esper 15 is being uploaded. Now you can use wildcards on the command line - BUT if you do you need to surround the expression with quotes eg:

esper --merge-records "LGNPC*.esp"

Also, remember filenames must go LAST -- after all other options. Also '?' does not work, only '*'.

There is a new option, --zero-hp-fix, which attempts to solve the zero HP bug reported by a few people earlier. See the readme.
User avatar
Louise Dennis
 
Posts: 3489
Joined: Fri Mar 02, 2007 9:23 pm

Post » Mon Mar 14, 2011 10:02 pm

Do you mean overwrite the plugin's hairs with those from Morrowind.esm? I don't think so, masters (ESMs) are all loaded before plugins (ESPs) so your plugins would always override previously loaded records.

It does not matter. I can convert esm-> esp. And you can make such option:
so that when you merge replace meshes on other randomly? (restriction only by race and six)
Simply, there are a lot of plugins that use the old models of faces and hairstyles. A change them manually in the CS tiring.
It would be useful.
User avatar
brenden casey
 
Posts: 3400
Joined: Mon Sep 17, 2007 9:58 pm

Post » Mon Mar 14, 2011 10:10 pm

New esper64 version:

http://www.4shared.com/file/241100378/50aefed1/esper64_Beta15.html

eeeickythump: Thanks for the new version!


(It would be nice if you can place the download link at the first post)
User avatar
Kara Payne
 
Posts: 3415
Joined: Thu Oct 26, 2006 12:47 am

Post » Mon Mar 14, 2011 8:59 pm


Esper 15 is being uploaded. Now you can use wildcards on the command line - BUT if you do you need to surround the expression with quotes eg:

esper --merge-records "LGNPC*.esp"

Also, remember filenames must go LAST -- after all other options. Also '?' does not work, only '*'.

There is a new option, --zero-hp-fix, which attempts to solve the zero HP bug reported by a few people earlier. See the readme.
Either your upload speed's really bad or, ~9 hours later, something's gone wrong with the monolithic EXE download.

I haven't chipped in much here recently, been a bit tied up with mlox.
User avatar
SWagg KId
 
Posts: 3488
Joined: Sat Nov 17, 2007 8:26 am

Post » Mon Mar 14, 2011 10:06 pm

Either your upload speed's really bad or, ~9 hours later, something's gone wrong with the monolithic EXE download.

I haven't chipped in much here recently, been a bit tied up with mlox.


Bitbucket was playing up yesterday. The file is showing up in the downloads section now, let me know if it's corrupted etc.

I should have also mentioned: esper now has default names for the output file, according to what it is doing.
So for merging records it is called esper_merged_records.esp
User avatar
Kaley X
 
Posts: 3372
Joined: Wed Jul 05, 2006 5:46 pm

Post » Mon Mar 14, 2011 8:51 pm

A little report on esper14.
Looks like batch option "-b" doesn't work as expected, for me at least :)
I tried to merge records from all my plugins using the example from esper's readme "esper.exe -b pluginlist.txt -r" where pluginlist.txt is a plain text file generated by esper using the simple command "esper.exe -l >pluginlist.txt" (I've cleaned out the files paths and vanilla esm's of course).. nothing happened, just the usual help screen popped.
OK then.. let's take another try "esper.exe --batchfile pluginlist.txt -r". Wohoo it worked! :)

P.S. Wow! The resulting output.esp file exceeds my usual merged objects.esp three times in size. Looks like esper does it job very thoroughly :thumbsup:
User avatar
Lance Vannortwick
 
Posts: 3479
Joined: Thu Sep 27, 2007 5:30 pm

Post » Mon Mar 14, 2011 8:52 pm

Bitbucket was playing up yesterday. The file is showing up in the downloads section now, let me know if it's corrupted etc.
Grabbed! Download's fine. Thanks :)

I should have also mentioned: esper now has default names for the output file, according to what it is doing.
So for merging records it is called esper_merged_records.esp
Cool, thanks for adding that.
A little report on esper14.
Looks like batch option "-b" doesn't work as expected, for me at least :)
I tried to merge records from all my plugins using the example from esper's readme "esper.exe -b pluginlist.txt -r" where pluginlist.txt is a plain text file generated by esper using the simple command "esper.exe -l >pluginlist.txt" (I've cleaned out the files paths and vanilla esm's of course).. nothing happened, just the usual help screen popped.
OK then.. let's take another try "esper.exe --batchfile pluginlist.txt -r". Wohoo it worked! :)
To use a batchfile you need to use either "-B" or "--batchfile" a lowercase "b" does not work

P.S. Wow! The resulting output.esp file exceeds my usual merged objects.esp three times in size. Looks like esper does it job very thoroughly :thumbsup:
I think part of that is that TESTool will review the merged objects against the values in the last loaded plugin and discard those objects where the values in merged objects.esp and the last loaded plugin match. I don't believe esper does this.
One can reduce the size of the esper plugin somewhat by using TESPCD, this will compare objects in the plugin against the same objects in all the ESMs you have loaded. Any objects that're the same as in the ESMs can then be removed ("Auto-delete Unclean") by TESPCD.
User avatar
Benji
 
Posts: 3447
Joined: Tue May 15, 2007 11:58 pm

Post » Mon Mar 14, 2011 5:28 pm

To use a batchfile you need to use either "-B" or "--batchfile" a lowercase "b" does not work

The readme needs revision then :)

I think part of that is that TESTool will review the merged objects against the values in the last loaded plugin and discard those objects where the values in merged objects.esp and the last loaded plugin match. I don't believe esper does this.
One can reduce the size of the esper plugin somewhat by using TESPCD, this will compare objects in the plugin against the same objects in all the ESMs you have loaded. Any objects that're the same as in the ESMs can then be removed ("Auto-delete Unclean") by TESPCD.

You're right! TESPCD detected couple of hundreds of 'unclean' records, shrinking the resulting file to approximately 90 percent of original :)
User avatar
Jessica Nash
 
Posts: 3424
Joined: Tue Dec 19, 2006 10:18 pm

Post » Mon Mar 14, 2011 2:17 pm

The readme needs revision then :)
Actually, the readme's fine:
--batchfile , -B
is a text file containing one filename per line. Esper will use
all the files named in as if they were specified on the command line.
Although stating that "-b" will not work might be worth adding. Dunno :shrug:
User avatar
Dorian Cozens
 
Posts: 3398
Joined: Sat May 26, 2007 9:47 am

Post » Mon Mar 14, 2011 11:19 pm

Actually, the readme's fine:Although stating that "-b" will not work might be worth adding. Dunno :shrug:


Actually, the almost whole command line was just a copy-paste from "Examples of usage" section, as I mentioned earlier..

Examples of usage:
==================
....
Merge records from all the plugins named in "tomerge.txt" (one per line):

esper -b tomerge.txt -r
....


That's all. :shrug:
User avatar
Cassie Boyle
 
Posts: 3468
Joined: Sun Nov 05, 2006 9:33 am

PreviousNext

Return to III - Morrowind