[WIP] New merge objects program

Post » Mon Mar 14, 2011 3:25 pm

What command line are you using? Are you only merging 2 plugins each time?


I tried to merge only Robe Replacer and PTE.esp (esper -r IceBradyRobeReplacerALL.esp "Piratelords Trade Enhancements.esp", same thing with esper -r -a.
User avatar
Lizbeth Ruiz
 
Posts: 3358
Joined: Fri Aug 24, 2007 1:35 pm

Post » Mon Mar 14, 2011 5:41 pm


@Dragon32:
* The strange problem with SKIL records being merged is fixed.
Yep, that one is squashed

* The lone "T"s and "NIL"s were some debugging output, now gone.
Yep

Still seem to have the same issue that the original record from the relevant ESM (if applicable) isn't used as the Original record so I'm not seeing a true merge of the records. Some discussion about that, above. I still think to get a properly functioning merge program there needs to be a compare against the Beth ESMs (Tetchy's idea of cribbing the TESTool DAT file seems to be a good one, guess that'd speed things up...)
Could you perhaps post what you believe the merged record should contain? I still think it is behaving correctly. A merge where there are only 2 copies of the record is equivalent to just overwriting the older copy with the newer. You need 3 or more copies of the record for fine grained record merging to be useful.
I guess my assumption has always been that when merging plugins one would always want to merge records so that they take account of the records in the Bethesda ESMs, even if this is not explicitly stated.
But, if someone is just merging two plugins then I guess they don't want interference from the Bethesda ESMs (one can always include those if need be), so i guess it's OK... :unsure:

I thought I'd have a stab at seeing what Esper makes of the "NPCs falling down dead" bug that seems to plague TESTool. I looked through TESTool's Merged Objects and found this NPC with 0 health:
NPC_ new_shoes bragor    > Morrowind.esm        FNAM      New-Shoes Bragor        RNAM      Wood Elf        CNAM      Thief        ANAM      Thieves Guild        BNAM      b_n_wood elf_m_head_04        KNAM      b_n_wood elf_m_hair_03        SCRI      prisonerScript        NPDT      Level=9, Disp=50, Faction=4, Rank=1, Unknown(0,0,0), Gold=0        FLAG      00000018        NPCO      1, common_belt_02        NPCO      1, common_shirt_05        NPCO      1, exquisite_shoes_01        NPCO      1, common_pants_03        AIDT      Hello=30, Fight=30, Flee=30, Alarm=0, Flags=00000000        AI_W      Distance=0, Duration=, Hour=5, Idle(0,60,20,10,10,0,10,0)    > Changing Faces Replacer.esm        FNAM      New-Shoes Bragor        RNAM      Wood Elf        CNAM      Thief        ANAM      Thieves Guild        BNAM      b_n_wood elf_m_head_28        KNAM      b_n_wood elf_m_hair_07        SCRI      prisonerScript        NPDT      Level=9, Disp=50, Faction=4, Rank=1, Unknown(243,88,124), Gold=0        FLAG      00000018        NPCO      1, common_belt_02        NPCO      1, common_shirt_05        NPCO      1, exquisite_shoes_01        NPCO      1, common_pants_03        AIDT      Hello=30, Fight=30, Flee=30, Alarm=0, Flags=00000000        AI_W      Distance=0, Duration=, Hour=5, Idle(0,60,20,10,10,0,10,0)    > LGNPC_Pelagiad_v1_22.esp        FNAM      New-Shoes Bragor        RNAM      Wood Elf        CNAM      lgnpc_Bragor_class        ANAM      Thieves Guild        BNAM      b_n_wood elf_m_head_04        KNAM      b_n_wood elf_m_hair_03        SCRI      max_lgnpc_pela_bragor_sc_01        NPDT      Level=9, Disp=50, Faction=4, Rank=1, Unknown(0,0,0), Gold=200        FLAG      00000018        NPCO      1, common_belt_02        NPCO      1, common_shirt_05        NPCO      1, exquisite_shoes_01        NPCO      -10, common_shoes_03        NPCO      -3, expensive_shoes_01        NPCO      -5, expensive_shoes_02        NPCO      -7, expensive_shoes_03        NPCO      1, lgnpc_horizon_shoes        NPCO      1, lgnpc_holy_sandals        NPCO      1, common_pants_03        NPCO      -3, lgnpc_sc_boundboots        AIDT      Hello=30, Fight=30, Flee=30, Alarm=0, Flags=00001066        AI_W      Distance=0, Duration=, Hour=5, Idle(0,60,20,10,10,0,10,0)----------------------------        FNAM      New-Shoes Bragor        RNAM      Wood Elf        CNAM      lgnpc_Bragor_class        ANAM      Thieves Guild        BNAM      b_n_wood elf_m_head_28        KNAM      b_n_wood elf_m_hair_07        SCRI      max_lgnpc_pela_bragor_sc_01        NPDT      Level=9, Disp=50, Faction=4, Rank=1, Unknown(243,88,124), Gold=200        FLAG      00000018        NPCO      1, common_belt_02        NPCO      1, common_shirt_05        NPCO      1, exquisite_shoes_01        NPCO      1, common_pants_03        AIDT      Hello=30, Fight=30, Flee=30, Alarm=0, Flags=00001066        AI_W      Distance=0, Duration=, Hour=5, Idle(0,60,20,10,10,0,10,0)        NPCO      -10, common_shoes_03        NPCO      -3, expensive_shoes_01        NPCO      -5, expensive_shoes_02        NPCO      -7, expensive_shoes_03        NPCO      1, lgnpc_horizon_shoes        NPCO      1, lgnpc_holy_sandals        NPCO      -3, lgnpc_sc_boundboots
So trying with Esper I ended up with the same problem, an NPC with 0 health. Command used:
esper --debug -r "LGNPC_Pelagiad_v1_22.esp" "Changing Faces Replacer.esm" morrowind.esm >output.txt
Merged records:Original:#   #   #   #   #   #   #   #   #   #   #   #   #   #   #   #>New:#   #   #   #   #   #   #   #   #   #   #   #   #   #   #   #>Merged:#   #   #   #   #   #   #   #   #   #   #   #   #   #   #   #>[...]Merged records:Original:#   #   #   #   #   #   #   #   #   #   #   #   #   #   #   #>New:#   #   #   #   #   #   #   #   #   #   #   #   #   #   #   #   #   #   #   #   #   #   #>Merged:#   #   #   #   #   #   #   #   #   #   #   #   #   #   #   #>
Which makes me none the wiser :(

I've just tried a big merge of all my plugins, received this error:
In file #P"I:/Bethesda Softworks/Morrowind/Data Files/newarrows.esp", record type "CREA", last subrecord "NPCS":
SIMPLE-ERROR: Illegal record name "_W♫ " at position 325405 in #P"I:/Bethesda Softworks/Morrowind/Data Files/newarrows.esp"
Exiting...
Sounds like something wrong with my copy of the Marksman mod (although never had any issues in game with it), I did change it so that Cliff Racers didn't shoot fireballs...

No idea what I did to break these plugins:
In file #P"I:/Bethesda Softworks/Morrowind/Data Files/_FIENDS(Creatures Only).esp", record type "CREA", last subrecord "NPCS":
SIMPLE-ERROR: Illegal record name "hiel" at position 9615 in #P"I:/Bethesda Softworks/Morrowind/Data Files/_FIENDS(Creatures Only).esp"
Exiting...
In file #P"I:/Bethesda Softworks/Morrowind/Data Files/TR_Dremora.esp", record type "CREA", last subrecord "NPCS":
SIMPLE-ERROR: Illegal record name "0 NP" at position 984 in #P"I:/Bethesda Softworks/Morrowind/Data Files/TR_Dremora.esp"
Exiting...
In file #P"I:/Bethesda Softworks/Morrowind/Data Files/Atronach Expansion X.ESP" record type "CREA", last subrecord "NPCS":
SIMPLE-ERROR: Illegal record name "une " at position 24582 in #P"I:/Bethesda Sotworks/Morrowind/Data Files/Atronach Expansion X.ESP"
Exiting...
Nice that Esper's picking these up for me :)

Would it be possible for an option to have Esper check my plugins like this? Might be quicker without the merging being needed. I guess these are caused by low-level editors. Anyway, these take a while.

When I finish my big merge I'll have a shufti through and see if I spot anything weird. :)

[Edit: Yay, finished. Created a 5.67MB Output.esp :o Thought I'd run it through TESPCD, quite a few duplicate entries. Removing those in TESPCD leads to a 5.49MB file. Just thought I'd mention it.]
User avatar
Nicole Coucopoulos
 
Posts: 3484
Joined: Fri Feb 23, 2007 4:09 am

Post » Tue Mar 15, 2011 3:28 am

I converted IceBradyRobeReplacerPLUS and PTE to the handy estxt format to show you the actual problem. The expensive_robe_01 is just an example. It happens with all records, regardless of NPC, Armor, Weapon, Clothing or anything else.

IceBradyRobeReplacerPLUS.esp:(CLOT 218 (CLOT NAME 18 (ESPER::STRING-VALUE "expensive_robe_01")) (CLOT MODL 31 (ESPER::STRING-VALUE "c\\C_M_Robe_expensive_1_GND.NIF")) (CLOT FNAM 15 (ESPER::STRING-VALUE "Expensive Robe")) (CLOT CTDT 12 (ESPER::CLOTHING-TYPE 4) (ESPER::CLOTHING-WEIGHT 3.0)  (ESPER::CLOTHING-VALUE 10) (ESPER::CLOTHING-ENCHANT-POINTS 50)) (CLOT ITEX 26 (ESPER::STRING-VALUE "c\\Tx_robe_expensive_1.tga")) (CLOT INDX 1 (ESPER::ENUM-VALUE :CUIRASS)) (CLOT BNAM 20 (ESPER::STRING-VALUE "c_m_robe_expensive_1")) (CLOT CNAM 13 (ESPER::STRING-VALUE "Ice_RobeExp01")) (CLOT INDX 1 (ESPER::ENUM-VALUE :LEFT-ANKLE)) (CLOT INDX 1 (ESPER::ENUM-VALUE :RIGHT-ANKLE)))   Piratelords Trade Enhancements.esp  (CLOT 197 (CLOT NAME 18 (ESPER::STRING-VALUE "expensive_robe_01")) (CLOT MODL 31 (ESPER::STRING-VALUE "c\\C_M_Robe_expensive_1_GND.NIF")) (CLOT FNAM 15 (ESPER::STRING-VALUE "Expensive Robe")) (CLOT CTDT 12 (ESPER::CLOTHING-TYPE 4) (ESPER::CLOTHING-WEIGHT 3.0)  (ESPER::CLOTHING-VALUE 53) (ESPER::CLOTHING-ENCHANT-POINTS 75)) (CLOT ITEX 26 (ESPER::STRING-VALUE "c\\Tx_robe_expensive_1.tga")) (CLOT INDX 1 (ESPER::ENUM-VALUE :CUIRASS)) (CLOT BNAM 20 (ESPER::STRING-VALUE "c_m_robe_expensive_1")) (CLOT INDX 1 (ESPER::ENUM-VALUE :LEFT-ANKLE)) (CLOT INDX 1 (ESPER::ENUM-VALUE :RIGHT-ANKLE)))  merged esp (esper -o merged.estxt -r IceBradyRobeReplacerPLUS.esp "Piratelords Trade Enhancements.esp"):  (CLOT 197 (CLOT NAME 18 (ESPER::STRING-VALUE "expensive_robe_01")) (CLOT MODL 31 (ESPER::STRING-VALUE "c\\C_M_Robe_expensive_1_GND.NIF")) (CLOT FNAM 15 (ESPER::STRING-VALUE "Expensive Robe")) (CLOT CTDT 12 (ESPER::CLOTHING-TYPE 4) (ESPER::CLOTHING-WEIGHT 3.0)  (ESPER::CLOTHING-VALUE 53) (ESPER::CLOTHING-ENCHANT-POINTS 75)) (CLOT ITEX 26 (ESPER::STRING-VALUE "c\\Tx_robe_expensive_1.tga")) (CLOT INDX 1 (ESPER::ENUM-VALUE :CUIRASS)) (CLOT BNAM 20 (ESPER::STRING-VALUE "c_m_robe_expensive_1")) (CLOT INDX 1 (ESPER::ENUM-VALUE :LEFT-ANKLE)) (CLOT INDX 1 (ESPER::ENUM-VALUE :RIGHT-ANKLE)))


As you can see, the merged esp contains all records made by PTE, but not the records of the Robe Replacer. Especially this record is missing:

(CLOT CNAM 13 (ESPER::STRING-VALUE "Ice_RobeExp01"))

which renders this mod useless because it's adding a female version of all robes.
User avatar
BRIANNA
 
Posts: 3438
Joined: Thu Jan 11, 2007 7:51 pm

Post » Mon Mar 14, 2011 11:24 pm

I guess my assumption has always been that when merging plugins one would always want to merge records so that they take account of the records in the Bethesda ESMs, even if this is not explicitly stated.
But, if someone is just merging two plugins then I guess they don't want interference from the Bethesda ESMs (one can always include those if need be), so i guess it's OK...


I made esper only process the plugins/masters that you tell it to process, in order to (1) allow precise control over merging and (2) avoid the slow and tedious processing of the three big ESMs every time you do a merge.
So yes, you do need to tell it that you want (eg) Morrowind.esm to be processed as well.

I thought I'd have a stab at seeing what Esper makes of the "NPCs falling down dead" bug that seems to plague TESTool.


I duplicated what you did. I think I see the problem, and I think it's a problem with how Morrowind encodes NPCs, rather than with esper or TEStool.

NPC_ records contain a subrecord called NPDT which can exist in either a SHORT form (12 bytes) or a LONG form (52 bytes). The short form only contains disposition, faction info, and gold. The long version also contains stats like health, mana, strength, etc. So an NPC_ record with the short version of NPDT does not contain any information about how many hit points that NPC has. "New shoes bragor" in the plugins you used, only contains the short form of NPDT.

I have no idea how MW decides how many hit points to give these NPCs. I wonder if they "inherit" stats somehow and this process gets broken during merging, and MW then decides the NPC must have zero HP. A "hack fix" would be to give all such NPCs long-form NPDT subrecords with eg 30 hit points, 30 strength, etc, as I imagine most of them are probably intended to be "boring" shopkeepers or townspeople.

If anyone can tell me where MW gets hit point data etc from for these NPCs I would be grateful.


I've just tried a big merge of all my plugins, received this error:
(snip)


Not sure what's going on there... I tried esper on Bethesda's newarrows,esp and it processed fine. Have you modified those plugins somehow?
Maybe someone else could confirm that esper reads newarrows.esp OK on their computer?

[Edit: Yay, finished. Created a 5.67MB Output.esp :o Thought I'd run it through TESPCD, quite a few duplicate entries. Removing those in TESPCD leads to a 5.49MB file. Just thought I'd mention it.]


I'm glad it behaved itself with the rest of your plugins. Thanks again for your ongoing interest. The bugs are slowly being hunted down and stomped on...
User avatar
Rob
 
Posts: 3448
Joined: Fri Jul 13, 2007 12:26 am

Post » Tue Mar 15, 2011 1:50 am

I converted IceBradyRobeReplacerPLUS and PTE to the handy estxt format to show you the actual problem. The expensive_robe_01 is just an example. It happens with all records, regardless of NPC, Armor, Weapon, Clothing or anything else.

(snip)

As you can see, the merged esp contains all records made by PTE, but not the records of the Robe Replacer. Especially this record is missing:

(CLOT CNAM 13 (ESPER::STRING-VALUE "Ice_RobeExp01"))

which renders this mod useless because it's adding a female version of all robes.


Thanks for that. I believe I have fixed the problem. Basically, merging armour and clothing records is quite complicated because certain of their subrecords occur in "clusters" (INDX, BNAM + CNAM). When merging, each of these clusters needs to be compared against matching clusters in the plugins that have already been processed ("match" means the INDX subrecords are identical). Its like subrecords within subrecords. What esper WAS doing was removing all the INDX, CNAM and BNAM subrecords en bloc from the "old" record and replacing them with the subrecords from the new record.

I think its behaviour is now correct. I tested it on the plugins you used, and it worked.

Esper beta 10 incorporates this fix and has been uploaded to the usual place.

Again, thanks to yourself and all the other brave testers.
(has anyone other than me tried using esper's output in a game?)
User avatar
Jack
 
Posts: 3483
Joined: Sat Oct 20, 2007 8:08 am

Post » Tue Mar 15, 2011 6:03 am

Thanks! Is it fixed for the NPCs as well?
User avatar
Roy Harris
 
Posts: 3463
Joined: Tue Sep 11, 2007 8:58 pm

Post » Mon Mar 14, 2011 11:02 pm

I am using this program and I have a problem with it. I noticed that the Fay ability is being changed by esper when I use it to make a merged objects.esp with all of my mods. The problem is that the only mod I have that changes it is btb character.esp which makes it give 20 intelligence and 1.0x max magicka and when I use merged objects.esp it removes the 1.0x max magicka. I have already checked and that is the only mod that changes that ability and once I removed the change from merged objects.esp the ability returned to the changes made by btb charecter.esp.
User avatar
Juliet
 
Posts: 3440
Joined: Fri Jun 23, 2006 12:49 pm

Post » Mon Mar 14, 2011 4:46 pm


Not sure what's going on there... I tried esper on Bethesda's newarrows,esp and it processed fine. Have you modified those plugins somehow?
Maybe someone else could confirm that esper reads newarrows.esp OK on their computer?
It's probably my mistake. I'm sure I've edited most of those plugins giving errors (so I've probably edited all of them) and whilst it's a while ago I bet I used Enchanted Editor and messed something up. Anyway, after opening and saving the offending plugins in the TES:CS (then cleaning of any bogus crap that'd introduced) meant that the plugins were processed fine by Esper. So, no idea what the problem was but I have a solution.

[Edit:

Esper beta 10 incorporates this fix and has been uploaded to the usual place.
Hmmm, there might be a problem with this version. I create a new (full plugin merge records) output.esp and neither Enchanted Editor or MWEdit will open it, both give Out of Memory errors. TESAME will, however, open it but if I try and edit anything I get Access Violation (memory-related) errors. TESTool crashes on attempting to clean it. Loading output.esp into the TES:CS and resaving, without changing anything, means the file shrinks to ~1.5MB (from ~5.5MB)]
User avatar
Minako
 
Posts: 3379
Joined: Sun Mar 18, 2007 9:50 pm

Post » Mon Mar 14, 2011 2:53 pm

[Edit:Hmmm, there might be a problem with this version. I create a new (full plugin merge records) output.esp and neither Enchanted Editor or MWEdit will open it, both give Out of Memory errors. TESAME will, however, open it but if I try and edit anything I get Access Violation (memory-related) errors. TESTool crashes on attempting to clean it. Loading output.esp into the TES:CS and resaving, without changing anything, means the file shrinks to ~1.5MB (from ~5.5MB)]


Oops. I forgot to recalculate the size of records after the new type of merge, so CLOT and ARMO records were expecting to be a different size from their actual size. Fixed.

@FallenWizard: re fixing NPCs, do you mean the "0 HP" problem? No I haven't done anything about that as it's unclear whether my idea is the "right" one.

@quenthar: I think I have fixed this. It was losing ENCH subrecords from birthsigns when they contained more than one. See if it works for you now.

Beta 11 is up. It contains the above fixes and the requested feature "caching". If you do:
esper --build-cache

It will scan the three big Bethesda ESM files and create a file called esper.cache in the Data Files directory, which contains offset information for all the records in each file. This process takes a while (and doesn't produce much screen output -- don't panic). Once it is done, you shouldn't have to do it again. The cache is only used when merging RECORDS. Tell esper to use it by giving it the --use-cache option, e.g:
esper --use-cache -r plugin1.esp plugin2.esp -o out.esp


I can't really tell how much faster it is myself, interested in impressions.
User avatar
Hella Beast
 
Posts: 3434
Joined: Mon Jul 16, 2007 2:50 am

Post » Tue Mar 15, 2011 5:23 am

No, the scripts problem with NPCs. It's fixed already and is working.

By the way: Thanks for the new version! Caching sounds very good.

EDIT:

Caching doesn't work. --build-cache throws an error:

D:\Morrowind\Data Files>esper --build-cache
Esper, version Beta 11
Released under GPL; see the file LICENSE for details.

Command line arguments received:
("--build-cache")

Caching D:/Morrowind/Data Files/Morrowind.esm...
Caching D:/Morrowind/Data Files/Tribunal.esm...
Caching D:/Morrowind/Data Files/Bloodmoon.esm...
STORE-ERROR: No applicable primary methods for #ORE:BACKEND-STORE #x8DD3C16>
Applicable methods: (#RESOLVING-BACKEND T T)>)
Exiting...
User avatar
City Swagga
 
Posts: 3498
Joined: Sat May 12, 2007 1:04 am

Post » Mon Mar 14, 2011 1:54 pm

I get that too, FallenWizard:
esper --build-cache
Esper, version Beta 11
Released under GPL; see the file LICENSE for details.

Command line arguments received:
("--build-cache")

Caching I:/Bethesda Softworks/Morrowind/Data Files/Morrowind.esm...
Caching I:/Bethesda Softworks/Morrowind/Data Files/Tribunal.esm...
Caching I:/Bethesda Softworks/Morrowind/Data Files/Bloodmoon.esm...
STORE-ERROR: No applicable primary methods for #
Applicable methods: (#)
Exiting...

Figured out a way to test if the cache actually speeded things up too :( Admittedly a bit of a bodge...

A while back there was some problem with specifying plugins to operate on via a TXT file, tested this again (both Windows and Unix style line endings) and it works in esper11, as long as the -B switch is used (as per documentation), -b doesn't work. --batchfile is fine but --Batchfile isn't

As to the NPDT SHORT and LONG data, from the Morrowind Mod Maker's Manual:
Auto-Calculate Stats. Check this and the CS will calculate all stats, skills, and spells based upon race, class, and level. They will be grayed out. Uncheck to modify them manually again.
I haven't been able to track down the mechanics of that calculation. http://www.uesp.net/wiki/Morrowind:NPCs#Characteristics is incomplete and I couldn't find anything by searching here.

One thing that might happen from my http://www.gamesas.com/bgsforums/index.php?s=&showtopic=963061&view=findpost&p=13912003 is that someone turns off Auto-Calculate Stats for an NPC so they can add a spell to their inventory. So, an NPC's NPDT subrecord could be a SHORT for some and a LONG for this hypothetical mod.

Another thought is that in my example, above (New-shoes Bragor), he's given a custom class by the LGNPC mod. Could the fact that the custom class isn't defined in output.esp have an effect...? :unsure:
User avatar
Lifee Mccaslin
 
Posts: 3369
Joined: Fri Jun 01, 2007 1:03 am

Post » Mon Mar 14, 2011 1:45 pm

I have another thing to report: the output of --find-conflicts ("Files with conflicts") should sort the plugins like they are in the load order.

For example:

load order:

Morrowind.esm
Tribunal.esm
Bloodmoon.esm
Plugin 1.esp
Plugin 2.esp
Plugin 3.esp
Plugin 4.esp


--find-conflicts reports it like this:

Plugin 2.esp
Plugin 4.esp
Plugin 1.esp
Plugin 3.esp
User avatar
Robert
 
Posts: 3394
Joined: Sun Sep 02, 2007 5:58 am

Post » Mon Mar 14, 2011 3:55 pm

I get that too, FallenWizard:

Yup, caching wasn't quite ready for prime time... I believe I have fixed it now.

As to the NPDT SHORT and LONG data, from the Morrowind Mod Maker's Manual:I haven't been able to track down the mechanics of that calculation. http://www.uesp.net/wiki/Morrowind:NPCs#Characteristics is incomplete and I couldn't find anything by searching here.

One thing that might happen from my http://www.gamesas.com/bgsforums/index.php?s=&showtopic=963061&view=findpost&p=13912003 is that someone turns off Auto-Calculate Stats for an NPC so they can add a spell to their inventory. So, an NPC's NPDT subrecord could be a SHORT for some and a LONG for this hypothetical mod.


Hmmm... NPCs do have an "Autocalc" flag (in the FLAG subrecord). Anyone know what it does?

@FallenWizard: conflicted files should now be sorted by load order.

Esper beta 12 is released. Changes:
- caching is fixed.
- in output from find-conflicts, files are sorted according to load order.
- {{ESPER:NOMERGE}} is now obeyed by merge-plugins as well as merge-records.
- A few other minor fixes.
User avatar
Loane
 
Posts: 3411
Joined: Wed Apr 04, 2007 6:35 am

Post » Mon Mar 14, 2011 3:42 pm

The cache works well and I built it without problems and creating my merged-objects.esp only took 17 minutes. I'm not sure how much shorter that was since I always went and did something else while making it before but I know it was well shorter then it was since it took awhile to do the Morrowind.esm before and it didn't need to do that this time.

One good option to do with having a cache that would be good to have is being able to add any esm or esp to that cache. That way you could put mods that are permanently in your load order in your cache to decrease the time it takes to make your merged-objects.esp.
User avatar
Aaron Clark
 
Posts: 3439
Joined: Fri Oct 26, 2007 2:23 pm

Post » Mon Mar 14, 2011 4:46 pm

Thanks for new version, but there are still missing features:

- Some plugins are patches, and are meant to overwrite the objects of their parent files completely. It might be useful to give the merging program an "exception list" of mods which are meant to be used as pure replacers (might also be overly complicated though).


Would be nice for patch mods (for example the Better Clothes patch)

and this one:

Anyway, plugin masters seem to be set right here (Left Gloves is dependent on all three Bethesda ESMs), all are 1234 bytes still.



and another one:

Esper should name the "Merged records" esp different so mlox could pick it up. What about Esper_Merged_Records.esp?


By the way:
Caching works very well now. Good job.
User avatar
Vickytoria Vasquez
 
Posts: 3456
Joined: Thu Aug 31, 2006 7:06 pm

Post » Mon Mar 14, 2011 2:24 pm

Just some info on caching.

Merging all plugin records
Without caching: 10'04" and 10'01"
With caching: 06'49" and 06'50"

Creating the cache took 05'50"

Tried to merge records after deleting the cache file:
esper -r -a --use-cacheEsper, version Beta 12Released under GPL; see the file LICENSE for details.Command line arguments received:	("-r" "-a" "--use-cache")TYPE-ERROR: value NIL is not of the expected type STREAM.Exiting...
Might be worth capturing that error...


Hmmm... NPCs do have an "Autocalc" flag (in the FLAG subrecord). Anyone know what it does?
Well I assume that that's what determines whether stats are deduced (race, class and level) or defined. I.e. Short vs Long, above.

Although I'm not the best one for such low-level information :)
User avatar
Irmacuba
 
Posts: 3531
Joined: Sat Mar 31, 2007 2:54 am

Post » Tue Mar 15, 2011 4:56 am

Another thing:

I installed Windows 7 (64 bit version) and esper doesn't work with it. I just get a message that the program can't run on my OS version.

Since Clozure is also available for 64 Bit isn't it possible to release a 64 bit version?
User avatar
Channing
 
Posts: 3393
Joined: Thu Nov 30, 2006 4:05 pm

Post » Mon Mar 14, 2011 10:20 pm

Glad to see that caching seems to be working, and that it does actually speed things up going by peoples' timing data.

I should note that the versions I've been releasing are all compiled for maximum debuggability, not speed. Maybe I'll upload a speed-optimised version for the next release.

TYPE-ERROR: value NIL is not of the expected type STREAM.
Exiting...[/code]Might be worth capturing that error...

:splat: got it.

Well I assume that that's what determines whether stats are deduced (race, class and level) or defined. I.e. Short vs Long, above.
Although I'm not the best one for such low-level information :)


Well, I could put in a --force-npc-autocalc option and you could report back.

@FallenWizard

- Some plugins are patches, and are meant to overwrite the objects of their parent files completely. It might be useful to give the merging program an "exception list" of mods which are meant to be used as pure replacers (might also be overly complicated though).


Done in the next version. Esper will understand "{{ESPER:PATCH}}" in the plugin description. When merging records, that plugin will overwrite all older records.

Also the default output name will now be esper_output.esp

Finally:

I installed Windows 7 (64 bit version) and esper doesn't work with it. I just get a message that the program can't run on my OS version.
Since Clozure is also available for 64 Bit isn't it possible to release a 64 bit version?


Only if I had a 64 bit operating system -- which I don't.
Strange -- 64 bit OSes usually run 32-bit apps OK. Try some other command line programs (eg download a version of "grep").

Anyway: no, you're basically out of luck, unless... the only thing I can think of is a CLISP image. CLISP is a very portable version of common lisp that uses a byte compiler. I will look into making a CLISP image (should be portable across multiple platforms).
User avatar
Alan Whiston
 
Posts: 3358
Joined: Sun May 06, 2007 4:07 pm

Post » Mon Mar 14, 2011 2:56 pm

Since I redid my merged-objects.esp some npcs are missing their torso. I can see through it and if I uncheck the merged-objects.esp the npcs torso comes back.

EDIT: I tried starting a new game to see if that was the problem but that didn't help but I did find out something new. It is not just the torso that can be effected by this a Jiub was missing his groin and the guard out on the deck of the ship you arrive in had part of his right pauldron missing.

PS. The problem with the Fey power is gone and it shows up correctly now.
User avatar
matt white
 
Posts: 3444
Joined: Fri Jul 27, 2007 2:43 pm

Post » Mon Mar 14, 2011 6:41 pm


Well, I could put in a --force-npc-autocalc option and you could report back.
So force all NPCs to be auto-calc?

Also, just wanted to re-mention the scanning of BSAs when checking resources, still doesn't seem to be happening.
User avatar
Tom
 
Posts: 3463
Joined: Sun Aug 05, 2007 7:39 pm

Post » Mon Mar 14, 2011 8:35 pm

Only if I had a 64 bit operating system -- which I don't.
Strange -- 64 bit OSes usually run 32-bit apps OK. Try some other command line programs (eg download a version of "grep").

Anyway: no, you're basically out of luck, unless... the only thing I can think of is a CLISP image. CLISP is a very portable version of common lisp that uses a byte compiler. I will look into making a CLISP image (should be portable across multiple platforms).


Well, I tried to use Clozure myself and asdf-install can't install any packages required by esper :/

The only thing I get is:

; Warning: Cannot find tar command NIL.


I installed Cygwin and wanted to point *gnu-tar-program* to C:\cygwin\bin\tar.exe.

But everything I try, it doesn't work.
User avatar
Lavender Brown
 
Posts: 3448
Joined: Tue Jul 25, 2006 9:37 am

Post » Mon Mar 14, 2011 3:57 pm

So force all NPCs to be auto-calc?

Also, just wanted to re-mention the scanning of BSAs when checking resources, still doesn't seem to be happening.


No, I meant force the NPCs with short data to be autocalc'd.

@quenthar: could you run 'esper -p merged-records.esp -o merged.estxt' and post the record for Jiub?

@FallenWizard:
The libraries are all very widely used. It looks like asdf-install is expecting to be able to call 'tar' (an archive extraction program) to extract an archive it's downloaded.

You could try downloading a set of "unix tools" and trying again
Alternatively, I suggest just downloading them individually, and extracting each to its own directory.
You will need ITERATE, IEEE-FLOATS, CL-FAD, CLOSER-MOP, FLEXI-STREAMS, CL-PPCRE, CL-STORE, and COMMAND-LINE-ARGUMENTS.
Search for each on http://cliki.net
Also download ASDF (asdf.lisp -- you may have it already).
Then create a file called init.lisp somewhere, containing:

(load "c:/path/to/asdf.lisp")(defvar *libraries*  (list "c:/directory/of/esper/"		"c:/directory/of/iterate/"		"c:/directory/of/cl-ppcre/"		"c:/directory/of/ieee-floats/"	  ;; ...etc.... one string for every library dir   ))(dolist (lib *libraries*)   (push lib asdf:*central-registry*))


The above is not very clever but works.
Run clozure with the command line option -l /path/to/init.lisp
You may find it complains about "can't find package XXX" where it's not a package in the above list. If so, look again on cliki.net and install as above.
If clozure loads and you find yourself at a lisp command prompt, type
(asdf:oos 'asdf:load-op :esper).....(in-package :esper)

Then type
(ccl:current-directory)  ; to show current dir(setf (ccl:current-directory) "new/directory")	; if you want to change dir(main "--all --merge-records" ); example of running esper from lisp prompt


I don't think I'll be successful at getting a clisp image to work -- problem with streams. So the above is your best chance on your new OS unfortunately.

EDIT: PS sorry, I see you already know about tar.
User avatar
Jose ordaz
 
Posts: 3552
Joined: Mon Aug 27, 2007 10:14 pm

Post » Tue Mar 15, 2011 1:05 am

Esper v13 is up.
- Understands {{ESPER:PATCH}} in plugin description -- makes plugin overwrite older records when merging records.
- Default output file is now esper_output.esp
- Compiled with full optimisation. Interested to hear speed comparisons.
User avatar
Klaire
 
Posts: 3405
Joined: Wed Sep 27, 2006 7:56 am

Post » Tue Mar 15, 2011 2:37 am

The command you gave me didn't work so I rebuilt my merged list and below is what I find for Jiub. I even tried with the new version you just released and it still does it.

(NPC_ 304 (NPC_ NAME 13 (ESPER::STRING-VALUE "chargen name")) (NPC_ FNAM 5 (ESPER::STRING-VALUE "Jiub")) (NPC_ RNAM 9 (ESPER::STRING-VALUE "Dark Elf")) (NPC_ CNAM 6 (ESPER::STRING-VALUE "Thief")) (NPC_ ANAM 1 (ESPER::STRING-VALUE "")) (NPC_ BNAM 18 (ESPER::STRING-VALUE "1EM_b_n_DE_head05")) (NPC_ KNAM 15 (ESPER::STRING-VALUE "DS_DE_m_hair02")) (NPC_ SCRI 15 (ESPER::STRING-VALUE "CharGenNameNPC")) (NPC_ NPDT 52 (ESPER::NPC-LEVEL 3) (ESPER::NPC-STRENGTH 55)  (ESPER::NPC-INTELLIGENCE 35) (ESPER::NPC-WILLPOWER 42)  (ESPER::NPC-AGILITY 48) (ESPER::NPC-SPEED 47) (ESPER::NPC-ENDURANCE 42)  (ESPER::NPC-PERSONALITY 54) (ESPER::NPC-LUCK 40)  (ESPER::NPC-SKILLS (5 5 10 10 10 20 10 5 23 5 5 5 5 5 5 5 5 5 34 34 34 34 40                      20 20 20 20))  (ESPER::NPC-UNKNOWN1 0) (ESPER::NPC-HEALTH 60) (ESPER::NPC-SPELL-POINTS 17)  (ESPER::NPC-FATIGUE 187) (ESPER::NPC-DISPOSITION 50)  (ESPER::NPC-REPUTATION 0) (ESPER::NPC-FACTION-RANK 0) (ESPER::NPC-UNKNOWN2 0)  (ESPER::NPC-GOLD 0)) (NPC_ FLAG 4 (ESPER::NPC-FLAGS (:NONE?))) (NPC_ NPCO 36 (ESPER::ITEM-COUNT 1)  (ESPER::ITEM-NAME "common_pants_01

User avatar
Mrs. Patton
 
Posts: 3418
Joined: Fri Jan 26, 2007 8:00 am

Post » Tue Mar 15, 2011 12:14 am

Since I redid my merged-objects.esp some npcs are missing their torso. I can see through it and if I uncheck the merged-objects.esp the npcs torso comes back.

EDIT: I tried starting a new game to see if that was the problem but that didn't help but I did find out something new. It is not just the torso that can be effected by this a Jiub was missing his groin and the guard out on the deck of the ship you arrive in had part of his right pauldron missing.
I've had problems with missing torsos etc. when I've messed up with clothing and forgot about bodyparts. Jiub wears "common_pants_01" and checking out my current (made with v13) output.esp there are a whole bunch of male and female body parts assigned to it. Forgive the format, this is a quick dump of output.esp from Enchanted Editor:
	NAME	common_pants_01 	MODL	c\C_M_Pants_GND_common00.nif 	FNAM	Common Pants 	CTDT	       @  	ITEX	c\TX_pantscom00.tga 	INDX		BNAM	BC_common_pants_01_Ml 	CNAM	BC_common_pants_01_Fl 	INDX		BNAM	c_m_pants_a_common00 	CNAM	c_f_pants_a_common00 	INDX		BNAM	c_m_pants_ul_common00 	CNAM	c_f_pants_ul_common00 	INDX		BNAM	c_m_pants_ul_common00 	CNAM	c_f_pants_ul_common00 	INDX		BNAM	c_m_pants_k_common00 	CNAM	c_f_pants_k_common00 	INDX		BNAM	c_m_pants_k_common00 	CNAM	c_f_pants_k_common00 	INDX		BNAM	c_m_pants_a_common00 	CNAM	c_f_pants_a_common00
At a guess, I have multiple plugins each assigning a different male and female bodypart to these common pants. This also looks like it's affecting ARMO too.

Assuming I'm right (and I haven't checked this against the full text output to check which plugins are modifying this object) these multiple BNAMs (male body part) and CNAMs (female body part) are what is causing the problem here. I'd suggest that for CLOT and ARMO a "last loaded mod wins" approach is taken for which BNAM or CNAM takes precedent. Assuming that (when including the Bethesda ESMs) the BNAM or CNAM isn't the default, Bethesda value.
No, I meant force the NPCs with short data to be autocalc'd.
Heh. Of course, good plan. I'll check out the new version.

Oh, and as to the plugin name the new, default output file being called "esper_output.esp" is fine. What I was getting at was that if we could ensure that when someone does a "esper -a -r" for merged objects then that name is common to as many as possible, of course people can just rename it. By having a common name for this output, which I think will be the main use of esper, then I can amend the mlox rules to match this default name.

As most people who will use this are on Windows and most Windows users run a mile when they see the command line I was thinking, when this gets out of beta, I'd knock up a MS-DOS batch file for this.

So, esper.exe and "runme.bat" would go into Data Files, the batch file would check it's in the same place as Morrowind.esm and then check for the presence of esper.exe. If neither of those were there then it'd show an appropriate error. Next it'd check for the esper cache file and create it if necessary. Finally run esper -a -r Then it'd play a little tune

Actually, maybe not the last bit.

Oh, and btw what's the reason behind Linux users' antipathy to DOS text (CR + LF)? I know Unix-format is linefeed only but I'm intrigued... Both you and John.Moonsugar seem to suffer from it :)


[Edit: Timing info for v13-
esper -a -r without cache:
08'51"
08'46"

output file is still "output.esp"

I can't test with a cache as I get this error when using --build-cache
[...]Read "DIAL": "DIAL-rat"Size expected 446, read 446Read "INFO": "INFO-15683254982582712181"Size expected 435, read 435Read "INFO": "INFO-2353312470281431780"TYPE-ERROR: value #P"I:/Bethesda Softworks/Morrowind/Data Files/Morrowind.esm" is not of the expected type (OR STRING SYMBOL CHARACTER).Exiting...
Cache built OK before, in esper v12.

Also I can confirm that trying to --use-cache without one results in a nice, plain language error message]
User avatar
Myles
 
Posts: 3341
Joined: Sun Oct 21, 2007 12:52 pm

PreviousNext

Return to III - Morrowind

cron