[RELz] tes3cmd, a small tool for modifying TES3 plugins : III - Morrowind

» Mon Aug 23, 2010 4:53 pm

Tes3cmd and its counterpart tes3lint are the most powerful tools available for anolyzing and comparing the content of two different mods (or a mod to the master files). They return very specific information that makes it easy to locate and correct the problems identified. I cannot begin to estimate the tens of hours of time these utilities have saved me in cleaning mods of potential conflicts and other errors. :trophy:

» Mon Aug 23, 2010 6:58 pm

Since the topic of fixing fogbugs in a plugin came up in another thread, I thought I'd reply here:

Well, the http://mlox.googlecode.com/svn/trunk/util/tes3cmd allows you to run arbitrary perl code on a plugin, so it can do this too:

tes3cmd modify -t cell -e 'my $mod = 0; foreach my $subrec (@$r) { if (exists $subrec->[1]->{Fog_Density} and $subrec->[1]->{Fog_Density} == 0) { $subrec->[1]->{Fog_Density} = 0.01; warn "Fogbug fixed for $subrec->[0]\n"; $mod = 1; } }; $mod;' dh_thriftshop.esp

I wouldn't expect that to be obvious

But these little bits of code could easily be put into a small file, and executed with the -p switch. So someone could cobble something up and pass it around and then it would be like:

tes3cmd modify -t cell -p fix_fogbugs.pl dh_thriftshop.esp

and fix_fogbugs.pl would basically contain the perl code to modify the appropriate fields.

The "modify" subcommand is an "alpha" feature of tes3cmd. I need to rewrite the interface, and maybe wrap the code that accesses records in an Object Oriented interface. But I've already used it a bit to wreck major damage on Arktwend, and it really was extremely useful for me.

I've tried to use your example, but for now it seems I am not able to make it work; it is certainly a powerful feature though, I think it will open many possibilities.
For now, I'll stick with tes3cmd Version: 0.36 and this batch file

perl tes3cmd fogpatch -D > debug.txt

which outputs the list of cells to fix in the debug.txt file; You need to open and fix the single cells by hand using TESCS, but it is still a great aid.
P.S. I've tried the same command

perl tes3cmd fogpatch -D > debug.txt

using last tes3cmd Version: 0.37, but it does not seem to work
For now I consider

perl tes3cmd fogpatch

a must for the mod list of any user affected by the fog bug

:thumbsup:

» Mon Aug 23, 2010 3:33 pm

I've tried to use your example, but for now it seems I am not able to make it work; it is certainly a powerful feature though, I think it will open many possibilities.

Hmm, I wonder if that's due to shell character handling in Windows. Well, I checked in a slightly newer version of tes3cmd, which doesn't need a matching switch on the command line, so if anyone wants the very latest, there's a new version.

Here's the above recipe for fixing fogbugs in a plugin, just cut and paste into a text file called "fixfog.pl", and put it in your "Data Files" directory, or wherever you want to modify your given plugin:

use strict;sub main {	my($record_type, $record_ref, $unknown, $flags) = @_;	my $modified = 0;	foreach my $subrecord (@$record_ref) {		my($type, $subrecord_ref) = @$subrecord;		if (exists $subrecord_ref->{Fog_Density} and $subrecord_ref->{Fog_Density} == 0) {			$subrecord_ref->{Fog_Density} = 0.01;			$modified++;		}	}	return($modified);}

and then run this command:

perl tes3cmd mod -t cell -p fixfog.pl myplugin.esp

I just tested this on Windows XP, and it looks like it works for me.

» Mon Aug 23, 2010 10:17 am

-clip-
Here's the above recipe for fixing fogbugs in a plugin... -snip-

Will this directly modify the plugin without making a backup?

» Mon Aug 23, 2010 3:07 pm

Will this directly modify the plugin without making a backup?

It directly modifies the plugin and also makes a backup.

The backup for plugin.esp is named plugin~1.esp. Backups aren't overwritten, but the number is incremented and a new one is made.

I'll probably make a small change to how this works in the future so it's even more foolproof, but currently, you should not lose anything if something goes wrong, just use the backup file.

Edit: Of course, I think a wise user would always keep backups of their own. As a programmer, I made plans to handle errors, but of course I can make mistakes too. Personally, I'm always very very cautious when modifying plugins. And make sure to check the results afterwords (tes3cmd dump is good for that)

» Mon Aug 23, 2010 5:13 pm

Terrific! The question was posed on behalf of the masses.
And yes, I always keep backups of original plugins as well as those modified to my liking. B)

» Mon Aug 23, 2010 11:33 am

If you're looking for things to include in tes3cmd

How about a check for "SignRotate Contamination"? Wrye used to have some information on his page under the GMST Vaccine plugin, but that doesn't seem to have made the move to the site's new home. Here's an http://web.archive.org/web/20080612060251/wrye.ufrealms.net/GMST+Vaccine.html link. I've already seen this in Ald Vendras (I think Psyringe had it included in mlox).

I'm not sure how easy such a check would be to make but AFAIK there's nothing else out there that can search for this issue.

» Mon Aug 23, 2010 6:41 pm

If you're looking for things to include in tes3cmd

How about a check for "SignRotate Contamination"? Wrye used to have some information on his page under the GMST Vaccine plugin, but that doesn't seem to have made the move to the site's new home. Here's an http://web.archive.org/web/20080612060251/wrye.ufrealms.net/GMST+Vaccine.html link. I've already seen this in Ald Vendras (I think Psyringe had it included in mlox).

I'm not sure how easy such a check would be to make but AFAIK there's nothing else out there that can search for this issue.

It's an easy check. I know I looked at this before for tes3lint because I have some data files with signrotate stuff in them lying around my test directory.

Oh, http://www.gamesas.com/bgsforums/index.php?s=&showtopic=938026&view=findpost&p=13643019. Yes. I'll put it on my todo list to make tes3lint specifically check for the signrotate and similar script contamination.

» Mon Aug 23, 2010 5:11 pm

It's an easy check. I know I looked at this before for tes3lint because I have some data files with signrotate stuff in them lying around my test directory.

Oh, http://www.gamesas.com/bgsforums/index.php?s=&showtopic=938026&view=findpost&p=13643019. Yes. I'll put it on my todo list to make tes3lint specifically check for the signrotate and similar script contamination.

Cough.

'Course I remember that, I was just checking if you did :hehe:

So, the split between tes3cmd and tes3lint is that tes3lint is for diagnosing problems and tes3cmd for fixing them (and other modifications to plugins)? Ever thought of amalgamating them?

» Mon Aug 23, 2010 10:42 am

So, the split between tes3cmd and tes3lint is that tes3lint is for diagnosing problems and tes3cmd for fixing them (and other modifications to plugins)?

Yup. As it stands today.

Ever thought of amalgamating them?

That's not entirely unpossible.

I will have to use the right incantation ... One tool to rule them all, one tool to find them, one tool to munge them all, and in the darkness...

» Mon Aug 23, 2010 11:12 am

Hmm, I wonder if that's due to shell character handling in Windows. Well, I checked in a slightly newer version of tes3cmd, which doesn't need a matching switch on the command line, so if anyone wants the very latest, there's a new version.

Here's the above recipe for fixing fogbugs in a plugin, just cut and paste into a text file called "fixfog.pl", and put it in your "Data Files" directory, or wherever you want to modify your given plugin:

use strict;sub main {	my($record_type, $record_ref, $unknown, $flags) = @_;	my $modified = 0;	foreach my $subrecord (@$record_ref) {		my($type, $subrecord_ref) = @$subrecord;		if (exists $subrecord_ref->{Fog_Density} and $subrecord_ref->{Fog_Density} == 0) {			$subrecord_ref->{Fog_Density} = 0.01;			$modified++;		}	}	return($modified);}

and then run this command:

perl tes3cmd mod -t cell -p fixfog.pl myplugin.esp

I just tested this on Windows XP, and it looks like it works for me.

Hmm, I wonder if that's due to shell character handling in Windows. Well, I checked in a slightly newer version of tes3cmd, which doesn't need a matching switch on the command line, so if anyone wants the very latest, there's a new version.

Here's the above recipe for fixing fogbugs in a plugin, just cut and paste into a text file called "fixfog.pl", and put it in your "Data Files" directory, or wherever you want to modify your given plugin:

use strict;sub main {	my($record_type, $record_ref, $unknown, $flags) = @_;	my $modified = 0;	foreach my $subrecord (@$record_ref) {		my($type, $subrecord_ref) = @$subrecord;		if (exists $subrecord_ref->{Fog_Density} and $subrecord_ref->{Fog_Density} == 0) {			$subrecord_ref->{Fog_Density} = 0.01;			$modified++;		}	}	return($modified);}

and then run this command:

perl tes3cmd mod -t cell -p fixfog.pl myplugin.esp

I just tested this on Windows XP, and it looks like it works for me.

YES! your fixfog.pl, coupled with last svn version of tes3cmd.pl (previous version did not seem to work for me) and good old small batch did more or less what I was looking for! :clap: :twirl:
fixfog.bat

FOR %%A IN (*.es?) DO perl tes3cmd.pl mod -t cell -p fixfog.pl "%%A"

» Mon Aug 23, 2010 10:20 pm

YES! your fixfog.pl, coupled with last svn version of tes3cmd.pl (previous version did not seem to work for me) and good old small batch did more or less what I was looking for!

Cool. I think the problem for previous versions was that I forgot to put a "-m ." in the command line, which means "match all records", but the latest version of tes3cmd assumes this is the default behavior for the modify command.

» Mon Aug 23, 2010 10:45 pm

Could an option be added to tes3cmd to compare a plugin against the master bethesda esms and then clean out any redundant AMBI and WHGT entries? How about the bogus CELL records that TESCS sometimes adds?

I imagine that could be done in several steps using the compare function and then delete function once the record id is identified, but it would be nice to be able to have tes3cmd do that with just on command line option. Since the record type would always be AMBI or WHGT, and the bogus CELL records are always the same set of cells (AFAIK), that should help simplify things. :angel:

[edit] I know that TESTool does this as part of its automated plugin cleaning routine, but having it just clean out the redundant AMBI, WHGT, and bogus CELL records while leaving the other record types alone is not possible.

» Mon Aug 23, 2010 4:08 pm

Could an option be added to tes3cmd to compare a plugin against the master bethesda esms and then clean out any redundant AMBI and WHGT entries? How about the bogus CELL records the TESCS sometimes adds?

That would be pretty straightforward to do. I'll put it on my todo list, and look into it. Can't really give a timeframe, though.

» Mon Aug 23, 2010 2:23 pm

That would be pretty straightforward to do. I'll put it on my todo list, and look into it. Can't really give a timeframe, though.

Great! Thanks for even considering it. -_-

» Mon Aug 23, 2010 9:13 pm

Bump for a small request... Could body part ids be renamed with this? It doesn't seem to do that atm (unless I messed up the regex, which is quite possible... :unsure: ).

Anyway I don't think I said it before so: thanks for another great tool! :goodjob:

» Mon Aug 23, 2010 3:17 pm

Bump for a small request... Could body part ids be renamed with this? It doesn't seem to do that atm (unless I messed up the regex, which is quite possible... :unsure: ).

Yes, I think the new "modify" command in the http://mlox.googlecode.com/svn/trunk/util/tes3cmd should do it, but it hasn't been tested much:

tes3cmd modify -r '/FROM/TO/' plugin.esp

Will replace the regular expression FROM with the string TO, wherever it is found. tes3cmd will make a backup of the original when you do this, but I suggest you do so also, and double check results.

You should be able to limit what records you modify with the -t RECORDTYPE (match given record types) and -m PATTERN (match records with pattern) modifiers (See also -M, -f, -F in "tes3cmd help modify"). Suggest using "tes3cmd dump -m FROM plugin.esp" first, to see what records you will match with the FROM pattern.

Let me know if that does what you are looking for.

» Mon Aug 23, 2010 4:55 pm

Er... If it's possible I can't figure out how to do it (just get an error whatever I've tried, but I'm probably doing something wrong :banghead: ).

What I was trying to do is just either prefix or append a few characters to the id. I get the impression I should be able to do that if I can figure out what to put in the command line?

» Mon Aug 23, 2010 9:17 pm

Er... If it's possible I can't figure out how to do it (just get an error whatever I've tried, but I'm probably doing something wrong :banghead: ).

Can you post the command line you used and the error you got?

What I was trying to do is just either prefix or append a few characters to the id. I get the impression I should be able to do that if I can figure out what to put in the command line?

I just tried the following on a copy of The Trouble With Millie:

tes3cmd mod -r "/zm imposter head bodypart/TEST zm imposter head bodypart/" foo.esp

and it prepended "TEST" to the head body part in both the BODY record where it is defined, and the ARMO record where it is used.

» Mon Aug 23, 2010 3:30 pm

OK now I feel dumb :embarrass:

Thanks to your example (and some sleep) I can at least replace a literal string with another now. But I'm still not sure how to restrict it to a particular record type? I did this and it worked (in a file because I hate typing

):

perl "C:\Program Files\Bethesda Softworks\Morrowind\tes3cmd\tes3cmd.pl" mod -r "/as_b_v_nord_f_head_01/zz_test/" "C:\Program Files\Bethesda Softworks\Morrowind\Data Files\Copy of vampireHeadsEtc_005.esp"pause

But this didn't work:

perl "C:\Program Files\Bethesda Softworks\Morrowind\tes3cmd\tes3cmd.pl" mod -s body.name -r "/as_b_v_nord_f_head_01/zz_test/" "C:\Program Files\Bethesda Softworks\Morrowind\Data Files\Copy of vampireHeadsEtc_005.esp"pause

-and neither did a huge bunch of variations. What am I doing wrong? :confused:

PS Sorry, I'm a bit slow with this stuff sometimes!

» Mon Aug 23, 2010 11:30 am

Thanks to your example (and some sleep) I can at least replace a literal string with another now. But I'm still not sure how to restrict it to a particular record type?

Try this:

perl "C:\Program Files\Bethesda Softworks\Morrowind\tes3cmd\tes3cmd.pl" mod -t body -r "/as_b_v_nord_f_head_01/zz_test/" "C:\Program Files\Bethesda Softworks\Morrowind\Data Files\Copy of vampireHeadsEtc_005.esp"pause

The -t body will restrict the changes just to records of type: BODY.

» Mon Aug 23, 2010 12:56 pm

Thank you!! :bigsmile:

BTW: I had it fixed in my head that just using "body" (without specifying "name") would change the nif reference as well, which was definitely not what I wanted. Turns out it doesn't, so that's all OK (I just tried replacing the "as_" this time).

Thanks so much for that! This is going to be really useful :thumbsup:

» Mon Aug 23, 2010 3:04 pm

Could an option be added to tes3cmd to compare a plugin against the master bethesda esms and then clean out any redundant AMBI and WHGT entries? How about the bogus CELL records that TESCS sometimes adds?

I imagine that could be done in several steps using the compare function and then delete function once the record id is identified, but it would be nice to be able to have tes3cmd do that with just on command line option. Since the record type would always be AMBI or WHGT, and the bogus CELL records are always the same set of cells (AFAIK), that should help simplify things. :angel:

[edit] I know that TESTool does this as part of its automated plugin cleaning routine, but having it just clean out the redundant AMBI, WHGT, and bogus CELL records while leaving the other record types alone is not possible.

I second this one... I have wanted something to do this since ...well a long time. I'm patient so if its on your "todo" list... that rocks.

Thks

» Mon Aug 23, 2010 9:35 pm

Just curious, are there any programs that are able to batch replace any given object with another?

A bit like search and replace does, but for the whole plugin instead of just one cell at a time

» Mon Aug 23, 2010 11:09 am

Just curious, are there any programs that are able to batch replace any given object with another?

Well, tes3cmd can do that, like in the example I gave for melian above, but you have to be very careful as it's really doing a string replacement.

A bit like search and replace does, but for the whole plugin instead of just one cell at a time

The construction set allows you to apply the search and replace to all cells at once, if you deselect the option that says to apply to the current cell. Or did you mean something else?