How could I forget that question. :facepalm:

Anyway thanks for the reminder. :wink_smile:

:whisper: You can still use the feature of BAIN in Wrye Mash and here is the http://www.gamesas.com/index.php?/topic/952489-relz-bain-for-mash/page__view__findpost__p__13761743. :read:

Essentially, yes, BUT if you tell it to install a mod and that mod has some files that share the same name as those already on record, it makes a back-up of the files that are going to be replaced, so the new ones can still be 'underwritten' later if necessary.

What I have in mind would have all the functionality of the current "Installers" tab, but without the need to have a duplicate of every file.

The answer is just use Anneal in the pop up menu (right click on the selected mod archive).

No worries, I can see you answer questions to a lot of guys around here.


I don't understand how the feature you speak about works yet, but I'll have a look at it, thx!

Just read this about BAIN of http://cs.elderscrolls.com/constwiki/index.php/Bain#Complex_Packages into one archive and that information can give you a hint of what I mean about using BAIN in Wrye Mash. :read:

What? Where are these files backed up?

I was under the impression that the only way to get back the files that were overwritten was to anneal the old files back in.

Such that if I use a mod manager other than BAIN (or manually) to install over texture replacers that were installed via BAIN then decide to remove them -- the files would still be missing as far as BAIN was concerned and the only way to get the ones that BAIN thinks should be there is to anneal.

I'm not aware of any backing up of files - that is what the archives themselves are for.

What am I missing here?

This isn't a current feature, it's what Mash would look like to me in an ideal world. That way the archives wouldn't be needed after installation.

Then I could say to you:

I'm not aware of any reason to keep the archives - that is what backing up the files is for.

Well since many archives are already compressed they may end up taking less space than backing up the extracted files. Whether you save the files in a back up or in the original archive - what is the difference?

I don't see the need myself - actually except that what your proposing seems to be adding another layer prone to errors. Annealing takes care of all of this and automates the exacting out of archives as needed when overwrite are removed. So the information is saved, but the file is kept in the archive.

I generally repackage everything anyway and disc space is not a concern to me at all.

Plus updating mods is easier by simply replacing the archive then clicking anneal.

Given that harddrive space is about ￡0.04/gb, I can't see the issue with harddrive space. You could probably store every Morrowind mod ever made for less than ￡10.

You also have to remember that the cost would be far more CPU cycles are needed to install/remove mods. If you had a mod that basically overwrote a lot of another mod it would be fairly costly to backup the to be overwritten data (compressing is an expensive operation). The design would also be far more fragile. For example there is the issue that if you manually installed a mod for some reason you totally mess up the installers.

In other words, it would be a lot of work (rewriting and testing a lot of logic, basically most of it) and have some major downsides. I don't think it is worth it for saving someone ￡1 of harddrive space.

Yes Anneal do takes care of such things and the disc space doesn't concerns me at all. But I didn't do that with patches though.


I think that he means the "Convert to a project..." feature there is in BAIN for a mod archive or have I misunderstood him completely

I figured it would be a lot of work. Just thought I'd throw it out there. I still think that if it did become a working system, it could be better than the current system, but I understand that, practically, it's just not worth it.

I'm currently stuck with some really old hardware so disk space is a concern for me, but hopefully I'll be getting a new laptop soon so this won't be an issue.

I went back and removed the hyphens "-" from any package name, and got a slightly different traceout than previously, but ultimatley it was the same UnicodeDecodeError and position at the end. So other than the parenthesis ( ) and spaces, nothing out of the ordinary. These are all http://www.asciitable.com/, however, so not sure about your comment about using non ascii file names.

I next tried WMSA 84 dcg003 again, and now it failed too, same traceout. So this time I made sure to uninstall all Python and Wrye Mash installs, then reinstalled WMSA 84 dcg003 (Melchor's version) and WMSA 85 (Yakoby's version).

First tried Melchor's ... it worked. Closed, reopened, it did a quick scan rather than a full refresh. Yay! :twirl:
Next tried Yakoby's ... it worked, too! Same process, did a quick scan the second time. Double yay! :twirl: :rock:

So ... what happened? Was it the installation of Python that broke things? I can't imagine why. Python was installed with WMSA 84 dcg003, and it worked originally, until I dropped WMSA 85 in on top of it. So I must have broken something when I did that. :shrug:

Haven't tried to reinstall wxPython and try legacy Wrye Mash to see if this will break it all over again. Not sure that I want to now that I have a working version of WMSA 85...

*******************************************************

Now for something completely different ... :blink:


1. Wrye Mash 85 still displays "Wrye Mash 84" in the title.

2. How about you compile in this little http://www.thedarklands.com/wrye_monkey.ico with your next build (right-click, save target as) or your own if you don't like using Wrye's avatar? Always annoys me when programs (and their shortcuts) display some wretched default program window icon, and not with something distinct that makes it easy to identify. Besides ... it's classy. :teehee:


3. All versions I've seen of Wrye Mash seem to have a bit of a quirk in the Installers tab...

While the list of package names would properly display an ampersand (&) in the names, it would drop it from the subpackage and plugin name.

Example...

"Imperial Chain Boots & Gauntlets" package name properly displays in the packages window.

"00 Imperial Chain B&G" subpackage displays incorrectly as "00 Imperial Chain B G" with an extra space where the & should be.

"Imperial Chain Boots & Gauntlets.esp" would display incorrectly as "Imperial Chain Boots Gauntlets.esp" with an extra space where the & should be.

Other than that, thanks and great work! Appreciate you taking this on as a side project when you know we're all going to be bugging you to death to fix things and tack on new features. :ahhh:

Take care,

-RMWChaos

Oh bloody hell ... a new error.

When clicking the "Readme" option in the Mods tab got two traceouts, one right after the other. Subsequently clicking "Readme" gives the second error twice in a row afterwards.

Traceout #1...

Traceback (most recent call last):  File "masher.pyo", line 5549, in Execute  File "masher.pyo", line 2571, in __init__  File "wx\lib\iewin.pyo", line 57, in __init__  File "wx\lib\activex.pyo", line 105, in __init__  File "comtypes\client\__init__.pyo", line 110, in GetBestInterface  File "comtypes\client\_generate.pyo", line 120, in GetModuleLookupError: unknown encoding: mbcs

Traceout #2...

Traceback (most recent call last):  File "masher.pyo", line 5551, in Execute  File "masher.pyo", line 2749, in SetMod  File "masher.pyo", line 2796, in SetDocTypeAttributeError: 'DocBrowser' object has no attribute 'mainSizer'

Separately, another question. The point of my going to WMSA 85 was to get the integrated TES3CMD feature, which worked to clean a mod; however, where in the heck does it put the -cleaned.log and backup .esp files? Looked in /Morrowind, in /Data Files, in /tes3cmd, and in /Wrye Mash, but can't seem to locate it.

Probably in the docs somewhere ... going off to read for a bit then. :read:

I think you did the right thing when you didn't reinstall wxPython again once it was removed, but I do know that Melchor's WMSA didn't install any Python related components upon an installation of WMSA. Just three required windows library was installed.


I've not seen the Wrye Mash 84 title just Wrye Mash 85 as it should be, but I agree with you about the Wrye Mash icons to be included in the next build.

Will commit the current fixes soon.


Cannot reproduce.




Should be pulled from Wrye Mash.txt. Ensure it is up to date

Fixed.



Backup esps should be in the default location. I don't know if things are logged. Probably not. You can however view the log for each file by clicking on it on the cleaning output thing.

EDIT:
You can now save the log

----

EDIT:
Done.


Done in local version. Will push to remove soon.

----



Fixed.

I will try and provide a new stand alone build tomorrow.

Can someone remind me about whether I should be updating the masters for plugins under the mods tab - like you do for masters under the saves tab?

I forget.

There is a bunch of yellow - and the best I seem to be getting is green.

thanks

The most obvious one is the Better Heads Tribunal Addon.esm, Better Heads Bloodmoon Addon.esm and I always update it's master, so if you mean those two that's okay because it won't effect the game it self. In fact by doing that will reduce the errors you might have when you're launching the game, which is logged in the Warnings.txt file. I also know that there is some mods (esm / esp or both) out there, which requires to be updated in the same way and afaik the effect by doing this is the same.

Well I reviewed the original readme for mash and the included on for this version and they both state:

I'm not seeing red and I think I'm seeing a lot of yellow because I used RMWChaos' adjusted bat files to do safer cleaning of the Tribunal and Bloodmoon esm. Those are the one's showing yellow.

So with each BAIN package I make now (often themed) I will:
1. Place the esm or esp in the data folder.
2. Run mlox (along with others I suspect might conflict to get best info).
3. Clean with adjust tes3cmd bat by RMWChaos.
4. Sync to load ESM/Update and save in Wrye Mash.
5. Cut and past the result back into the folders that are my catalog of BAIN archives.
... that is as thorough as I can think to be with what I know. That way when the final product BAIN archive is cataloged by the installers tab - no further maintenance on the mod is needed - and is install and stop thinking about it.

Space is not a real issue for me with 7 terabytes. I keep an extensive catalog of the mods. This 'effort' (if you could call it that) is to finally get a complete (enough) Morrowind mod catalog ready to use and fully up to date. The catalog for Oblivion is far more of a pain as the mods are updating often.

I was trying to get a stand alone build out today, however I found a bug with the placement of the installers directory on a new install, so I want to fix that before doing a build and I think it needs more time than I am able to give it today.

To revisit my question above about whether I should update masters of mods (esp/esm) prior to repackaging them. Most of these questions should likely go into a cleaning mods thread, but it is mash that is giving me info I don't understand.

Mostly I've had yellow masters and hitting update has turned the masters green. Seems OK since I did do a cleaning of the Tribunal and Bloodmoon esm and that is what is causing that.

I've run across a few where the esp (not the line - just the checkbox) is orange - likewise so are the masters. The readme only mentions line color of orange for mods (not the masters or checkboxes) and that this is an issue of:
I know it says that only red is dangerous ... it is just that I find anything but greens, blues, and purples unsettling and I want to understand what is happening with this and why.

One instance of this is with both esm and esp from the http://planetelderscrolls.gamespy.com/View.php?view=Mods.Detail&id=2556.

On another note when updating the masters for the GIANTS patch: http://planetelderscrolls.gamespy.com/View.php?view=Mods.Detail&id=8112 (new patch to an old mod) I get this mash error:

Traceback (most recent call last):  File "F:\Bethesda Games\Morrowind\Mopy\masher.py", line 1380, in OnSave    fileRefs.load()  File "F:\Bethesda Games\Morrowind\Mopy\mosh.py", line 5081, in load    record = Scpt(name,size,delFlag,recFlag,ins,True)  File "F:\Bethesda Games\Morrowind\Mopy\mosh.py", line 1905, in __init__    Record.__init__(self,name,size,delFlag,recFlag,ins,unpack)  File "F:\Bethesda Games\Morrowind\Mopy\mosh.py", line 851, in __init__    if ins: self.load(ins,unpack)  File "F:\Bethesda Games\Morrowind\Mopy\mosh.py", line 862, in load    self.loadData(ins)  File "F:\Bethesda Games\Morrowind\Mopy\mosh.py", line 1922, in loadData    raise Tes3Error(self.inName,_('Unknown SCPT record: ')+name)mosh.Tes3Error: Z4K_Giants_AddToLev_Patch.esp: Unknown SCPT record: DELE

Thanks for any insight/help

That's generally what I do. All my mods in are in project form right now as I go through the process of cleaning and testing. Once I've updated everything so that the ESPs use the cleaned / updated masters, I update the project and will later 7zip them into a pacakge.


From the Wrye Mash Help file, Doubling, Master List Updating...

Meaning (probably) there is a master that this master relies upon that is loading later and should load earlier. :shrug: Read down further in that section for when to update masters and how to go about it.


Interesting ... I use the same patch, but did not run into any problems with it on WMSA 84dcg003 (Maloch's). Haven't reinstalled my mods yet since moving to WMSA 85 (Yakoby's); so couldn't tell you if this is a problem with your plugin or a glitch in the Mash code.

I'll take a look at it a bit later and let you know what I find out. In the meantime, you might try the following...

1. Unload all mods.
2. Reload a "dirty" (not cleaned) version of GIANTS.
3. Reload a "dirty" (not cleaned) Z4K's add to level patch.
4. Test for the error again.

I read some where that GIANTS should not be cleaned, but am looking into that. As I recall, GIANTS has both an ESM and an ESP (control file), and the ESP is what you want to be careful about cleaning because it intentionally has a bunch of records that are duplicates of the GIANTS.esm file. This prevents changes required for GIANTS to work from being overwritten by later mods. Same is true of several other mods.

I'm looking into several of these now. TES3CMD can be controlled to leave certain records alone and clean others. It might be as simple as telling it to clean anything that matches Morrowind.esm, Tribunal.esm, and Bloodmoon.esm, but ignore any duplicates to other masters. I'll post that in the other thread related to cleaning once I finish playing around with everything that TES3CMD can do.

Hope that helps,

-RMWChaos

@Yakoby,

I'd love to get my hands on your Mash 85 source code, or at the very least masher.py.

Not sure if you're using the same tes3cmd hack that Abot produced...

#--begin changes by abot# tes3cmd.exe clean --replace [selected files]#------------------------------------------------------------------------------class File_tes3cmd(Link):    def AppendToMenu(self,menu,window,data):        Link.AppendToMenu(self,menu,window,data)        menuItem = wx.MenuItem(menu,self.id,_('tes3cmd.exe clean --replace [selected files]'))        menu.AppendItem(menuItem)        if len(data) < 1: menuItem.Enable(False)    def Execute(self,event):        message = _("This command will call Data Files\\tes3cmd.exe clean --replace passing selected files.")        if ContinueQuery(self.window,message,'masher.File_tes3cmd.continue',_('tes3cmd.exe clean --replace [selected files]')) != wx.ID_OK:            return        #--Scan        fileInfos = self.window.data        path = os.path.join(settings['mwDir'],'Data Files')        exe = os.path.join(path,'tes3cmd.exe')        args = ['tes3cmd.exe']        args.append('clean')        for fileName in self.data:            args.append('"' + fileName + '"')        args.append('--replace')        cwd = os.getcwd()        os.chdir(path)        os.spawnv(os.P_NOWAIT,exe,args)        os.chdir(cwd)        #--Refresh        fileInfos.refreshDoubleTime()        self.window.Refresh()#--end changes by abot

...but I'm playing around with it by adding some additional args to allow for a customizable backup ESP/M and -cleaned.log location. The --replace option *should* create a "plugin~1.esp" in the same location as the original "plugin.esp"; however, I did not find this to be the case with the integrated tes3cmd.

Additionally, it might be worthwhile to create an "exclusion" list of ESP/M files that do not get fully cleaned, such as not removing the GMSTs from certain mods that require them or not removing duplicate entries in ESP control files against their masters (i.e. GIANTS). This would be a user-modifiable list, of course, with some default plugins and masters listed.

Of course, that sets up some additional work to support it and keep it updated, which considering your original post that you were only sharing your own personal work on this with others and couldn't promise anything going forward, might not be something you want to do.

In any case, there is a great deal of room to add some "intelligence" to how tes3cmd cleans mods within Mash, and thought it would be fun to work on it, unless you're already planning to do something similar.

I can do all this pretty simply via command line, but will have to do a bit of research to translate that into python code, which sounds like just the kind of fun project that gets my juices flowing. :hubbahubba:

Regards,

-RMWChaos

@RMWChaos:
The source code is here:
https://github.com/Yacoby/Wrye-Mash

The tes3cmd code is significantly more complex than Abots. Basically it is a https://github.com/Yacoby/Wrye-Mash/blob/master/Mopy/mash/tes3cmd/gui.py that calls a https://github.com/Yacoby/Wrye-Mash/blob/master/Mopy/mash/tes3cmd/__init__.py for every file. The wrapper is a bit more complex than simply spawning the tes3cmd exe in that it is threaded to maintain gui responsiveness while processing the files. This allows things like stopping the process or skipping it if you tried cleaning something that is taking ages. (If you are looking at the code and going wtf, I can write some more documentation for it)

The output of tes3cmd is parsed and displayed in the GUI.


I implemented the things from your .bat file to be on the safe side. I have yet to check if they are needed or not. I get the idea from John's documentation that he intends the clean function not to do any 'bad' damage ever.


I think I hid the backups as I didn't like the fact that it put them in the same directory as the main data files. I left the backups in the default directory with the hide backups arg. Quite possibly it is in Data Files/tes3cmd. :shrug:


Even in esp control files, duplicates shouldn't matter. Surly?

Great, I'll start taking a look at it and see how much stuff I can break. Will have to go back and reinstall python now. Which version are you using, btw? Want to make sure our environments are similar to prevent any differeing behavior during testing.

Once I get a chance to look at the code, and after I finish un-crossing my eyes, I'll let you know if I have any questions.

Yes, I saw the GUI output, but had expected a -cleaned.log file somewhere to review in case I needed to check on what may have broken a mod after cleaning it. Ran into this immediately after I accidently closed the GUI output and couldn't locate a log file to review what it cleaned.


Oh Dear Lord. We're all in trouble now if you're implementing anything I thought up. :geek: I can tell you that if you attempt to clean Tribunal.esm or Bloodmoon.esm with tes3cmd, it WILL remove the EVIL GMSTs, which TMK are supposed to be in those masters only.

Here's an example output of cleaning Tribunal.esm (the important part is really at the end of the output where you see 61 Bloodmoon and 5 Tribunal Evil-GMSTs were removed)...

All those duplicate records, junk cells, and redundant cells can go, but the Tribunal Evil-GMSTs should stay. Not sure about the Bloodmoon ones, though. They could potentially go as well, and with tes3cmd you can easily tell it to ignore only records that match Tribunal Evil-GMST ... same for Bloodmoon with its Evil-GMST records.

Now why would Tribunal even have Bloodmoon Evil-GMSTs in it if it was created before ...? Patch, maybe, introduced it?


I seriously could not find it anywhere, not even in /Data Files/tes3cmd. According to the http://code.google.com/p/mlox/wiki/Tes3cmd, the default backup location (when enabled with --hide-backups) is /home/doug/wine/morrowind/drive_c/morrowind/Data Files/tes3cmd/backups, which sounds more like it's made for the Linux OS than Windows.

The documentation suggests you should be able to change the default backup directory with the --backup-dir option; however, in testing this breaks the batch commands because tes3cmd now uses that location as the original location of the file to clean. I'm going to play around with it some more today to see if I can get it to work properly.

Haven't tried playing with the --output-dir to see what problems that may cause.

My vote would be to default tes3cmd mod backups either to the /Data Files/Mash/Backups folder, where backups are created via the Mods tab menu File-->Backups option, or to add a new line in the Settings Window for "tes3cmd backups" that the user can set. I can see an argument for both, but am concerned about overwriting one backup with another if using the first method.

Oh wait, this is easier than I thought ... well, maybe. You could have Mash automatically create a copy of the ESP/M using it's backup code before firing off tes3cmd. The problem is that tes3cmd only creates a backup if it finds something to clean. So, if you already created a backup of the 'dirty' mod the first time, and now have a clean mod, running the Mash backup would overwrite that file with the cleaned version and you'd lose your original dirty one.

That tells me Mash backups and tes3cmd backups should be in different folders. Unless there's something here I don't know about Mash code and how it makes backups of mods ... does it do a CRC check or something to make sure that two mods with the same name but different content do not overwrite each other?


Maybe I'm misunderstanding, but the duplicates in ESP control files are intended to prevent overwriting of ESM entries by other mods, but I really haven't done a deep dive into say the GIANTS ESP control file to see what it contains as compared to its ESM. So you don't want to clean those records, or you cancel out the purpose of the control file entirely.

Thanks for the source code and for wasting your time listening to anything I have to say! :whisper:

Take care,

-RMWChaos

Well to be clear the z4K's esp is meant to work with the original GIANTS esm and that is what I had loaded (and was not red itself). The Unofficial Patch had both red.

And no it is not about masters out of order. The only mods loaded that these have as masters are (and at most) Morrowind.esm, Tribunal.esm, and Bloodmoon.esm - and they were in that order. No other plugins even in the data folder. I only move in and out what I'm currently working and do not have a cluttered data folder.

While the checkboxes are orange - if I check the box the line does not turn orange. I can though click update masters and all the orange goes away.

Sorry if this seems off topic, Yacoby, I'm asking here because this is what mash is telling me and I'm hoping that understanding why this happens will lead to more insight into cleaning.