[RELZ/WIPZ]PyFFI: Python File Format Interface

Post » Tue May 11, 2010 9:27 am

Thread updated on March 26th, 2011

http://www.gamesas.com/index.php?/topic/1158842-inforelz-pyffi-python-file-format-interface/

Post Update Headlines

PyFFI version 2.1.9 has gone stable.

Quick Links

http://www.gamesas.com/index.php?/topic/1158842-inforelz-pyffi-python-file-format-interface/
http://www.gamesas.com/index.php?/topic/1123756-inforelz-pyffi-python-file-format-interface/
http://www.gamesas.com/index.php?/topic/1112284-inforelz-pyffi-python-file-format-interface/
http://www.gamesas.com/index.php?/topic/1061938-info-relzpyffi-python-file-format-interface/
http://www.gamesas.com/?showtopic=1014947

PyFFI Thread 4 - too old and has been purged.
PyFFI Thread 3 - too old and has been purged.
PyFFI Thread 2 - too old and has been purged.
PyFFI Thread 1 - too old and has been purged.

http://pyffi.sourceforge.net/
http://www.python.org/download/releases/2.6.6/ (required to Run PyFFI)
https://github.com/amorilia/pyffi/downloads
http://sourceforge.net/tracker/?group_id=199269 (sub-trackers for bugs, requests etc.)
https://github.com/amorilia/pyffi (ie the earliest that you can see changes/additions that will be in the next release is here)
http://www.tesnexus.com/downloads/file.php?id=37463 A set of INI files and batch scripts designed for complete bulk mesh optimizations. Official support thread http://www.gamesas.com/index.php?/topic/1172029-relz-oblivion-pyffi-optimization-kit/
http://www.tesnexus.com/downloads/file.php?id=37548 For those who cannot or wish not to run the full optimization process. Official support thread http://www.gamesas.com/index.php?/topic/1173353-oblivion-pyffi-binary-patch-kit/



ESSENTIAL INFO FOR ALL USERS


What is PyFFI?

PyFFI stands for Python File Format Interface and was written almost entirely by Amorilia (Contributors: wz, taarna23, seith, tazpn, PacificMorrowind)

Latest Changelog

https://github.com/amorilia/pyffi/blob/python25/CHANGELOG.rst

Release 2.1.9 (26 March 2011)
==============================

* Added new script to make and apply patches (functionality is identical to and OnmyojiOmn's and jonwd7's pyffi patcher scripts, but it is written in Python to work on all platforms).
* New fix_emptyskeletonroots spell (automatically included in the optimize spell) to fix issues with nifs that do not have their NiSkinInstance skeleton root set (fixes #3174085, reported by xjdhdr).
* Fixed logging issue on Windows platform with multithreading enabled (fixes issue #3174339, reported by xjdhdr).
* Fixed QSkope shortcut issue when path contains spaces (reported by Brumbek).
* Skip terminal chars in mopper output (fixes issues with running mopper under wine).
* Bugfix in patch_recursive_make/apply scripts for "deep" folder layouts (fixes issue #3193914, reported by xjdhdr).
* Do not pack collisions in OL_CLUTTER (fixes issue #3194017 reported by Gratis_monsta).
* Fixed issue with skipping terminal chars in mopper output on Windows platform (fixes issue #3205569, reported and diagnosed by ulrim).
* Disabled copy in patch_recursive_apply script (see issue #3219744, suggested by ulrim).
* Pass additional arguments of patch_recursive_apply/make to the patch command (see issue #3219744, suggested by ulrim).
* Sane default settings for bhkRigidBody unknowns to ensure that constraints behave properly (contributed by Koniption).

NOTICE: If anyone has performed mesh optimizations on a previous version of 2.1.9, you will need to redo the optimization process, as crucial fixes to possible disappearing clutter items have been made.

FAQ

Q - Is it safe?
A - Yes; many Oblivion players and modders have run it on thousands of nifs with no problems. There is of course the potential for problems but if so as long as you made a backup it is relatively painless to revert some meshes. Of course there are some meshes that you should NOT run it on (any with egms is the main type) - if you do run it on them you will likely run into problems.

Q - Is it safe to PyFFI meshes that are already PyFFI'd?
A - While performance benefits will be negligible, yes, it is perfectly safe to run meshes that are already PyFFI'd through PyFFI again.

Q - Will running a mesh that had errors introduced into it by a previous version of PyFFI have those errors repaired by running a later version on the bad mesh?
A - No, PyFFI cannot repair any damage that has already been done, and you will need to run the vanilla version of the mesh through PyFFI again. Make sure you keep a backup of the default meshes BSA in case you run into any problem meshes.

Q - Why is PyFFI increasing the file size of my meshes?! Shouldn’t optimization be reducing their size?
A - As of PyFFI 2.1.6, much of the optimization work now occurs through triangulation (improves performance, but increases file size) rather than stripification (improves performance while decreasing file size).

Q - Does PyFFI support muli-core processors?
A - Yes, PyFFI will utilize multiple processor cores and hyperthreading if they are available.

Q – Does PyFFI support the 64bit version of Python?
A – No, PyFFI does not officially support 64bit Python, and using it with the 64bit version can cause numerous problems.

Q - Is there a visual quality loss?
A - No, PyFFI removes duplicate and empty data from the nif file.

Q- What's the latest stable version of PyFFI?
A- As of March 26th, 2011, 2.1.9

Q- What's the latest in-development version of PyFFI?
A- Currently none, the latest Dev. version has gone stable.

Q- Are there any meshes that I should not put through PyFFI?
A- Yes, there are some meshes that can cause problems.

- Any meshes with FaceGen files attached to them (that is, any NIFs that are accompanied by an EGM or TRI mesh). These should not be optimized, or visual anomalies (E.G. tears in the model) might result.
- The Roothavok meshes. It is currently unknown why these don't optimize properly.

Q - I got a weird bunch of test in the command line/in the log file after running PyFFI, what does it mean?
A - Usually that would be in the form of something like this:
*** TEST FAILED ON C:\PyFFI_Op\XMSGroundFleshContainer01.nif ****** If you were running a script that came with PyFFI, then ****** please report this as a bug (include the file) on ****** [src="http://forums.bethsoft.com/index.php?/topic/1172342-relzwipzpyffi-python-file-format-interface/http://sourceforge.net/tracker/?group_id=199269"]http://sourceforge.net/tracker/?group_id=199269[/url] ***


Q - Why am I not getting the option to run PyFFI when I right click the Oblivion_Optimize.INI?
A - You have a third party text editor (E.G. Notepad++) that is interfering with the drop-down options menu and preventing this option from displaying. This is fixed by uninstalling the interfering text editor.


It means that there is something about the nif that PyFFI doesn't know what to do with or something like that, or sometimes a bug in the spell. If you get that it means the file will not have been optimized and will have been left unchanged so no fixing is necessary to play with it. However if you can report that here or on the sourceforge tracker if you have a sourceforge account, it can probably be fixed. To be fixed we would need
A: to know what spell you were running
B: a copy of the nif or kf.

What is of interest for Oblivion modding right now is what it can do with nif/kf files:
It contains a script called niftoaster.py which is an python wrapper script that runs other scripts on files fed to it. Niftoaster can do a whole host of things to nifs/kfs including: optimizing models, fixing many different model errors, changing the collision type/material of nifs etc. etc. It is also highly extendable to do whatever you want.

One of the most important spells (the name for each of the scripts) is Optimize; it actually runs a whole bunch of the fixing and optimizing spells in tandem; for mod users it is really the only one you're interested in; and for a lot of mod makers it is also the only one your interested in.


Why Should You Care?

Really you don't have to; however, if you make meshes you should, do retexturing etc. or if you want to try and squeeze a little bit of extra performance out of Oblivion.

With the latest versions of PyFFI, performance increases can be quite substantial, so running optimization spells on your meshes (including the vanilla meshes packed in the BSA Archives) is highly recommended.

Installation

Notice for Windows Vista/7 Users: Do not install Python or PyFFI to the Program Files directory. UAC will make life miserable for you if you do.

Prerequisite:http://www.python.org/download/releases/2.6.6/
Currently only the 32bit version is supported.

(If using the Installer version)
-Download https://github.com/amorilia/pyffi/downloads (Windows Installer)
-Run the Installer and follow the onscreen instructions.

(If using the manual version)
-Download https://github.com/amorilia/pyffi/downloads (.ZIP Archive).
-Uninstall any previous versions of PyFFI
-Unzip the package to wherever you wish for it to go (Windows Vista/7 users, see notice above).
-Run the following command from the command prompt window:
cd C:\(Navigate to your PyFFI directory here)C:\python setup.py install

->Example:
cd C:\PyFFIpython setup.py install

(You need to have Python registered in your system’s environment variables for the python command to function properly)
-If you have setuptools installed, you can streamline the process by running this command line:
easy_install –U PyFFI


Uninstallation

(If using the Installer version)
-Run the PyFFI_Uninstall.exe

(If using the manual version)
-Delete your PyFFI Directory
-Delete the PyFFI folders found in the Lib\site-packages folders in your PyFFI 2.5 and Python 2.6 directories.

Using PyFFI

The scripts PyFFI uses are termed “Spells.” PyFFI provides several spells for you to use, the most important of which is “Optimize”. There are several other spells for mod developers to use, but the layman user will likely only be interested in the optimize spell.

There are far too many spells available to go over in one forum thread, so only the most important ones will be described here:

Optimize
Modify_makefarnif
Modify_texturepath
Modify_collisiontype
Modify_Scaleanimationtime
Modify_reverseanimation


INFO CONCERNING MOD USERS/PLAYERS AND DEVELOPERS


Spell: Optimize

This is where the brunt of PyFFI is located. The optimize spell removes duplicate/ empty/ uneeded data from your meshes, decreasing the workload for your GPU to render them. This can result in noticeable performance increases if you are to, say, PyFFI the entire vanilla mesh set.

How To:

http://tesivpositive.animolious.com/index.php?page=pyffi but for those who may just need a refresher, I'll give you the quick version:

First, you will need the meshes that you wish to optimize. These are usually found loose or packed into a Bethesda Softworks Archive (.BSA) file. If they are loose, it is best that you find the package for the mod whom the meshes belong to and extract them to a folder outside the Oblivion directory. It is not recommended to run PyFFI on the meshes folder already inside the data folder for Oblivion.

If the meshes are packed in a BSA, use a tool like Oblivion Mod Manager or BSA Commander to unpack the archive to a separate folder.

Once you have the meshes you wish to optimize, you have two options:

1. Right-click the meshes folder and select Optimize with PyFFI
2. Place the meshes into your PyFFI’s in folder (ex: C:\Program Files\PyFFI\utilities\toaster\in) and run PyFFI from the oblivion_optimize.ini

The former option optimizes the meshes and overwrites the originals. The latter option writes the new meshes to an \out folder so the original meshes are retained.

Further optimization for _far.NIFs (VWD meshes) can be called with the following batch script:
"C:\Python26\python.exe" "C:\Python26\Scripts\niftoaster.py" --noninteractive opt_cleanfarnif --dest-dir= --source-dir= --pause --overwrite "%1" > "PyFFILog.txt"


Once you are finished, gather up all the meshes into one spot (only really concerns users of option 2). If you the meshes were provided loose, rebuild the mod's package if you intend to use BAIN or OBMM installation so to include the optimized meshes. If you are optimizing a BSA, repack the BSA, and rename it to whatever it has been defined to be in your sArchiveList INI entry (if optimizing the Vanilla BSA) or to ensure your mod's .ESP file will allow the game to find it. Don't forget to reset the BSA timestamps, either, or you may find the vanilla meshes overriding your mod-added meshes!

Example of PyFFI in Action doing the Optimize spell:

reading C:\PyFFI_Op\Weapons\braided\claymorefine.nifchecking for duplicate source textureschecking for duplicate propertiesremoving duplicate and empty childrenoptimizing geometriesoptimizing block 'Claymore:0'removing duplicate vertices(num vertices was 732 and is now 615)recalculating strips(strip length was 1518 and is now 1188)(average strip length is 20.917508)stitching strips (using 103 stitches)recalculating tangent spacewriting C:\PyFFI_Op\Weapons\braided\claymorefine.nif...


Known Issues

Take heed that there are several meshes that are incompatible with the optimization process:

-Any hair mesh
-Roothavok meshes (the hanging roots you commonly see in caves)
-Any mesh with a .EGM association (commonly helmets)
-Any mesh with a .TRI association (teeth, eyes, etc.)

Using option 1, therefore, is not recommended unless you are certain none of the above meshes exist in the set you are optimizing. The oblivion_optimize.INI comes with a skip list that will tell PyFFI to skip the problem meshes during the optimization process, and it works pretty well, so if you are, for instance, optimizing the entire vanilla meshes BSA, it is recommended to use option 2.

Watch out for these meshes if you are doing a vanilla mesh optimization:
meshes\dungeons\caves\exterior\ centrancechighlandsm01.nif
meshes\dungeons\caves\exterior\ centrancegoldcoastsm01.nif
meshes\dungeons\caves\exterior\ centrancegreatforestsm02.nif
meshes\dungeons\caves\exterior\ centrancemtnsnowlg01.nif
meshes\dungeons\caves\exterior\ centrancemtnsnowlg02.nif
meshes\dungeons\caves\exterior\ centrancemtnsnowlg03.nif
meshes\dungeons\caves\exterior\ centrancemtnsnowsm01.nif
meshes\dungeons\caves\exterior\ centrancemtnsnowsm02.nif
meshes\dungeons\caves\exterior\ centrancemtnsnowsm03.nif
meshes\dungeons\caves\exterior\ centrancerockmosssm01.nif
meshes\dungeons\caves\exterior\ centrancerockybeachsm01.nif
meshes\dungeons\caves\exterior\ centrancewestwsm01.nif
meshes\dungeons\caves\exterior\ centrancewestwsm02.nif


Their names contain leading spaces that it has been reported by Windows 7 64bit users get truncated if you move the meshes folder around after you’ve finished the optimization process. If you try to use those meshes in that state in the game, the game will report them as missing. Windows 7 64bit users should DEFINITELY watch out for this, and Windows 7 32bit and Windows Vista (all versions) users should be aware of the issue. Windows XP users are safe from this.

If you've used PyFFI before, previous to v.1.0.0 of PyFFI did not triangulate NifTriStrips with lengths less than 10. Unfortunately this causes Oblivion issues and while cause frame stuttering, and on low performance machines can virtually halt the game.

In addition, the 3dsMax NifTools exporter bloats the exported nif file with duplicate everything (or very nearly duplicate everything). This is a known problem, which I believe is due to 3dsMax and not directly to the NifTools.

Blender and Maya do not suffer this problem.

Though that does not mean that duplicate data will not appear in meshes made with Blender or Maya.

Cool technique (courtesy of Arthmoor and H2ODK):

If it helps anyone, this is my current Oblivion_optimize.ini file:
[main]; run optimize spellspell = optimize[options]; any patterns of files that should be skipped, being as conservative as possible; (without quotes, separate different regular expressions by a space); at the moment:; - skipping hair nifs (vertex ordering!); - skipping roothavok nifs (not sure why, investigating); - skipping any nif that is known to have an egm or tri associated with it;   find . -name "*.egm" -or -name "*.tri" | sed 'sX.*/XXg' | sed 'sX.tri$XXg' | sed 'sX.egm$XXg' | sed 'y/ABCDEFGHIJKLMNOPQRSTUVWXYZ/abcdefghijklmnopqrstuvwxyz/' | sort | uniq | xargs;   (not necessary if all egm files are included, such as in vanilla Oblivion,;   but some mods only include nifs without egm files, and this makes sure;   these cases are handled as well); note: you can safely remove this list if you are sure that you are; optimizing fully extracted folders onlyskip = (?i)(?

That nasty looking skip sequence at the end was something H2ODK came up with and works well, but it needs a slightly altered process to be sure everything gets done. That's where oblivion_optimize2.ini file comes in:
[main]; run optimize spellspell = optimize[options]; any patterns of files that should be skipped; regex expressions; only run on files with these patternsonly = (?i)gnd[.]nif$ (?i)meshes.clutter


When I do an optimization, I use two batch files. #1:
"C:\Python26\python.exe" "C:\Python26\Scripts\niftoaster.py" --noninteractive --ini-file="C:\Program Files (x86)\PyFFI\utilities\toaster\default.ini" --ini-file="C:\Program Files (x86)\PyFFI\utilities\toaster\oblivion_optimize.ini" --dest-dir= --source-dir= --pause --overwrite "%1" > "PyFFILog.txt"


#2:
"C:\Python26\python.exe" "C:\Python26\Scripts\niftoaster.py" --noninteractive --ini-file="C:\Program Files (x86)\PyFFI\utilities\toaster\default.ini" --ini-file="C:\Program Files (x86)\PyFFI\utilities\toaster\oblivion_optimize2.ini" --dest-dir= --source-dir= --pause --overwrite "%1" > "PyFFILog.txt"


The first one takes ages to complete, the second usually goes by fairly quick because it's only looking for a specific set of info. What these do though is add a much more robust extra layer of "just in case" so that it REALLY won't touch any helmets and/or hairs and such.

I also have one other used specifically for cleaning any loose _far.nif files:
"C:\Python26\python.exe" "C:\Python26\Scripts\niftoaster.py" --noninteractive opt_cleanfarnif --dest-dir= --source-dir= --pause --overwrite "%1" > "PyFFILog.txt"


You may also notice that they all drop a log file when they're done. That's handy for quick-scanning for the word error or warning to see if any bugs cropped up. It also reduces the amount of spam from the handler messages.



PyFFI Patcher

http://www.tesnexus.com/downloads/file.php?id=37548

One of the many hurdles involved with the usage of PyFFI was the inability to redistribute the optimized meshes for the game for copyright reasons. This is now no longer the case with PyFFI Patchers. This separate utility allows for the user to instead download a patch file containing only the differences between the vanilla meshes and the PyFFI optimized meshes and simply apply the mesh patches to the old meshes.

The advantage? It takes a lot less time and is much less involved for a user to install these PyFFI Patches rather than run the PyFFI optimization process themselves.

The downside? These patch files will need to be redistributed every time PyFFI is updated to improve the optimization process for Oblivion's meshes. This will mean increased time spent on downloads.

Is it worth it? If you do not have the time available to invest in learning how to use PyFFI, or you wish to perform a one-off optimization of your meshes, then yes, it is worth it. If you wish to remain on top of the latest developments for PyFFI, however, it may be easier to simply run the optimization process yourself.

A fairly recent addition in the form of the PyFFI Binary Patch KIt has now been released, as well. This tool not only makes it easier to obtain the PyFFI optimized meshes by allowing the users to patch their own meshes, but it also makes it easier to create your own binary patch files, too. Here's how it is done:

With 'Oblivion - PyFFI Binary Patch Kit' you'd have to:
    place the original meshes inside 'in'
    place the optimized meshes inside 'out'
    open 'patcher.py' with a text editor and at the end change
    scan(s_dir, s_list)#scan(t_dir, t_list)scan(p_dir, p_list)#create_patch(s_list, t_list)apply_patch(s_list, p_list)
    to
    scan(s_dir, s_list)scan(t_dir, t_list)scan(p_dir, p_list)create_patch(s_list, t_list)#apply_patch(s_list, p_list)
    run the patcher and patch files are created inside 'patch' folder, recursively.


Many thanks to ulrim for his work on this. :)


INFO CONCERNING MOD DEVELOPERS ONLY


Spell: Modify_makefarnif

This spell will generate VWD meshes (_far.NIFs). It is called with the following batch script/command prompt:
 “C:\Python26\python.exe” “C:\Python26\Scripts\niftoaster.py” modify_makefarnif  --only _far.nif$ --dest-dir= --source-dir= --pause --overwrite "%1" > "PyFFILog.txt” 



Spell: Modify_collisiontype

This lets you quickly (in seconds) and accurately and on as many nifs as you want at once, change to collision type to anything that is supported (and easy to add more if desired). A real boon for Max users until the nifscripts properly support non-static collision export. The format for calling it is batch file/command prompt of:
cd C:\Python26python.exe Scripts/niftoaster.py modify_collisiontype -a static "C:\meshes" (replacing as need the mesh path and the python path, and static with the type of collision desired)



Spell: Modify_scaleanimationtime

As usual real quick and easy method to: slow down or speed up animations. will change the speed of basically any animation type (not tested on bsplines yet) called by:
cd C:\Python26python.exe Scripts/niftoaster.py modify_scaleaniamtiontime -a 0.6 "C:\meshes" (replacing as need the mesh path and the python path and the number with the desired multiplier)



Spell: Modify_reverseanimation

As usual real quick and easy method to do something... in this case reverse an animation - ie make a close animation from an open animation for a door etc.
cd C:\Python26python.exe Scripts/niftoaster.py modify_reverseanimation "C:\meshes" (unteste for bspline type animations so far)



Spell: Modify_texturepath

lets you quickly (in seconds or minutes depending on number of nifs) and accurately and on as many nifs as you want at once, change the source texture paths from x/x/wall01.dds to /whateveryouwant/wall01.dds - a real boon when it comes to retexturing a whole tileset. The format for calling it is batch file/command prompt of:
cd C:\Python26python.exe Scripts/niftoaster.py modify_texturepath -a textures/PM/BloodyAyleid/ "C:\meshes"


(replacing as need the mesh path and the python path, and the texture path with the desired texture path).

Now I'm sure some of you are thinking hmm none of those do anything for me but I sure could use something that batch changed X... but I don't know any Python oh well. Never fear; if you want a script you can always ask - not promising it will get done, but significantly more likely than without asking (and if you don't believe me just ask DaMage).

As always if something isn't clear ask and someone knowledgeable will try to answer.
User avatar
Kyra
 
Posts: 3365
Joined: Mon Jan 29, 2007 8:24 am

Post » Mon May 10, 2010 10:36 pm

Hmmm... I use x64 Python and while problems aren't present.
User avatar
k a t e
 
Posts: 3378
Joined: Fri Jan 19, 2007 9:00 am

Post » Tue May 11, 2010 4:22 am

Has anyone made an optimized Oblivion - Meshes.bsa with PyFFI >= 2.1.7 ? I'm having trouble with the latest versions.

I don't know if there's much difference from that beta in the first post but you can find the final release of 2.1.7 here: http://sourceforge.net/projects/pyffi/files/pyffi/2.1.7/
User avatar
Kerri Lee
 
Posts: 3404
Joined: Sun Feb 25, 2007 9:37 pm

Post » Tue May 11, 2010 1:20 am

Just wanted to point out that the current beta at beta 5 :) It supposedly fixes the mod-added clutter falling through floor thing...

@ulrim:
Like I mentioned in the last thread, I'm currently running a fully PyFFIed Meshes.bsa (using 2.1.9b3 without the skiplist) and have yet to run across anomalies :) Although I don't recommend everyone to try without the skiplist... the beta versions are pretty safe for pyffication.
User avatar
LittleMiss
 
Posts: 3412
Joined: Wed Nov 29, 2006 6:22 am

Post » Tue May 11, 2010 5:31 am

I'm having problems with 2.1.9 Beta 4. Whenever I right-click on a meshes folder and choose "Optimize with PyFFI", the command prompt window opens and closes quickly, and the meshes folder does not get optimized. Any ideas as to why this is happening?
User avatar
Natalie Taylor
 
Posts: 3301
Joined: Mon Sep 11, 2006 7:54 pm

Post » Tue May 11, 2010 8:08 am

I believe I did a full optimize with 2.1.8, but it's been a long time since my last big optimization attempt. I am 100% certain I did a 2.1.7 optimization, though, and I'm having no problems with it.
User avatar
Brandon Bernardi
 
Posts: 3481
Joined: Tue Sep 25, 2007 9:06 am

Post » Tue May 11, 2010 8:46 am

Did anyone had any problems with http://www.tesnexus.com/downloads/file.php?id=36882?

I couldn't get it to work, it generated only empty folders.
User avatar
Cat
 
Posts: 3451
Joined: Mon Dec 18, 2006 5:10 am

Post » Tue May 11, 2010 3:05 pm

Just a quick note to let you the links for 2.1.7b5 under Installation in your opening post don't work.
User avatar
Euan
 
Posts: 3376
Joined: Mon May 14, 2007 3:34 pm

Post » Tue May 11, 2010 9:46 am

I've figured my problem and have now a good Oblivion.bsa.

Tomorrow I'll update the pyffi kit with an improved interface and an important fix for the optimization of far.nif meshes.
User avatar
JESSE
 
Posts: 3404
Joined: Mon Jul 16, 2007 4:55 am

Post » Tue May 11, 2010 12:48 am

I'm trying to figure out a way to tweak the color information on massive amounts of meshes, can pyffi do that?

I'm talking about NiMaterialProperty: the 4 color fields (ambient, diffuse, specular, emissive) and glossiness.
User avatar
Marcia Renton
 
Posts: 3563
Joined: Fri Jan 26, 2007 5:15 am

Post » Tue May 11, 2010 4:17 am

I'm not quite sure where to report this... I'm having a problem with 2.1.9 beta 5 or the new PyFFI Optimization Kit (version 6). I've had a few cases of stuff falling through the floor, so I thought I would try the new version along with the optimization kit. Well the stuff didn't fall through the floor, but the new meshes didn't work either...

Here's the deal:

This is for my Imperial Furniture Renovated mod. The way this works is when the player buys a large piece of furniture, they are actually given a marker. The player drops the marker on the floor and clicks on it. The marker is replaced with a havoked version of the item. The player can drag the havoked version of the item to fine-tune the position. Then they click on it again and are presented with a menu. If they select "lock in position" from the menu, the havoked item is replaced with a static version. All that works fine. Now, normally the player can click on the static version and a menu will pop up with various options. That does not work with the newly optimized meshes. As soon as I remove the optimized meshes, the menu pops up as it's supposed to. Furthermore, with the optimized meshes, the collision is largely gone. I can almost walk completely through the object. However, it seems no matter where I click, the menu will not appear. Returning to the previous version of the meshes (which were optimized with 2.1.6) works fine. The collision completely surrounds the object as well.
User avatar
Sammygirl500
 
Posts: 3511
Joined: Wed Jun 14, 2006 4:46 pm

Post » Tue May 11, 2010 11:23 am

I've been testing 2.1.8 and 2.1.9b5 and I find the same results. Collision box optimization must be removing it altogether.

This has nothing to do with the skip list on the kit. From my experience optimizing the meshes, PyFFI 2.1.6 worked best without these problems.

edit: actually, it has to do with it - seems I have to check for collision optimization because PyFFI is going bananas with that.
User avatar
Arnold Wet
 
Posts: 3353
Joined: Fri Jul 07, 2006 10:32 am

Post » Tue May 11, 2010 4:55 am

@andalaybay:
You should really start an artifact in https://sourceforge.net/tracker/?group_id=199269&atid=968813 so that amorilla can check your meshes (both original & pyffied) and possibly fix it :) My experience says she responds faster to posts there than here...

It'd be great if this could be resolved within PyFFI itself.
User avatar
Rude_Bitch_420
 
Posts: 3429
Joined: Wed Aug 08, 2007 2:26 pm

Post » Tue May 11, 2010 9:21 am

@andalaybay:
You should really start an artifact in https://sourceforge.net/tracker/?group_id=199269&atid=968813 so that amorilla can check your meshes (both original & pyffied) and possibly fix it :) My experience says she responds faster to posts there than here...

It'd be great if this could be resolved within PyFFI itself.


Done. It's similar to Ulrim's problem as well. Also, it only happens with static meshes. The havoked meshes seem to be working fine...
User avatar
Kate Murrell
 
Posts: 3537
Joined: Mon Oct 16, 2006 4:02 am

Post » Tue May 11, 2010 7:29 am

The last large scale optimization I did was with 2.1.7. I haven't really done much of anything yet on 2.1.8 and haven't even downloaded 2.1.9. Nothing has been out of place in my mad wanderings so far. I certainly haven't been able to walk through bridges or had stuff fall on the floor, but then I honestly don't recall what version I last optimized most of the mods I have installed with.
User avatar
Baby K(:
 
Posts: 3395
Joined: Thu Nov 09, 2006 9:07 pm

Post » Tue May 11, 2010 7:11 am

I recently downloaded and installed 2.1.9b5, an upgrade from 2.1.8. Haven't yet created any meshes with it, other than one throw-away test run using both PyFFi and the new Optimizer. I had planned to redo my Oblivion meshes BSA (currently optimized with 2.1.7) as well as finally optimizing several mods that have thus far escaped treatment. But I'm seeing reports that 2.1.9b5 might be a bit "glitchy". Is that indeed the case? If so, what is the most recent fully stable and glitch-free version?

Thanks in advance!
User avatar
Carlos Rojas
 
Posts: 3391
Joined: Thu Aug 16, 2007 11:19 am

Post » Tue May 11, 2010 7:52 am

Just a quick heads up that the havok issue (collisions being removed in many if not most static objects) has been identified & fixed in 2.1.9-beta6 (just released). Thanks to Ulrim for reporting this on the tracker (indeed, please use it, I may not keep track of everything that is posted here!!):

https://github.com/amorilia/pyffi/downloads
User avatar
Alexander Lee
 
Posts: 3481
Joined: Sun Nov 04, 2007 9:30 pm

Post » Tue May 11, 2010 5:22 pm

PyFFI 2.1.9 beta 6 has fixed the collision problems I was having. At least the MOPP based static meshes are fine. I will need to continue testing with other kinds of static collision meshes.
User avatar
Connie Thomas
 
Posts: 3362
Joined: Sun Nov 19, 2006 9:58 am

Post » Tue May 11, 2010 1:47 am

OP update:

-2.1.9b6 changelog

-PyFFI Thread 4 has returned to the depths of Oblivion.

-Inclusion of a new headlines section so you can easily see the latest updates and so I don't have to waste space making posts like these. Why didn't I think of this before? :facepalm:
User avatar
RObert loVes MOmmy
 
Posts: 3432
Joined: Fri Dec 08, 2006 10:12 am

Post » Tue May 11, 2010 2:19 am

Not sure if this is an harmful issue or just an annoyance.

PyFFi version affected: v219b6 (and earlier today with 219b5)

ISSUE: "Mopper.exe has stopped working"

This is a new one for me. I ran in to it this evening first while optimizing my "TamRes - meshes.bsa" with the combo of PyFFi 219b6 and the new PyFFi Optimizer. The Optimizer batch file had been running for some time when two "Mopper.exe has stopped working" message boxes appeared in rapid succession. At that time CPU usage dropped from 100% to basically nothing. I clicked CLOSE to terminate the two stopped instances of Mopper. PyFFi immediately resumed processing. CPU usage returned to 100%. Mopper kicked in as needed for the remainder of the PyFFi run. So far as I know I received no error messages (though I'm not sure where I'd look for them.)

Thinking the problem might lie with the PyFFi Optimizer, I copied the untreated TamRes meshes into standard PyFFi and ran oblivion_optimize.ini. It behaved exactly like my earlier attempt with the Optimizer. Two "Mopper.exe has stopped working" messages appeared in what seems the same spots they had before, bringing progress to a standstill. Additionally, the PyFFi progress window stopped advancing. And as before, as soon as I terminated those two stopped mopper instances PyFFi kicked back in and completed its run, with mopper kicking in as needed.

This happened to me once early today with PyFFi v219b5 (and the Optimizer). I could not repeat the issue then so assumed it was a fluke.

Has any damage been done by mopper ceasing to function during a run? Have files been corrupted? I didn't see anything in the progress window to indicate corruption or an error of any sort. Is this a known ignorable issue with the latest versions of PyFFi, annoying but harmless? Is there a fix?

I again state that this occurs with PyFFi 219b6 (and before that 219b5), and those versions of PyFFi in conjunction with the Optimizer. Python is 2.6.6.

Any insight on this is most appreciated.

Thanks in advance!
-Decrepit-
User avatar
Marcus Jordan
 
Posts: 3474
Joined: Fri Jun 29, 2007 1:16 am

Post » Tue May 11, 2010 1:47 am

Sounds like something I ran into with another mod once where an instance of a bad mesh caused mopper.exe to fail. Never found the cause because I couldn't find anything in the output log to track it down with.

The ones where mopper.exe failed won't have been written so you shouldn't need to worry about it. Although narrowing down which meshes are causing this may still be useful.
User avatar
dell
 
Posts: 3452
Joined: Sat Mar 24, 2007 2:58 am

Post » Tue May 11, 2010 8:00 am

@andalaybay: Thanks for confirming!

@decripit: As Arthmoor says, it is nothing to worry deeply about. Likely the mopper is failing internally for some reason regarding the geometry. The core functionallity of the mopper is closed source (unfortunately) so chances are fairly high that it cannot be fixed. But if you could post the nif or nifs that are affected by this on the bug tracker, maybe a workaround is possible. Link to bug tracker FYI:

http://sourceforge.net/tracker/?atid=968813&group_id=199269&func=browse
User avatar
David John Hunter
 
Posts: 3376
Joined: Sun May 13, 2007 8:24 am

Post » Tue May 11, 2010 3:45 am

@andalaybay: Thanks for confirming!

@decripit: As Arthmoor says, it is nothing to worry deeply about. Likely the mopper is failing internally for some reason regarding the geometry. The core functionallity of the mopper is closed source (unfortunately) so chances are fairly high that it cannot be fixed. But if you could post the nif or nifs that are affected by this on the bug tracker, maybe a workaround is possible. Link to bug tracker FYI:

http://sourceforge.net/tracker/?atid=968813&group_id=199269&func=browse

That I will. Only, in this case I wouldn't know what to send. The BSA I optimized was quite large, and as Arthmoor mentions there's apparently no log entry to handily track down the mesh(es) in question. I'll try and remember the other mod that caused mopper to fail. It's mesh folder was much smaller. Might not be that difficult to pinpoint the culprit there.

If it means anything to you ... and it doesn't to me ... here's http://i307.photobucket.com/albums/nn303/Decrepit_Waste/ESIV%20errors/MopperExeStopped1.jpg at that point. Sorry for the somewhat blurry image. I needed to reduce image size to conserve online storage space.
User avatar
Erin S
 
Posts: 3416
Joined: Sat Jul 29, 2006 2:06 pm

Post » Tue May 11, 2010 1:32 pm

I'd check the number of PyFFI jobs you were running. Then, take note of the last number of meshes optimized within that window. Perhaps it helps track down to where to start testing.

Then, isolate those meshes individualy and run PyFFI on them to check for the culprit. Much easier to narrow down.
User avatar
Rusty Billiot
 
Posts: 3431
Joined: Sat Sep 22, 2007 10:22 pm

Post » Tue May 11, 2010 3:29 am

Hello,

I have some questions in regards to PyFFI and was wondering if someone could help me out. I am new to PyFFI and decided I wanted to try optimizing my meshes. I wanted to try the "Cool technique (courtesy of Arthmoor and H2ODK)", but I am bit confused on what I need to do with it. I understand I need to edit my Oblivion_optimize.ini file and replace the current spell with the new one, so I did. That seemed easy enough, but then I kept reading and I seen that I needed to create another .ini file called Oblivion_optimize2.ini and put the other spell that is listed under the "Cool technique (courtesy of Arthmoor and H2ODK)", so I did that as well.

The next part under the "Cool technique (courtesy of Arthmoor and H2ODK)" is where I am having trouble since I don't understand what I need to do. What do the lines below mean and what do I need to do with them?

When I do an optimization, I use two batch files. #1:

"C:\Python26\python.exe" "C:\Python26\Scripts\niftoaster.py" --noninteractive --ini-file="C:\Program Files (x86)\PyFFI\utilities\toaster\default.ini" --ini-file="C:\Program Files (x86)\PyFFI\utilities\toaster\oblivion_optimize.ini" --dest-dir= --source-dir= --pause --overwrite "%1" > "PyFFILog.txt"

#2:

"C:\Python26\python.exe" "C:\Python26\Scripts\niftoaster.py" --noninteractive --ini-file="C:\Program Files (x86)\PyFFI\utilities\toaster\default.ini" --ini-file="C:\Program Files (x86)\PyFFI\utilities\toaster\oblivion_optimize2.ini" --dest-dir= --source-dir= --pause --overwrite "%1" > "PyFFILog.txt"

The first one takes ages to complete, the second usually goes by fairly quick because it's only looking for a specific set of info. What these do though is add a much more robust extra layer of "just in case" so that it REALLY won't touch any helmets and/or hairs and such.

I also have one other used specifically for cleaning any loose _far.nif files:

"C:\Python26\python.exe" "C:\Python26\Scripts\niftoaster.py" --noninteractive opt_cleanfarnif --dest-dir= --source-dir= --pause --overwrite "%1" > "PyFFILog.txt"

You may also notice that they all drop a log file when they're done. That's handy for quick-scanning for the word error or warning to see if any bugs cropped up. It also reduces the amount of spam from the handler messages.



Another question I have is, do I need to keep uninstalling PyFFI and reinstalling it when the updates come out? From what it looks like, there are always a bunch of updates coming out.

Lastly, will I have to keep having to reoptimize my meshes everytime a new update or version comes for PyFFI?

Any help is much appreciated and thank you in advance!
User avatar
Scott
 
Posts: 3385
Joined: Fri Nov 30, 2007 2:59 am

Next

Return to IV - Oblivion