http://www.gamesas.com/index.php?/topic/1158842-inforelz-pyffi-python-file-format-interface/
Post Update Headlines
PyFFI version 2.1.9 has gone stable.
Quick Links
http://www.gamesas.com/index.php?/topic/1158842-inforelz-pyffi-python-file-format-interface/
http://www.gamesas.com/index.php?/topic/1123756-inforelz-pyffi-python-file-format-interface/
http://www.gamesas.com/index.php?/topic/1112284-inforelz-pyffi-python-file-format-interface/
http://www.gamesas.com/index.php?/topic/1061938-info-relzpyffi-python-file-format-interface/
http://www.gamesas.com/?showtopic=1014947
PyFFI Thread 4 - too old and has been purged.
PyFFI Thread 3 - too old and has been purged.
PyFFI Thread 2 - too old and has been purged.
PyFFI Thread 1 - too old and has been purged.
http://pyffi.sourceforge.net/
http://www.python.org/download/releases/2.6.6/ (required to Run PyFFI)
https://github.com/amorilia/pyffi/downloads
http://sourceforge.net/tracker/?group_id=199269 (sub-trackers for bugs, requests etc.)
https://github.com/amorilia/pyffi (ie the earliest that you can see changes/additions that will be in the next release is here)
http://www.tesnexus.com/downloads/file.php?id=37463 A set of INI files and batch scripts designed for complete bulk mesh optimizations. Official support thread http://www.gamesas.com/index.php?/topic/1172029-relz-oblivion-pyffi-optimization-kit/
http://www.tesnexus.com/downloads/file.php?id=37548 For those who cannot or wish not to run the full optimization process. Official support thread http://www.gamesas.com/index.php?/topic/1173353-oblivion-pyffi-binary-patch-kit/
ESSENTIAL INFO FOR ALL USERS
What is PyFFI?
PyFFI stands for Python File Format Interface and was written almost entirely by Amorilia (Contributors: wz, taarna23, seith, tazpn, PacificMorrowind)
Latest Changelog
https://github.com/amorilia/pyffi/blob/python25/CHANGELOG.rst
Release 2.1.9 (26 March 2011)
==============================
* Added new script to make and apply patches (functionality is identical to and OnmyojiOmn's and jonwd7's pyffi patcher scripts, but it is written in Python to work on all platforms).
* New fix_emptyskeletonroots spell (automatically included in the optimize spell) to fix issues with nifs that do not have their NiSkinInstance skeleton root set (fixes #3174085, reported by xjdhdr).
* Fixed logging issue on Windows platform with multithreading enabled (fixes issue #3174339, reported by xjdhdr).
* Fixed QSkope shortcut issue when path contains spaces (reported by Brumbek).
* Skip terminal chars in mopper output (fixes issues with running mopper under wine).
* Bugfix in patch_recursive_make/apply scripts for "deep" folder layouts (fixes issue #3193914, reported by xjdhdr).
* Do not pack collisions in OL_CLUTTER (fixes issue #3194017 reported by Gratis_monsta).
* Fixed issue with skipping terminal chars in mopper output on Windows platform (fixes issue #3205569, reported and diagnosed by ulrim).
* Disabled copy in patch_recursive_apply script (see issue #3219744, suggested by ulrim).
* Pass additional arguments of patch_recursive_apply/make to the patch command (see issue #3219744, suggested by ulrim).
* Sane default settings for bhkRigidBody unknowns to ensure that constraints behave properly (contributed by Koniption).
NOTICE: If anyone has performed mesh optimizations on a previous version of 2.1.9, you will need to redo the optimization process, as crucial fixes to possible disappearing clutter items have been made.
FAQ
Q - Is it safe?
A - Yes; many Oblivion players and modders have run it on thousands of nifs with no problems. There is of course the potential for problems but if so as long as you made a backup it is relatively painless to revert some meshes. Of course there are some meshes that you should NOT run it on (any with egms is the main type) - if you do run it on them you will likely run into problems.
Q - Is it safe to PyFFI meshes that are already PyFFI'd?
A - While performance benefits will be negligible, yes, it is perfectly safe to run meshes that are already PyFFI'd through PyFFI again.
Q - Will running a mesh that had errors introduced into it by a previous version of PyFFI have those errors repaired by running a later version on the bad mesh?
A - No, PyFFI cannot repair any damage that has already been done, and you will need to run the vanilla version of the mesh through PyFFI again. Make sure you keep a backup of the default meshes BSA in case you run into any problem meshes.
Q - Why is PyFFI increasing the file size of my meshes?! Shouldn’t optimization be reducing their size?
A - As of PyFFI 2.1.6, much of the optimization work now occurs through triangulation (improves performance, but increases file size) rather than stripification (improves performance while decreasing file size).
Q - Does PyFFI support muli-core processors?
A - Yes, PyFFI will utilize multiple processor cores and hyperthreading if they are available.
Q – Does PyFFI support the 64bit version of Python?
A – No, PyFFI does not officially support 64bit Python, and using it with the 64bit version can cause numerous problems.
Q - Is there a visual quality loss?
A - No, PyFFI removes duplicate and empty data from the nif file.
Q- What's the latest stable version of PyFFI?
A- As of March 26th, 2011, 2.1.9
Q- What's the latest in-development version of PyFFI?
A- Currently none, the latest Dev. version has gone stable.
Q- Are there any meshes that I should not put through PyFFI?
A- Yes, there are some meshes that can cause problems.
- Any meshes with FaceGen files attached to them (that is, any NIFs that are accompanied by an EGM or TRI mesh). These should not be optimized, or visual anomalies (E.G. tears in the model) might result.
- The Roothavok meshes. It is currently unknown why these don't optimize properly.
Q - I got a weird bunch of test in the command line/in the log file after running PyFFI, what does it mean?
A - Usually that would be in the form of something like this:
*** TEST FAILED ON C:\PyFFI_Op\XMSGroundFleshContainer01.nif ****** If you were running a script that came with PyFFI, then ****** please report this as a bug (include the file) on ****** [src="http://forums.bethsoft.com/index.php?/topic/1172342-relzwipzpyffi-python-file-format-interface/http://sourceforge.net/tracker/?group_id=199269"]http://sourceforge.net/tracker/?group_id=199269[/url] ***
Q - Why am I not getting the option to run PyFFI when I right click the Oblivion_Optimize.INI?
A - You have a third party text editor (E.G. Notepad++) that is interfering with the drop-down options menu and preventing this option from displaying. This is fixed by uninstalling the interfering text editor.
It means that there is something about the nif that PyFFI doesn't know what to do with or something like that, or sometimes a bug in the spell. If you get that it means the file will not have been optimized and will have been left unchanged so no fixing is necessary to play with it. However if you can report that here or on the sourceforge tracker if you have a sourceforge account, it can probably be fixed. To be fixed we would need
A: to know what spell you were running
B: a copy of the nif or kf.
What is of interest for Oblivion modding right now is what it can do with nif/kf files:
It contains a script called niftoaster.py which is an python wrapper script that runs other scripts on files fed to it. Niftoaster can do a whole host of things to nifs/kfs including: optimizing models, fixing many different model errors, changing the collision type/material of nifs etc. etc. It is also highly extendable to do whatever you want.
One of the most important spells (the name for each of the scripts) is Optimize; it actually runs a whole bunch of the fixing and optimizing spells in tandem; for mod users it is really the only one you're interested in; and for a lot of mod makers it is also the only one your interested in.
Why Should You Care?
Really you don't have to; however, if you make meshes you should, do retexturing etc. or if you want to try and squeeze a little bit of extra performance out of Oblivion.
With the latest versions of PyFFI, performance increases can be quite substantial, so running optimization spells on your meshes (including the vanilla meshes packed in the BSA Archives) is highly recommended.
Installation
Notice for Windows Vista/7 Users: Do not install Python or PyFFI to the Program Files directory. UAC will make life miserable for you if you do.
Prerequisite:http://www.python.org/download/releases/2.6.6/
Currently only the 32bit version is supported.
(If using the Installer version)
-Download https://github.com/amorilia/pyffi/downloads (Windows Installer)
-Run the Installer and follow the onscreen instructions.
(If using the manual version)
-Download https://github.com/amorilia/pyffi/downloads (.ZIP Archive).
-Uninstall any previous versions of PyFFI
-Unzip the package to wherever you wish for it to go (Windows Vista/7 users, see notice above).
-Run the following command from the command prompt window:
cd C:\(Navigate to your PyFFI directory here)C:\python setup.py install
->Example:
cd C:\PyFFIpython setup.py install
(You need to have Python registered in your system’s environment variables for the python command to function properly)
-If you have setuptools installed, you can streamline the process by running this command line:
easy_install –U PyFFI
Uninstallation
(If using the Installer version)
-Run the PyFFI_Uninstall.exe
(If using the manual version)
-Delete your PyFFI Directory
-Delete the PyFFI folders found in the Lib\site-packages folders in your PyFFI 2.5 and Python 2.6 directories.
Using PyFFI
The scripts PyFFI uses are termed “Spells.” PyFFI provides several spells for you to use, the most important of which is “Optimize”. There are several other spells for mod developers to use, but the layman user will likely only be interested in the optimize spell.
There are far too many spells available to go over in one forum thread, so only the most important ones will be described here:
Optimize
Modify_makefarnif
Modify_texturepath
Modify_collisiontype
Modify_Scaleanimationtime
Modify_reverseanimation
INFO CONCERNING MOD USERS/PLAYERS AND DEVELOPERS
Spell: Optimize
This is where the brunt of PyFFI is located. The optimize spell removes duplicate/ empty/ uneeded data from your meshes, decreasing the workload for your GPU to render them. This can result in noticeable performance increases if you are to, say, PyFFI the entire vanilla mesh set.
How To:
http://tesivpositive.animolious.com/index.php?page=pyffi but for those who may just need a refresher, I'll give you the quick version:
First, you will need the meshes that you wish to optimize. These are usually found loose or packed into a Bethesda Softworks Archive (.BSA) file. If they are loose, it is best that you find the package for the mod whom the meshes belong to and extract them to a folder outside the Oblivion directory. It is not recommended to run PyFFI on the meshes folder already inside the data folder for Oblivion.
If the meshes are packed in a BSA, use a tool like Oblivion Mod Manager or BSA Commander to unpack the archive to a separate folder.
Once you have the meshes you wish to optimize, you have two options:
1. Right-click the meshes folder and select Optimize with PyFFI
2. Place the meshes into your PyFFI’s in folder (ex: C:\Program Files\PyFFI\utilities\toaster\in) and run PyFFI from the oblivion_optimize.ini
The former option optimizes the meshes and overwrites the originals. The latter option writes the new meshes to an \out folder so the original meshes are retained.
Further optimization for _far.NIFs (VWD meshes) can be called with the following batch script:
"C:\Python26\python.exe" "C:\Python26\Scripts\niftoaster.py" --noninteractive opt_cleanfarnif --dest-dir= --source-dir= --pause --overwrite "%1" > "PyFFILog.txt"
Once you are finished, gather up all the meshes into one spot (only really concerns users of option 2). If you the meshes were provided loose, rebuild the mod's package if you intend to use BAIN or OBMM installation so to include the optimized meshes. If you are optimizing a BSA, repack the BSA, and rename it to whatever it has been defined to be in your sArchiveList INI entry (if optimizing the Vanilla BSA) or to ensure your mod's .ESP file will allow the game to find it. Don't forget to reset the BSA timestamps, either, or you may find the vanilla meshes overriding your mod-added meshes!
Example of PyFFI in Action doing the Optimize spell:
reading C:\PyFFI_Op\Weapons\braided\claymorefine.nifchecking for duplicate source textureschecking for duplicate propertiesremoving duplicate and empty childrenoptimizing geometriesoptimizing block 'Claymore:0'removing duplicate vertices(num vertices was 732 and is now 615)recalculating strips(strip length was 1518 and is now 1188)(average strip length is 20.917508)stitching strips (using 103 stitches)recalculating tangent spacewriting C:\PyFFI_Op\Weapons\braided\claymorefine.nif...
Known Issues
Take heed that there are several meshes that are incompatible with the optimization process:
-Any hair mesh
-Roothavok meshes (the hanging roots you commonly see in caves)
-Any mesh with a .EGM association (commonly helmets)
-Any mesh with a .TRI association (teeth, eyes, etc.)
Using option 1, therefore, is not recommended unless you are certain none of the above meshes exist in the set you are optimizing. The oblivion_optimize.INI comes with a skip list that will tell PyFFI to skip the problem meshes during the optimization process, and it works pretty well, so if you are, for instance, optimizing the entire vanilla meshes BSA, it is recommended to use option 2.
Watch out for these meshes if you are doing a vanilla mesh optimization:
meshes\dungeons\caves\exterior\ centrancegoldcoastsm01.nif
meshes\dungeons\caves\exterior\ centrancegreatforestsm02.nif
meshes\dungeons\caves\exterior\ centrancemtnsnowlg01.nif
meshes\dungeons\caves\exterior\ centrancemtnsnowlg02.nif
meshes\dungeons\caves\exterior\ centrancemtnsnowlg03.nif
meshes\dungeons\caves\exterior\ centrancemtnsnowsm01.nif
meshes\dungeons\caves\exterior\ centrancemtnsnowsm02.nif
meshes\dungeons\caves\exterior\ centrancemtnsnowsm03.nif
meshes\dungeons\caves\exterior\ centrancerockmosssm01.nif
meshes\dungeons\caves\exterior\ centrancerockybeachsm01.nif
meshes\dungeons\caves\exterior\ centrancewestwsm01.nif
meshes\dungeons\caves\exterior\ centrancewestwsm02.nif
Their names contain leading spaces that it has been reported by Windows 7 64bit users get truncated if you move the meshes folder around after you’ve finished the optimization process. If you try to use those meshes in that state in the game, the game will report them as missing. Windows 7 64bit users should DEFINITELY watch out for this, and Windows 7 32bit and Windows Vista (all versions) users should be aware of the issue. Windows XP users are safe from this.
If you've used PyFFI before, previous to v.1.0.0 of PyFFI did not triangulate NifTriStrips with lengths less than 10. Unfortunately this causes Oblivion issues and while cause frame stuttering, and on low performance machines can virtually halt the game.
In addition, the 3dsMax NifTools exporter bloats the exported nif file with duplicate everything (or very nearly duplicate everything). This is a known problem, which I believe is due to 3dsMax and not directly to the NifTools.
Blender and Maya do not suffer this problem.
Though that does not mean that duplicate data will not appear in meshes made with Blender or Maya.
Cool technique (courtesy of Arthmoor and H2ODK):
[main]; run optimize spellspell = optimize[options]; any patterns of files that should be skipped, being as conservative as possible; (without quotes, separate different regular expressions by a space); at the moment:; - skipping hair nifs (vertex ordering!); - skipping roothavok nifs (not sure why, investigating); - skipping any nif that is known to have an egm or tri associated with it; find . -name "*.egm" -or -name "*.tri" | sed 'sX.*/XXg' | sed 'sX.tri$XXg' | sed 'sX.egm$XXg' | sed 'y/ABCDEFGHIJKLMNOPQRSTUVWXYZ/abcdefghijklmnopqrstuvwxyz/' | sort | uniq | xargs; (not necessary if all egm files are included, such as in vanilla Oblivion,; but some mods only include nifs without egm files, and this makes sure; these cases are handled as well); note: you can safely remove this list if you are sure that you are; optimizing fully extracted folders onlyskip = (?i)(?
That nasty looking skip sequence at the end was something H2ODK came up with and works well, but it needs a slightly altered process to be sure everything gets done. That's where oblivion_optimize2.ini file comes in:[main]; run optimize spellspell = optimize[options]; any patterns of files that should be skipped; regex expressions; only run on files with these patternsonly = (?i)gnd[.]nif$ (?i)meshes.clutter
When I do an optimization, I use two batch files. #1:"C:\Python26\python.exe" "C:\Python26\Scripts\niftoaster.py" --noninteractive --ini-file="C:\Program Files (x86)\PyFFI\utilities\toaster\default.ini" --ini-file="C:\Program Files (x86)\PyFFI\utilities\toaster\oblivion_optimize.ini" --dest-dir= --source-dir= --pause --overwrite "%1" > "PyFFILog.txt"
#2:"C:\Python26\python.exe" "C:\Python26\Scripts\niftoaster.py" --noninteractive --ini-file="C:\Program Files (x86)\PyFFI\utilities\toaster\default.ini" --ini-file="C:\Program Files (x86)\PyFFI\utilities\toaster\oblivion_optimize2.ini" --dest-dir= --source-dir= --pause --overwrite "%1" > "PyFFILog.txt"
The first one takes ages to complete, the second usually goes by fairly quick because it's only looking for a specific set of info. What these do though is add a much more robust extra layer of "just in case" so that it REALLY won't touch any helmets and/or hairs and such.
I also have one other used specifically for cleaning any loose _far.nif files:"C:\Python26\python.exe" "C:\Python26\Scripts\niftoaster.py" --noninteractive opt_cleanfarnif --dest-dir= --source-dir= --pause --overwrite "%1" > "PyFFILog.txt"
You may also notice that they all drop a log file when they're done. That's handy for quick-scanning for the word error or warning to see if any bugs cropped up. It also reduces the amount of spam from the handler messages.
PyFFI Patcher
http://www.tesnexus.com/downloads/file.php?id=37548
One of the many hurdles involved with the usage of PyFFI was the inability to redistribute the optimized meshes for the game for copyright reasons. This is now no longer the case with PyFFI Patchers. This separate utility allows for the user to instead download a patch file containing only the differences between the vanilla meshes and the PyFFI optimized meshes and simply apply the mesh patches to the old meshes.
The advantage? It takes a lot less time and is much less involved for a user to install these PyFFI Patches rather than run the PyFFI optimization process themselves.
The downside? These patch files will need to be redistributed every time PyFFI is updated to improve the optimization process for Oblivion's meshes. This will mean increased time spent on downloads.
Is it worth it? If you do not have the time available to invest in learning how to use PyFFI, or you wish to perform a one-off optimization of your meshes, then yes, it is worth it. If you wish to remain on top of the latest developments for PyFFI, however, it may be easier to simply run the optimization process yourself.
A fairly recent addition in the form of the PyFFI Binary Patch KIt has now been released, as well. This tool not only makes it easier to obtain the PyFFI optimized meshes by allowing the users to patch their own meshes, but it also makes it easier to create your own binary patch files, too. Here's how it is done:
With 'Oblivion - PyFFI Binary Patch Kit' you'd have to:
- place the original meshes inside 'in'
place the optimized meshes inside 'out'
open 'patcher.py' with a text editor and at the end change
scan(s_dir, s_list)#scan(t_dir, t_list)scan(p_dir, p_list)#create_patch(s_list, t_list)apply_patch(s_list, p_list)to
scan(s_dir, s_list)scan(t_dir, t_list)scan(p_dir, p_list)create_patch(s_list, t_list)#apply_patch(s_list, p_list)run the patcher and patch files are created inside 'patch' folder, recursively.
Many thanks to ulrim for his work on this.
INFO CONCERNING MOD DEVELOPERS ONLY
Spell: Modify_makefarnif
This spell will generate VWD meshes (_far.NIFs). It is called with the following batch script/command prompt:
“C:\Python26\python.exe” “C:\Python26\Scripts\niftoaster.py” modify_makefarnif --only _far.nif$ --dest-dir= --source-dir= --pause --overwrite "%1" > "PyFFILog.txt”
Spell: Modify_collisiontype
This lets you quickly (in seconds) and accurately and on as many nifs as you want at once, change to collision type to anything that is supported (and easy to add more if desired). A real boon for Max users until the nifscripts properly support non-static collision export. The format for calling it is batch file/command prompt of:
cd C:\Python26python.exe Scripts/niftoaster.py modify_collisiontype -a static "C:\meshes" (replacing as need the mesh path and the python path, and static with the type of collision desired)
Spell: Modify_scaleanimationtime
As usual real quick and easy method to: slow down or speed up animations. will change the speed of basically any animation type (not tested on bsplines yet) called by:
cd C:\Python26python.exe Scripts/niftoaster.py modify_scaleaniamtiontime -a 0.6 "C:\meshes" (replacing as need the mesh path and the python path and the number with the desired multiplier)
Spell: Modify_reverseanimation
As usual real quick and easy method to do something... in this case reverse an animation - ie make a close animation from an open animation for a door etc.
cd C:\Python26python.exe Scripts/niftoaster.py modify_reverseanimation "C:\meshes" (unteste for bspline type animations so far)
Spell: Modify_texturepath
lets you quickly (in seconds or minutes depending on number of nifs) and accurately and on as many nifs as you want at once, change the source texture paths from x/x/wall01.dds to /whateveryouwant/wall01.dds - a real boon when it comes to retexturing a whole tileset. The format for calling it is batch file/command prompt of:
cd C:\Python26python.exe Scripts/niftoaster.py modify_texturepath -a textures/PM/BloodyAyleid/ "C:\meshes"
(replacing as need the mesh path and the python path, and the texture path with the desired texture path).
Now I'm sure some of you are thinking hmm none of those do anything for me but I sure could use something that batch changed X... but I don't know any Python oh well. Never fear; if you want a script you can always ask - not promising it will get done, but significantly more likely than without asking (and if you don't believe me just ask DaMage).
As always if something isn't clear ask and someone knowledgeable will try to answer.