Thread updated on March 7th, 2011

http://www.gamesas.com/index.php?/topic/1123756-inforelz-pyffi-python-file-format-interface/

Quick Links

http://www.gamesas.com/index.php?/topic/1123756-inforelz-pyffi-python-file-format-interface/
http://www.gamesas.com/index.php?/topic/1112284-inforelz-pyffi-python-file-format-interface/
http://www.gamesas.com/index.php?/topic/1061938-info-relzpyffi-python-file-format-interface/
http://www.gamesas.com/?showtopic=1014947
http://www.gamesas.com/?showtopic=976098

PyFFI Thread 3 - too old and has been purged.
PyFFI Thread 2 - too old and has been purged.
PyFFI Thread 1 - too old and has been purged.

http://pyffi.sourceforge.net/
http://www.python.org/download/releases/2.6.6/ (required to Run PyFFI)
https://github.com/amorilia/pyffi/downloads
http://sourceforge.net/tracker/?group_id=199269 (sub-trackers for bugs, requests etc.)
https://github.com/amorilia/pyffi (ie the earliest that you can see changes/additions that will be in the next release is here)
http://www.tesnexus.com/downloads/file.php?id=37463 A set of INI files and batch scripts designed for complete bulk mesh optimizations.
http://www.tesnexus.com/downloads/file.php?id=36882 For those who cannot or wish not to run the full optimization process.



ESSENTIAL INFO FOR ALL USERS


What is PyFFI?

PyFFI stands for Python File Format Interface and was written almost entirely by Amorilia (Contributors: wz, taarna23, seith, tazpn, PacificMorrowind)

Latest Changelog

Release 2.1.9b4 [in development] (6 March 2011)
==============================

* Bugfix in patch_recursive_make/apply scripts for "deep" folder layouts (fixes issue #3193914 reported by xjdhdr).


Release 2.1.8 (4 February 2011)
===============================

* Quickhull bugfix for precision argument in degenerate cases
(issue #3163949, fix contributed by liuhuanjim013).

* Fixed issue with detecting box shapes on degenerate collision meshes
(fixes issue #3145104, reported by Gratis_monsta).

* Ensure that enum has valid default value.

* Added CStreamableAssetData for Divinity 2 (reported by pertinen,
niftools issue #3164929).

* NiPixelData.save_as_dds fourcc flag bugfix.

* Added Rockstar .dir format (used in Bully SE).

* Added Rockstar .dir/.img unpack and pack scripts (only tested for Bully SE).


FAQ

Q - Is it safe?
A - Yes; many Oblivion players and modders have run it on thousands of nifs with no problems. There is of course the potential for problems but if so as long as you made a backup it is relatively painless to revert some meshes. Of course there are some meshes that you should NOT run it on (any with egms is the main type) - if you do run it on them you will likely run into problems.

Q - Is it safe to PyFFI meshes that are already PyFFI'd?
A - While performance benefits will be negligible, yes, it is perfectly safe to run meshes that are already PyFFI'd through PyFFI again.

Q - Will running a mesh that had errors introduced into it by a previous version of PyFFI have those errors repaired by running a later version on the bad mesh?
A - No, PyFFI cannot repair any damage that has already been done, and you will need to run the vanilla version of the mesh through PyFFI again. Make sure you keep a backup of the default meshes BSA in case you run into any problem meshes.

Q - Why is PyFFI increasing the file size of my meshes?! Shouldn’t optimization be reducing their size?
A - As of PyFFI 2.1.6, much of the optimization work now occurs through triangulation (improves performance, but increases file size) rather than stripification (improves performance while decreasing file size).

Q - Does PyFFI support muli-core processors?
A - Yes, PyFFI will utilize multiple processor cores and hyperthreading if they are available.

Q – Does PyFFI support the 64bit version of Python?
A – No, PyFFI does not support 64bit Python, and using it with the 64bit version can cause numerous problems.

Q - Is there a visual quality loss?
A - No, PyFFI removes duplicate and empty data from the nif file.

Q- What's the latest stable version of PyFFI?
A- As of February 4th, 2011, 2.1.8

Q- What's the latest in-development version of PyFFI?
A- As of March 6th, 2011, 2.1.9b4

Q- Are there any meshes that I should not put through PyFFI?
A- Yes, there are some meshes that can cause problems.

- any with egms (most open helmets and hair)
- problems have been previously reported with both bows and creatures but both are fixed
- a problem has been reported on the meshes/dungeons/misc/rootshavoc01.nif (and related 02,03,04,05,06 an 07 nifs), by Sproaty. Under investigation

Q - I got a weird bunch of test in the command line/in the log file after running PyFFI, what does it mean?
A - Usually that would be in the form of something like this:

*** TEST FAILED ON C:\PyFFI_Op\XMSGroundFleshContainer01.nif ****** If you were running a script that came with PyFFI, then ****** please report this as a bug (include the file) on ****** [src="http://forums.bethsoft.com/index.php?/topic/1158842-inforelz-pyffi-python-file-format-interface/http://sourceforge.net/tracker/?group_id=199269"]http://sourceforge.net/tracker/?group_id=199269[/url] ***

It means that there is something about the nif that PyFFI doesn't know what to do with or something like that, or sometimes a bug in the spell. If you get that it means the file will not have been optimized and will have been left unchanged so no fixing is necessary to play with it. However if you can report that here or on the sourceforge tracker if you have a sourceforge account, it can probably be fixed. To be fixed we would need
A: to know what spell you were running
B: a copy of the nif or kf.

What is of interest for Oblivion modding right now is what it can do with nif/kf files:
It contains a script called niftoaster.py which is an python wrapper script that runs other scripts on files fed to it. Niftoaster can do a whole host of things to nifs/kfs including: optimizing models, fixing many different model errors, changing the collision type/material of nifs etc. etc. It is also highly extendable to do whatever you want.

One of the most important spells (the name for each of the scripts) is Optimize; it actually runs a whole bunch of the fixing and optimizing spells in tandem; for mod users it is really the only one you're interested in; and for a lot of mod makers it is also the only one your interested in.


Why Should You Care?

Really you don't have to; however, if you make meshes you should, do retexturing etc. or if you want to try and squeeze a little bit of extra performance out of Oblivion.

With the latest versions of PyFFI, performance increases can be quite substantial, so running optimization spells on your meshes (including the vanilla meshes packed in the BSA Archives) is highly recommended.

Installation

Notice for Windows Vista/7 Users: Do not install Python or PyFFI to the Program Files directory. UAC will make life miserable for you if you do.

Prerequisite:http://www.python.org/download/releases/2.6.6/
Currently only the 32bit version is supported.

(If using the Installer version)
-Download https://github.com/downloads/amorilia/pyffi/PyFFI-2.1.7-beta4.e5134e7.win32.exe (Windows Installer)
-Run the Installer and follow the onscreen instructions.

(If using the manual version)
-Download https://github.com/downloads/amorilia/pyffi/PyFFI-2.1.7-beta4.e5134e7.zip (.ZIP Archive).
-Uninstall any previous versions of PyFFI
-Unzip the package to wherever you wish for it to go (Windows Vista/7 users, see notice above).
-Run the following command from the command prompt window:

cd C:\(Navigate to your PyFFI directory here)C:\python setup.py install

->Example:

cd C:\PyFFIpython setup.py install

(You need to have Python registered in your system’s environment variables for the python command to function properly)
-If you have setuptools installed, you can streamline the process by running this command line:

easy_install –U PyFFI

Uninstallation

(If using the Installer version)
-Run the PyFFI_Uninstall.exe

(If using the manual version)
-Delete your PyFFI Directory
-Delete the PyFFI folders found in the Lib\site-packages folders in your PyFFI 2.5 and Python 2.6 directories.

Using PyFFI

The scripts PyFFI uses are termed “Spells.” PyFFI provides several spells for you to use, the most important of which is “Optimize”. There are several other spells for mod developers to use, but the layman user will likely only be interested in the optimize spell.

There are far too many spells available to go over in one forum thread, so only the most important ones will be described here:

Optimize
Modify_makefarnif
Modify_texturepath
Modify_collisiontype
Modify_Scaleanimationtime
Modify_reverseanimation


INFO CONCERNING MOD USERS/PLAYERS AND DEVELOPERS


Spell: Optimize

This is where the brunt of PyFFI is located. The optimize spell removes duplicate/ empty/ uneeded data from your meshes, decreasing the workload for your GPU to render them. This can result in noticeable performance increases if you are to, say, PyFFI the entire vanilla mesh set.

How To:

First, you will need the meshes that you wish to optimize. These are usually found loose or packed into a Bethesda Softworks Archive (.BSA) file. If they are loose, it is best that you find the package for the mod whom the meshes belong to and extract them to a folder outside the Oblivion directory. It is not recommended to run PyFFI on the meshes folder already inside the data folder for Oblivion.

If the meshes are packed in a BSA, use a tool like Oblivion Mod Manager or BSA Commander to unpack the archive to a separate folder.

Once you have the meshes you wish to optimize, you have two options:

1. Right-click the meshes folder and select Optimize with PyFFI
2. Place the meshes into your PyFFI’s in folder (ex: C:\Program Files\PyFFI\utilities\toaster\in) and run PyFFI from the oblivion_optimize.ini

The former option optimizes the meshes and overwrites the originals. The latter option writes the new meshes to an \out folder so the original meshes are retained.

Further optimization for _far.NIFs (VWD meshes) can be called with the following batch script:

"C:\Python26\python.exe" "C:\Python26\Scripts\niftoaster.py" --noninteractive opt_cleanfarnif --dest-dir= --source-dir= --pause --overwrite "%1" > "PyFFILog.txt"

Once you are finished, gather up all the meshes into one spot (only really concerns users of option 2). If you the meshes were provided loose, rebuild the mod's package if you intend to use BAIN or OBMM installation so to include the optimized meshes. If you are optimizing a BSA, repack the BSA, and rename it to whatever it has been defined to be in your sArchiveList INI entry (if optimizing the Vanilla BSA) or to ensure your mod's .ESP file will allow the game to find it. Don't forget to reset the BSA timestamps, either, or you may find the vanilla meshes overriding your mod-added meshes!

Example of PyFFI in Action doing the Optimize spell:

reading C:\PyFFI_Op\Weapons\braided\claymorefine.nifchecking for duplicate source textureschecking for duplicate propertiesremoving duplicate and empty childrenoptimizing geometriesoptimizing block 'Claymore:0'removing duplicate vertices(num vertices was 732 and is now 615)recalculating strips(strip length was 1518 and is now 1188)(average strip length is 20.917508)stitching strips (using 103 stitches)recalculating tangent spacewriting C:\PyFFI_Op\Weapons\braided\claymorefine.nif...

Known Issues

Take heed that there are several meshes that are incompatible with the optimization process:

-Any hair mesh
-Roothavok meshes (the hanging roots you commonly see in caves)
-Any mesh with a .EGM association (commonly helmets)
-Any mesh with a .TRI association (teeth, eyes, etc.)

Using option 1, therefore, is not recommended unless you are certain none of the above meshes exist in the set you are optimizing. The oblivion_optimize.INI comes with a skip list that will tell PyFFI to skip the problem meshes during the optimization process, and it works pretty well, so if you are, for instance, optimizing the entire vanilla meshes BSA, it is recommended to use option 2.

Watch out for these meshes if you are doing a vanilla mesh optimization:


Their names contain leading spaces that it has been reported by Windows 7 64bit users get truncated if you move the meshes folder around after you’ve finished the optimization process. If you try to use those meshes in that state in the game, the game will report them as missing. Windows 7 64bit users should DEFINITELY watch out for this, and Windows 7 32bit and Windows Vista (all versions) users should be aware of the issue. Windows XP users are safe from this.

If you've used PyFFI before, previous to v.1.0.0 of PyFFI did not triangulate NifTriStrips with lengths less than 10. Unfortunatly this causes Oblivion issues and while cause frame stuttering, and on low performance machines can virtually halt the game.

In addition, the 3dsMax NifTools exporter bloats the exported nif file with duplicate everything (or very nearly duplicate everything). This is a known problem, which I believe is due to 3dsMax and not directly to the NifTools.

Blender and Maya do not suffer this problem.

Though that does not mean that duplicate data will not appear in meshes made with Blender or Maya.

Cool technique (courtesy of Arthmoor and H2ODK):




PyFFI Patcher

One of the many hurdles involved with the usage of PyFFI was the inability to redistribute the optimized meshes for the game for copyright reasons. This is now no longer the case with PyFFI Patcher. This separate utility allows for the user to instead download a patch file containing only the differences between the vanilla meshes and the PyFFI optimized meshes and simply apply the mesh patches to the old meshes.

The advantage? It takes a lot less time and is much less involved for a user to install these PyFFI Patches rather than run the PyFFI optimization process themselves.

The downside? These patch files will need to be redistributed every time PyFFI is updated to improve the optimization process for Oblivion's meshes. This will mean increased time spent on downloads.

Is it worth it? If you do not have the time available to invest in learning how to use PyFFI, or you wish to perform a one-off optimization of your meshes, then yes, it is worth it. If you wish to remain on top of the latest developments for PyFFI, however, it may be easier to simply run the optimization process yourself.


INFO CONCERNING MOD DEVELOPERS ONLY


Spell: Modify_makefarnif

This spell will generate VWD meshes (_far.NIFs). It is called with the following batch script/command prompt:

 “C:\Python26\python.exe” “C:\Python26\Scripts\niftoaster.py” modify_makefarnif  --only _far.nif$ --dest-dir= --source-dir= --pause --overwrite "%1" > "PyFFILog.txt” 

Spell: Modify_collisiontype

This lets you quickly (in seconds) and accurately and on as many nifs as you want at once, change to collision type to anything that is supported (and easy to add more if desired). A real boon for Max users until the nifscripts properly support non-static collision export. The format for calling it is batch file/command prompt of:

cd C:\Python26python.exe Scripts/niftoaster.py modify_collisiontype -a static "C:\meshes" (replacing as need the mesh path and the python path, and static with the type of collision desired)

Spell: Modify_scaleanimationtime

As usual real quick and easy method to: slow down or speed up animations. will change the speed of basically any animation type (not tested on bsplines yet) called by:

cd C:\Python26python.exe Scripts/niftoaster.py modify_scaleaniamtiontime -a 0.6 "C:\meshes" (replacing as need the mesh path and the python path and the number with the desired multiplier)

Spell: Modify_reverseanimation

As usual real quick and easy method to do something... in this case reverse an animation - ie make a close animation from an open animation for a door etc.

cd C:\Python26python.exe Scripts/niftoaster.py modify_reverseanimation "C:\meshes" (unteste for bspline type animations so far)

Spell: Modify_texturepath

lets you quickly (in seconds or minutes depending on number of nifs) and accurately and on as many nifs as you want at once, change the source texture paths from x/x/wall01.dds to /whateveryouwant/wall01.dds - a real boon when it comes to retexturing a whole tileset. The format for calling it is batch file/command prompt of:

cd C:\Python26python.exe Scripts/niftoaster.py modify_texturepath -a textures/PM/BloodyAyleid/ "C:\meshes"

(replacing as need the mesh path and the python path, and the texture path with the desired texture path).

Now I'm sure some of you are thinking hmm none of those do anything for me but I sure could use something that batch changed X... but I don't know any Python oh well. Never fear; if you want a script you can always ask - not promising it will get done, but significantly more likely than without asking (and if you don't believe me just ask DaMage).

As always if something isn't clear ask and someone knowledgeable will try to answer.

It's been more than a week since the previous thread hit ceiling, so I thought I'd do the TES modding community a small favor by reopening this thread and updating it with a bit of new info.

New in OP:

-Links to PyFFI 2.1.7b5 Downloads (Installer and Archive version).

-New FAQ entry for latest developmental releases.

-New FAQ entry for running PyFFI on already PyFFI optimized meshes.

-Complete changelog for 2.1.7b5 (and later, as further releases are made)

Man, there's already a 2.1.7 version out? I just finished running all my meshes through version 2.1.6. Have there been an major bugfixes or performance gains since then, and is it worth it to optimize everything again with the newest version?

2.1.7b5 seems to have a better ATVR than 2.1.6 and supposedly fixes a (rather inconspicuous) seam issue with rat underbellies and bear snouts. Most changes seem to be for FO3 and FNV. Here's the full changelog:

I am new to PyFFI and it isn't doing anything I'm sure I'm doing somthing wrong but I'm not sure what. I copy the nif files (from in this case meshes\architecture\leyawiin) to the "in" folder in C:\Program Files (x86)\PyFFI\utilities\toaster right click on the "oblivion_optimize.ini" select "Run PyFFI" and get this.

pyffi.toaster:INFO:toasting with 2 threads in chunks of 64 files pyffi.toaster:INFO:Finished. Press enter...  

No changes have beenmade and my "out" folder is still empty.

I have both PyFFI and Pylon2.6.6 installed. PyFFI is in (ProgramFiles(x86)\PyFFI) and Playon is installed in (LocalDisk(C:)\Pylon26) So What am I doing wrong.

Added to OP. Thanks for grabbing that.

I worked it out myself it turns out I had UAC on.

The file linked in the OP for the latest development version is beta 4, not beta 5. Beta 5 is the latest.

The problem is that you installed PyFFI in the "Program Files" folder. I recommend you turn UAC back on then reinstall PyFFI elsewhere.

Whoopsie. :blush: :facepalm: :toughninja:

Fixed.

Just a quick heads up that 2.1.7 is released:

http://sourceforge.net/projects/pyffi/files/pyffi/2.1.7/PyFFI-2.1.7.8fb5680.win32.exe/download

It is virtually identical to beta5, so no need to reinstall if you already have that one (unless you also want the docs). Please post any issues on the tracker.

https://sourceforge.net/tracker/?group_id=199269&atid=968813

I'm aware that there are a few more outstanding bugs - thanks to those who posted them on the tracker - they will be addressed in the next release.

I seem to be experiencing a problem with this pyffi thing. Everytime i place some model files in the "In" folder, and run the script/spell/whatever, the files in the "Out" folder are larger than the files inside the "In" folder. Am i doing something wrong?

[edit]
This seems only to be happening with armors that are non-shield...
[/edit]

Larger files aren't necessarily a bad thing It's working as intended™

Even if like 95% of the files are larger than before pyffi-ing?

Yup. PyFFI is designed to optimize not reduce, and optimization can mean converting non-optimized compressed stuff into perfectly optimized (but slightly larger) stuff

k thx for the answer! i was worried a bit that stuff went horribly wrong

Not sure if this is the correct thread for this question. Anyway, I can't really pyffi the vanilla meshes on my current ancient computer as it takes forever (more than 48 hours at least). On the other hand, that in itself shows that I probably would be helped a lot by optimized meshes so it's kind of a Catch 22. So I thought I'd use the http://www.tesnexus.com/downloads/file.php?id=19911 but it's rather old and I noticed in the OP that it "won't work for GOTY". I have the original game plus SI patched to 1.2.0.416. Will it work with that? Or is there some alternative?

I have just recenltly PyFFI-ed my Oblivion - Meshes.bsa using 2.1.6 on a P4 3.0 GHz HT (took 3 days on and off ) and the difference is just stunning. Performance-wise, it felt like upgrading my rig.

Now I'm reporting about two glitches that have happened as a result of the process, and perhaps somebody could point out how to tell the meshes responsible from the screenies and I would re-PyFFI them again with 2.1.7 and see if it fixes that.

The glitches aren't the kind that causes CTD or other unintended behavior, but are only visual glitches.

The first http://img820.imageshack.us/img820/6648/screenshot8jq.jpg is taken from an Oblivion realm, and as you can see the glitch or the seam is visible.

The second http://img824.imageshack.us/img824/7859/screenshot9iz.jpg is taken in Anvil behind Morvayn's Pacemakers IIRC. I have encircled in red line the glitch that is not apparently visible from the picture unless looked closely upon. Unlike the first, this glitch manifests itself by a blinking line above. And looking at it in various angles reveal the line clearly.

Could somebody advise a way on how to track down the nifs affected from the shots?

I also pyffied my stuff with 2.1.6. I can check about your pic #2 if I see the same thing as you. I'm in Anvil now.

I too have the original game with SI on top, and at first I tried with that and it worked. However, after reading all about the PyFFI-cation process and the benefits redoing the process with new versions, given that those files hosted from that link were made more than two years ago now, I did the process myself, and took several days, but it was worth it.

Thanks Tom. Awaiting your confirmation then. From the previous thread I also noticed the seam on an Ebony bow, and after comparing 2.1.6 vs. 2.1.7 PyFFI-cation of the vanilla mesh, I can say that the 2.1.7 is much better than the 2.1.6. However, the vanilla mesh doesn't show the seam as noticeable as the one from 2.1.7

Hmm, wish I'd seen this thread a bit sooner rather than starting my own.
All of my meshes have been optimized using PyFFI 2.1.5. I'd like to reoptimize them using 2.1.7, rather than going back to the original meshes, since in several cases I no longer have the backups :banghead:

Any recommendations as to whether this is a good idea/bad idea? I note that the 2.1.6 and 2.1.7 changelogs list a number of bug fixes.

Personally, I would recommend just letting your computer take the 48+ hours it needs. I know your pain - I have a midlevel machine and reoptimizing takes a couple days for me as well - but the performance difference is well worthwhile.
Two other tips I can give: first, make sure you have the latest version of Python 2 installed (and get the 64-bit version if you have a 64-bit machine). Second, I've noticed that PyFFI seems only able to use one core on my machine. Since I have a dual core, I can run two instances of PyFFI (on two different sets of meshes) and everything gets done faster!

Read the F.A.Qs, please.

I saw the answer you were referring to:

And I understand that re-optimizing meshes that were already optimized is unlikely to introduce any new errors.

However, my specific question has to do with the bugfixes noted in PyFFI's changelog for versions 2.1.6 and 2.1.7. That is, if one of the errors fixed by 2.1.6 or 2.1.7 occurred while 2.1.5 was optimizing, will the current version be able to correct that earlier error?
For example, this is noted in 2.1.6's changelog: "Bugfix in vertex map reordering (fixes most nifs reported in issue #3071616)". If that happened in one of my 2.1.5-optimized meshes, what will 2.1.7 do with that mesh? Would it be able to fix any such error, or do I need to go back to the original mesh?

(And yes, I'll readily admit that could be a stupid question. I'm still trying to learn PyFFI's details.)