[INFO] [RELZ]PyFFI - Python File Format Interface

Post » Sun Apr 25, 2010 11:37 am

To continue the PyFFiing discussions, a new thread is needed... not as needed as I thought since there is one still active (when last one got locked a couple weeks ago I was planning on starting a nice brand new one... but doing so anyways. However this thread a tad different... it is bit official (done by a part time PyFFI programmer (that would be me) and with Amorilia's (the head programmer and creator of PyFFI) go ahead), and has a bunch more info that hasn't really been on the radar of modders before.

Quick Links

    http://www.gamesas.com/bgsforums/index.php?showtopic=1014947
    http://www.gamesas.com/bgsforums/index.php?showtopic=976098
    http://www.gamesas.com/bgsforums/index.php?showtopic=890860
    PyFFI thread 2 - too old and has been purged.
    PyFFI thread 1 - too old and has been purged.
    http://pyffi.sourceforge.net/
    http://www.python.org/download/releases/2.6.4/ (required to Run PyFFI)
    http://sourceforge.net/project/showfiles.php?group_id=199269
    http://sourceforge.net/tracker/?group_id=199269 sub-trackers for bugs, requests etc.
    http://github.com/amorilia/pyffi (ie the earliest that you can see changes/additions that will be in the next release is here)
    http://github.com/PacificMorrowind/pyffi (has my changes before review, so may be less stable/have more bugs... but also have the earliest of my changes/additions)
    Just for mod users wanting to patch their meshes with very little chance of errors (but only a some meshes supported):
      http://www.tesnexus.com/downloads/file.php?id=19899(Thanks to OnmyojiOmn creating the patching method!)
      http://www.tesnexus.com/downloads/file.php?id=19911 (patches for vanilla, OOO HF, Operation Optimization, etc.) (won't work for GOTY)

    For mod users/makers wanting to optimize all their meshes:
      Get PyFFI
      then consider getting: http://www.tesnexus.com/downloads/file.php?id=25423 (not required but has nice features such as making PyFFI run as multiple processes, there by taking proper advantage of the CPU power of most computers, optimized files are saved to a separate folder that you select, etc)


What is PyFFI?
PyFFI stands for Python File Format Interface and was written almost entirely by Amorilia (Contributors: wz, taarna23, seith, tazpn, PacificMorrowind)

What is of interest for Oblivion modding right now is what it can do with nif/kf files:
It contains a script called niftoaster.py which is an python wrapper script that runs other scripts on files fed to it. Niftoaster can do a whole host of things to nifs/kfs including: optimizing models, fixing many different model errors, changing the collision type/material of nifs etc. etc. It is also highly extendable to do whatever you want.
One of the most important spells (the name for each of the scripts) is Optimize; it actually runs a whole bunch of the fixing and optimizing spells in tandem; for mod users it is really the only one you're interested in; and for a lot of mod makers it is also the only one your interested in.

Why Should You Care?
Really you don't have to; however, if you make meshes you should, do retexturing etc. or if you want to try and squeeze a little bit of extra performance out of Oblivion.

Instructions for mod users (and modders)
- download & install http://www.python.org/download/releases/2.6.4/
- download & install http://sourceforge.net/project/showfiles.php?group_id=199269 latest version (as of December 14, 2009 it is 2.0.5)
- Optionally download & install http://www.tesnexus.com/downloads/file.php?id=25423

Then either run http://www.tesnexus.com/downloads/file.php?id=25423 if using it or just copy the meshes you want to optimize to a folder outside your oblivion dir, and from that folder delete any files that have an egm (which would be files in a folder that are in the nature of helmet.nif and another by the name of helmet.egm), most open helmets and hair have egms. Then either right click that folder and select 'Optimize with PyFFI' or (easier) create a txt file, past this code:
@echo off:\cd C:\%pythonpath%@echo "Processing folder 1 of 1"python.exe Scripts/niftoaster.py optimize --noninteractive "C:\path\to\your\meshes" > "C:\path\to\your\meshes\log.txt"Pause
in, and replace the %pythonpath% with the path to your python directory (by default it is C:/Python26), replace "C:\path\to\your\meshes" with the real path to your meshes. and save it as *whateveryouwant*.bat and double click the file. Remember, make backups just in case (unless using http://www.tesnexus.com/downloads/file.php?id=25423, in which case it will automatically make backups). With the batch process/command line/right click folder->Optimize call, PyFFI will open any .nif, .kf, .kfa, .nifcache files, but unless it finds optimizable data types (geometries) it won't do anything to them - it will just open and close without saving. If there are optimizable data types, whether or not any optimization is possible the file will be saved (overwriting the original).

Example of PyFFI in Action doing the Optimize spell:
reading C:\PyFFI_Op\Weapons\braided\claymorefine.nifchecking for duplicate source textureschecking for duplicate propertiesremoving duplicate and empty childrenoptimizing geometriesoptimizing block 'Claymore:0'  removing duplicate vertices  (num vertices was 732 and is now 615)  recalculating strips  (strip length was 1518 and is now 1188)  (average strip length is 20.917508)  stitching strips (using 103 stitches)  recalculating tangent space  writing C:\PyFFI_Op\Weapons\braided\claymorefine.nif...


If you've used PyFFI before, previous to v.1.0.0 of PyFFI did not triangulate NifTriStrips with lengths less than 10. Unfortunatly this causes Oblivion issues and while cause frame stuttering, and on low performance machines can virtually halt the game.

In addition, the 3dsMax NifTools exporter bloats the exported nif file with duplicate everything (or very nearly duplicate everything). This is a known problem, which I believe is due to 3dsMax and not directly to the NifTools.

Blender and Maya do not suffer this problem.
Though that does not mean that duplicate data will not appear in meshes made with Blender or Maya.[/size]



FAQ?
Q - Is it safe?
A - Yes; many Oblivion players and modders have run it on thousands of nifs with no problems. There is of course the potential for problems but if so as long as you made a backup it is relatively painless to revert some meshes. Of course there are some meshes that you should NOT run it on (any with egms is the main type) - if you do run it on them you will likely run into problems.

Q - Is there a visual quality loss?
A - No, PyFFI removes duplicate and empty data from the nif file.

Q- What's the latest version of PyFFI?
A- As at December 13th 2009, 2.0.5 (and 2.0.6 is in development)

Q- Are there any meshes that I should not put through PyFFI?
A- Yes, there are some meshes that can cause problems.
    - any with egms (most open helmets and hair)
    - problems have been previously reported with both bows and creatures but both are fixed
    - a problem has been reported on the meshes/dungeons/misc/rootshavoc01.nif (and related 02,03,04,05,06 an 07 nifs), by Sproaty. [i]Under investigation[i]


Q- I got a weird bunch of test in the command line/in the log file after running PyFFI, what does it mean?
A - Usually that would be in the form of something like this:
*** TEST FAILED ON C:\PyFFI_Op\XMSGroundFleshContainer01.nif	****** If you were running a script that came with PyFFI, then		****** please report this as a bug (include the file) on			****** [url="http://sourceforge.net/tracker/?group_id=199269"]http://sourceforge.net/tracker/?group_id=199269[/url]			***

It means that there is something about the nif that PyFFI doesn't know what to do with or something like that, or sometimes a bug in the spell. If you get that it means the file will not have been optimized and will have been left unchanged so no fixing is necessary to play with it. However if you can report that here or on the sourceforge tracker if you have a sourceforge account, it can probably be fixed. To be fixed we would need a) to know what spell you were running B) a copy of the nif or kf.

Instructions for moddelers/animators/other modders
- download & install http://www.python.org/download/releases/2.6.4/
- download & install http://sourceforge.net/project/showfiles.php?group_id=199269 latest version (as of December 14, 2009 it is 2.0.5)

You're doing what I consider the funnest part of Oblivion... and with PyFFI, you get some nice tools to make your work faster (in some cases saving dozens of hours), easier, and less headache.

As well as the Optimize spell (which you should run, though if you use Blender to make your meshes you almost certainly don't need to) you also get:
    modify_texturepath (for retexturers more than modelers)
    modify_collisiontype
    modify_scaleanimationtime (in version 2.0.6)
    modify_reverseanimation (in version 2.0.6, only my fork currently)
    and lots of other stuff but those are some of most important that aren't run by Optimize.

and

modify_collisiontype: lets you quickly (in seconds) and accurately and on as many nifs as you want at once, change to collision type to anything that is supported (and easy to add more if desired). A real boon for Max users until the nifscripts properly support non-static collision export. The format for calling it is batch file/command prompt of:
cd C:\Python26python.exe Scripts/niftoaster.py modify_collisiontype -a static "C:\meshes"
(replacing as need the mesh path and the python path, and static with the type of collision desired)

modify_scaleanimationtime: as usual real quick and easy method to: slow down or speed up animations. will change the speed of basically any animation type (not tested on bsplines yet) called by:
cd C:\Python26python.exe Scripts/niftoaster.py modify_scaleaniamtiontime -a 0.6 "C:\meshes"
(replacing as need the mesh path and the python path and the number with the desired multiplier)

modify_reverseanimations: as usual real quick and easy method to do something... in this case reverse an animation - ie make a close animation from an open animation for a door etc.
cd C:\Python26python.exe Scripts/niftoaster.py modify_reverseanimation "C:\meshes"
(unteste for bspline type animations so far)

modify_texturepath: lets you quickly (in seconds or minutes depending on number of nifs) and accurately and on as many nifs as you want at once, change the source texture paths from x/x/wall01.dds to /whateveryouwant/wall01.dds - a real boon when it comes to retexturing a whole tileset. The format for calling it is batch file/command prompt of:
cd C:\Python26python.exe Scripts/niftoaster.py modify_texturepath -a textures/PM/BloodyAyleid/ "C:\meshes"
(replacing as need the mesh path and the python path, and the texture path with the desired texture path).

Now I'm sure some of you are thinking hmm none of those do anything for me but I sure could use something that batch changed X... but I don't know any Python oh well. Never fear; if you want a script you can always ask - not promising it will get done, but significantly more likely than without asking (and if you don't believe me just ask DaMage).

As always if something isn't clear ask and someone knowledgeable will try to answer.
Pacific Morrowind
User avatar
Ann Church
 
Posts: 3450
Joined: Sat Jul 29, 2006 7:41 pm

Post » Sun Apr 25, 2010 10:39 pm

thanks for the guide, I was planning on PyFFi'ing my meshes, but I was too confused. I think I can get it now, thanks!
:thumbsup:
User avatar
Nicole Elocin
 
Posts: 3390
Joined: Sun Apr 15, 2007 9:12 am

Post » Sun Apr 25, 2010 2:06 pm

thanks for the guide, I was planning on PyFFi'ing my meshes, but I was too confused. I think I can get it now, thanks!
:thumbsup:

great, and as Aellis said, I'd would reccomend you get the automator if you have any confusion - a smidge easier if you don't have a fair bit of computer management skills.
Okay, so I've run PyFFI on my meshes/rocks folder, and now I can't open up some of the rock meshes anymore with nifskope. Does this mean that something has gone wrong with the PyFFI'ing? I use the latest PyFFI and Python 2.6.4.

that does not sound good... definitely possible that they got borked... post em (original, and potentailly borked and details of what spells you applied - ie did you specificy Optimize or something else?) here or on the http://sourceforge.net/tracker/?atid=968813&group_id=199269&func=browse if you have a sourceforge account/don't mind creating one... (and what version of PyFFI... an update has been released pretty recently).
Pacifci Morrowind
User avatar
Amy Smith
 
Posts: 3339
Joined: Mon Feb 05, 2007 10:04 pm

Post » Sun Apr 25, 2010 2:39 pm

excellent thread with excellent info :)
User avatar
trisha punch
 
Posts: 3410
Joined: Thu Jul 13, 2006 5:38 am

Post » Sun Apr 25, 2010 11:16 am

To revive the discussion from yesterday that we had in another thread about pyffi-ing with the Alpha5 tool, I made a second run of the meshes form Oblivion vanilla and watched closely what happend in the process. I post some the key details here:

My operating system is still WinXP 32-bit.
I have the GOTY version of the game.
I downloaded and installed PyFFI version 2.0.5
I unpacked the Oblivion - Meshes.BSA to a backup folder with OBMM (BSA Browser function)
From the "meshes" subfolder" I selected 8 folders for pyffi-fication: Architecture, clutter, dungeons, furniture, landscape, plants, rocks and sky.
I copied those to another folder, lets call it Source.
The file count in the Source folder as per Explorer was 6063 files in 152 folders.
Out of these, Alpha5 would treat 6018 and skip 45 files.
The 45 files were skipped according to criteria I set in the exclusion list in the .ini file. These were eliminated by Alpha5 during a quick scan right at the beginning of the program. No problem with these ones.

Now comes the more intricate part:
Out of the remaining 6018 files selected for treatment, Alpha 5 would skip 57 files saying "PyFFI Fail...". I have condensed the output_log file to identify the 57 failed files:

Pyffi Fail Files Oblivion Vanilla game:Pyffi Fail - \architecture\anvil\anvildooruc01.nifPyffi Fail - \architecture\anvil\anvildooruc02.nifPyffi Fail - \architecture\basemantsections\ungrdltraphingedoor.nifPyffi Fail - \architecture\castle\anvil\ancastlecomplete01_far.nifPyffi Fail - \architecture\castle\kvatch\kvatch  castle int hallway01.nifPyffi Fail - \architecture\chorrol\interior\chorrolhousemiddle06interior.nifPyffi Fail - \architecture\chorrol\interior\chorrolhousemiddle07interior.nifPyffi Fail - \architecture\farmhouse\farmhouseinterior02.nifPyffi Fail - \architecture\imperialcity\interior\ictempleoneinteriordome01.nifPyffi Fail - \architecture\kvatch\kvatchcorner02.nifPyffi Fail - \clutter\candlefat03.nifPyffi Fail - \clutter\candleskinny04.nifPyffi Fail - \clutter\floorplane01.nifPyffi Fail - \clutter\middlecandlestickfloor01evil.nifPyffi Fail - \clutter\middlecandlestickfloor02.nifPyffi Fail - \clutter\middlecandlestickfloor03.nifPyffi Fail - \clutter\stonepedastellarge01.nifPyffi Fail - \clutter\farm\handscythe01.nifPyffi Fail - \clutter\farm\oar01.nifPyffi Fail - \clutter\lowerclass\lowerpainting01.nifPyffi Fail - \clutter\lowerclass\lowerpainting02.nifPyffi Fail - \clutter\lowerclass\lowerpainting03.nifPyffi Fail - \clutter\lowerclass\lowerpainting04.nifPyffi Fail - \clutter\lowerclass\lowerpainting07.nifPyffi Fail - \clutter\lowerclass\lowerpainting08.nifPyffi Fail - \clutter\lowerclass\lowerpainting09.nifPyffi Fail - \dungeons\ayleidruins\interior\arwelkydclusterfx01.nifPyffi Fail - \landscape\lod\60.-32.-32.32.nifPyffi Fail - \landscape\lod\60.-32.-64.32.nifPyffi Fail - \landscape\lod\60.-32.-96.32.nifPyffi Fail - \landscape\lod\60.-32.00.32.nifPyffi Fail - \landscape\lod\60.-32.32.32.nifPyffi Fail - \landscape\lod\60.-32.64.32.nifPyffi Fail - \landscape\lod\60.-64.-64.32.nifPyffi Fail - \landscape\lod\60.-64.00.32.nifPyffi Fail - \landscape\lod\60.-64.32.32.nifPyffi Fail - \landscape\lod\60.-64.64.32.nifPyffi Fail - \landscape\lod\60.-96.-64.32.nifPyffi Fail - \landscape\lod\60.-96.00.32.nifPyffi Fail - \landscape\lod\60.-96.32.32.nifPyffi Fail - \landscape\lod\60.-96.64.32.nifPyffi Fail - \landscape\lod\60.00.-32.32.nifPyffi Fail - \landscape\lod\60.00.-64.32.nifPyffi Fail - \landscape\lod\60.00.00.32.nifPyffi Fail - \landscape\lod\60.00.32.32.nifPyffi Fail - \landscape\lod\60.00.64.32.nifPyffi Fail - \landscape\lod\60.32.-32.32.nifPyffi Fail - \landscape\lod\60.32.00.32.nifPyffi Fail - \landscape\lod\60.32.32.32.nifPyffi Fail - \landscape\lod\60.32.64.32.nifPyffi Fail - \landscape\lod\60.64.-32.32.nifPyffi Fail - \landscape\lod\60.32.-64.32.nifPyffi Fail - \landscape\lod\60.64.-64.32.nifPyffi Fail - \landscape\lod\60.64.00.32.nifPyffi Fail - \landscape\lod\60.64.32.32.nifPyffi Fail - \landscape\lod\60.64.64.32.nifPyffi Fail - \landscape\lod\629780.00.00.32.nif


Subsequently I have not tested it on all, but the majority of these I *COULD* pyffi with the right-click method. I just tested that, but for the purpose of actually using the pyffied meshes in the game, I retained only all the "unfailed", "good" files.

So my question would be why would Alpha5 skip these particular 57 files? Are these possibly .nif files with the ominous "EGM association"? How would I know that any .nif had such an association prior to pyffi-ing it?

I also pyffied the vanilla SI meshes (from disc 2 of GOTY). From these, there were only 2 files that failed. I have not attempted to pyffi them with right-click-optimize.

Pyffi Fail Files Shivering Isle:Pyffi Fail - \dungeons\ruininteriors\hallsnarrow\ruinshallnxdeadendcollapse.nifPyffi Fail - \landscape\lod\40728.00.00.32.nif


Other than the issue with those failed files, I am very happy with how the pyffi-ing went and how the tool worked. Very handy indeed, and much less complicated that I thought! Have you guys any ideas or insights on the "failed" files? Thanks very much for your help.
User avatar
Maria Leon
 
Posts: 3413
Joined: Tue Aug 14, 2007 12:39 am

Post » Sun Apr 25, 2010 3:54 pm

To revive the discussion from yesterday that we had in another thread about pyffi-ing with the Alpha5 tool, I made a second run of the meshes form Oblivion vanilla and watched closely what happend in the process. I post some the key details here:

sounds good and normal
Now comes the more intricate part:
Out of the remaining 6018 files selected for treatment, Alpha 5 would skip 57 files saying "PyFFI Fail...". I have condensed the output_log file to identify the 57 failed files:

that means that the automator detected that the size difference was very little (in most cases anyways... haven't double checked all of those); therefor next to no or no performance improvement and why run something through any program for no gain... given that there is a (very small) chance of an error... and anyways not really any point.
So my question would be why would Alpha5 skip these particular 57 files? Are these possibly .nif files with the ominous "EGM association"? How would I know that any .nif had such an association prior to pyffi-ing it?

egm association is very easy to find... is there an egm withe same name as the nif (ie helmet.nif and helmet.egm) in the same folder? if so there is an EGM association if not there isn't... really simple. Also EGMs are only going to be found for open helms/hairs/some mod added clothes that use that kind of slot (ie hats), and some body parts...
I also pyffied the vanilla SI meshes (from disc 2 of GOTY). From these, there were only 2 files that failed. I have not attempted to pyffi them with right-click-optimize.

Pyffi Fail Files Shivering Isle:Pyffi Fail - \dungeons\ruininteriors\hallsnarrow\ruinshallnxdeadendcollapse.nifPyffi Fail - \landscape\lod\40728.00.00.32.nif


Other than the issue with those failed files, I am very happy with how the pyffi-ing went and how the tool worked. Very handy indeed, and much less complicated that I thought! Have you guys any ideas or insights on the "failed" files? Thanks very much for your help.

same thing on those probably (haven't double checked though). Thanks for the nice review/discussion... always good.
Pacific Morrowind
User avatar
Beth Belcher
 
Posts: 3393
Joined: Tue Jun 13, 2006 1:39 pm

Post » Sun Apr 25, 2010 3:34 pm

Does Pyffi's performance increase come from reduced stress on your GPU, or your CPU? I ask because I'm trying to squeeze as much performance out of my old processor as possible to delay the need to upgrade as fast as possible.

Ravin
User avatar
Chris Duncan
 
Posts: 3471
Joined: Sun Jun 24, 2007 2:31 am

Post » Sun Apr 25, 2010 10:15 pm

Deleted, double post. Sorry for that!
User avatar
Avril Louise
 
Posts: 3408
Joined: Thu Jun 15, 2006 10:37 pm

Post » Sun Apr 25, 2010 10:54 pm

Hello PacificMorrowind! Sorry I saw your reply only today!! I have forgotten to check back to the thread!

I still wish you a very Happy New Year!!

Now to business:

First off let me say that whatever Alpha5 and PyFFI were doing to my meshes, it works just beautifully in-game. That is I don't notice anything abnormal, all looks graphically perfectly normal. I even became more adventurous and pyffied all sorts of other meshes like clothes and armor and weapons. Specifically:

- I did a full PyFFI-ing of the http://www.tesnexus.com/downloads/file.php?id=28975, that is *all* meshes in there. The only skipping criteria I set in Alpha5 was "Hair".
- I also did a full PyFFI on the http://www.tesnexus.com/downloads/file.php?id=26551 but only on the OOO version of that. Again only skipping "Hair".

As I say, these are working perfectly. (But I still have unpyffied backups just in case...)

As to those 57 "failed" files I wrote about earlier: Whatever it is, based on my positive experience as per above, it must be no-issue after all. Having said that, I am not quite convinced that they "failed" because of no size reduction. When I look at the output_log file, Alpha5 distinguishes between "No copy....." and "PyFFI Failed..." The "No copy" files are the ones where no reduction was achieved, the failed one's I would think must be because of something else. I am fairly certain of that because the "No copy" file list is considerably longer than the "failed" one. Out of the some 6000 vanilla meshes files pyffied, only some 4500 (IIRC) made it into my new "Actually PyFFied Meshes" archive. The some 1500 meshes that didn't make it were the 57 "failed" and all the "No copies". So "No copies" were about 1400+ (I would need to check back the exact numbers but you get the magnitude).

Again, I think we should not lose any sleep about the issue because the end result is what it is - and it is great as far as I can tell. Maybe only the author of Alpha5 would be able to tell us why they "failed" and what it means - if it means anything. Maybe it is something of an old mesh issue. You remember the one error I reported on the Sourceforge bug report page. You guys found out it was some old-format mesh used in an irrelevant spot. Maybe those "failed" files Alpha5 found are similar in nature. But that's really just a speculation on my part.

Overall I am very happy with that whole PyFFI thing, except for the fact that I'm a total layman as to what it really does.
User avatar
Alina loves Alexandra
 
Posts: 3456
Joined: Mon Jan 01, 2007 7:55 pm

Post » Sun Apr 25, 2010 12:49 pm

As to those 57 "failed" files I wrote about earlier: Whatever it is, based on my positive experience as per above, it must be no-issue after all. Having said that, I am not quite convinced that they "failed" because of no size reduction.


I'm happy to share an interesting insight here: pyffi's optimize spell actually used to optimize for file size, however, some modders who experimented with the optimizer in extremis found that my optimizer sometimes created nifs which resulted in lower framerates (the cases were sparse, but certainly not isolated). Fortunately, the cause of this was identified, and a solution too: the bottom line is that some nifs will run better as a trishape instead of a tristrip. However, in this case, the file size will increase.

So, lower file size isn't always higher frame rate, and the author of the automator tool should probably consider removing this check from his tool, because doing so you may actually lose some of the optimizations.

Aside, my plan for one of the next iterations of pyffi is to include its own version of an automator tool, far more configurable than the alpha tool, and written in pure python and integrated into pyffi.
User avatar
Ernesto Salinas
 
Posts: 3399
Joined: Sat Nov 03, 2007 2:19 pm

Post » Sun Apr 25, 2010 12:22 pm

Q- Are there any meshes that I should not put through PyFFI?
A- Yes, there are some meshes that can cause problems.

- problems have been previously reported with both bows and creatures but both are fixed


So based on the above, does this mean that it is safe to scan the creatures folders with PyFFI 2.0.5?
User avatar
Greg Cavaliere
 
Posts: 3514
Joined: Thu Nov 01, 2007 6:31 am

Post » Sun Apr 25, 2010 4:49 pm

So, lower file size isn't always higher frame rate, and the author of the automator tool should probably consider removing this check from his tool, because doing so you may actually lose some of the optimizations.

Aside, my plan for one of the next iterations of pyffi is to include its own version of an automator tool, far more configurable than the alpha tool, and written in pure python and integrated into pyffi.


So are you saying that the right-click optimization method is actually the better one - from an optimization viewpoint - than the Alpha tool? Because with the right-click method (and the tool called "PyFFI Automator" which is different one than Alpha) I did notice that some meshes were bigger after pyffi-ing than they were before. I was wondering if this was actually intended or not.

Integrating a tool right into PyFFI would be the best. I'm looking forward to this! :thumbsup:
User avatar
tannis
 
Posts: 3446
Joined: Sat Dec 09, 2006 11:21 pm

Post » Sun Apr 25, 2010 1:10 pm

So based on the above, does this mean that it is safe to scan the creatures folders with PyFFI 2.0.5?


I would think so - maybe someone who has actually tried can confirm or disconfirm this - however if you still encounter problems please report them.

So are you saying that the right-click optimization method is actually the better one - from an optimization viewpoint - than the Alpha tool? Because with the right-click method (and the tool called "PyFFI Automator" which is different one than Alpha) I did notice that some meshes were bigger after pyffi-ing than they were before. I was wondering if this was actually intended or not.


Indeed, the larger size in such cases is almost always intended (there will be extremely pathalogical cases where it isn't, but it's really not worth bothering about).

I've not written the alpha tool but I think you can configure it so it effectively skips the file size test: simply set the size_test variable to something large enough e.g. 1000. Until it is fully integrated into pyffi, I would recommend you to use the alpha tool, since it brings some very nice features that pyffi is currently missing (but I'm working to change that!).

Aside, I'm rethinking the whole optimization process: currently it ignores the vertex cache completely, and from what I learnt on the web, this can really lead to suboptimal meshes. Triangle ordering turns out to be far more important than strip length, so for one of the next iterations of pyffi (not the one I'm about to release though) you should again see a performance boost. Don't know how much the boost will be. I will ask for testers when it's ready.
User avatar
George PUluse
 
Posts: 3486
Joined: Fri Sep 28, 2007 11:20 pm

Post » Sun Apr 25, 2010 1:23 pm

Well, I'm learing by the minute, very cool, thanks!

I've not written the alpha tool but I think you can configure it so it effectively skips the file size test: simply set the size_test variable to something large enough e.g. 1000. Until it is fully integrated into pyffi, I would recommend you to use the alpha tool, since it brings some very nice features that pyffi is currently missing (but I'm working to change that!).

Actually I was thinking of this workaround myself after reading your first post. :) Any yes, Alpha is definitely to prefer. It has the skip list, performance of Alpha tool is very much better than right-clicking, plus it has that automatic association check for .egm files which right-click does not have.

I have a fairly good rig with E8400 dual core. It can produce pyffi-ed files pretty quickly. If you ever need computing power assistance/testing please let me know.

As to pyffi-ing creatures: Yes I did that with pyffi 2.0.5 and I say results are perfectly fine. But I have not seen each and every creature yet in my game. I play vanilla + OOO. Everything looks perfectly normal after pyffification. In fact I would like to see a screenshot once of a mesh gone wrong by pyffi, as until now, I think I've never encounterd any problems. Are there some screenies of pyffi defects to view somewhere?

Aside, I'm rethinking the whole optimization process: currently it ignores the vertex cache completely, and from what I learnt on the web, this can really lead to suboptimal meshes. Triangle ordering turns out to be far more important than strip length, so for one of the next iterations of pyffi (not the one I'm about to release though) you should again see a performance boost. Don't know how much the boost will be. I will ask for testers when it's ready.

As to that paragraph, you may think that I'm somewhat of a half-baked expert on pyffi. But really I am not :). I understand ZERO of that, I must admit. I have no understanding of the underlying concepts and why one would "pyffi" meshes and all. Acutally I would like to. Is there a primer on "pyffi", like "The PyFFI Primer for 5 year-olds" or something? Yes I understand that it cuts down on some redundancies and thus enhances performance, and I have some experience by now doing it with the tools that there are. But really that's about it.
User avatar
Megan Stabler
 
Posts: 3420
Joined: Mon Sep 18, 2006 2:03 pm

Post » Sun Apr 25, 2010 9:08 pm

As to pyffi-ing creatures: Yes I did that with pyffi 2.0.5 and I say results are perfectly fine. But I have not seen each and every creature yet in my game.


Great! Thanks for confirming.

I play vanilla + OOO. Everything looks perfectly normal after pyffification. In fact I would like to see a screenshot once of a mesh gone wrong by pyffi, as until now, I think I've never encounterd any problems. Are there some screenies of pyffi defects to view somewhere?


Actually I got one just recently (not for pyffi, but for the blender importer/exporter, it came down to the same thing however) from a Fallout 3 modder. It only happens for models that are extremely detailed (and, if this happens it can be fixed :) ). Defects can happen because insufficient digits were taken into account when comparing floats. The solution is simple of course: increase the number of digits. But, if you see a difference, it should only be in *very* small details.

As to that paragraph, you may think that I'm somewhat of a half-baked expert on pyffi. But really I am not :). I understand ZERO of that, I must admit. I have no understanding of the underlying concepts and why one would "pyffi" meshes and all. Acutally I would like to. Is there a primer on "pyffi", like "The PyFFI Primer for 5 year-olds" or something? Yes I understand that it cuts down on some redundancies and thus enhances performance, and I have some experience by now doing it with the tools that there are. But really that's about it.


It's really dead simple, and here's a primer: essentially, pyffi's optimize spell
  • removes redundant data (so far only in the geometry, and in the properties - in theory this could be extended to also reduce the size of animations etc.)
  • finds the best representation - trishape or tristrip - for geometry, in terms of frame rate (not file size)

That's really all there is to it. What I said is that now I'm looking into improving vertex cache usage to get potentially (only testing will reveal how relevant it really is...) yet another boost in frame rates.
User avatar
Rachel Cafferty
 
Posts: 3442
Joined: Thu Jun 22, 2006 1:48 am

Post » Sun Apr 25, 2010 3:03 pm

Amorilia, so if any defects after pyffification should be usually minor, I will look real close at my game in the future. But as I said, certainly looking at the game casually with a normally critical eye, it sure worked very well pyffi-ing creatures and all. The one thing that I have never pyffied until now is "Hair" meshes. Everything else I am pyffi-ing routinely now.

You think it's worth a trial to pyffi hair too? I'll do it if you want me to.

Thanks also for the short primer :) I thought there was a lot more to it, but yeah, great! As I also said, whenever I can be of help in your project with testing and stuff, please let me know. You are always welcome to send me a PM at your leisure. Have a great weekend still! :thumbsup:
User avatar
Amie Mccubbing
 
Posts: 3497
Joined: Thu Aug 31, 2006 11:33 pm

Post » Sun Apr 25, 2010 3:27 pm

A quick heads up that I've just released PyFFI 2.1.0 - a *huge* thanks to PacificMorrowind for all his help making this release reality!

https://sourceforge.net/projects/pyffi/files/

This release features a much improved windows installer for 64 bit systems, compatibility fixes for Python 2.5 users, pep8 updates and improved py3k support (via 2to3), a whole load of new spells (check the changelog), and a whole series of new command line options (akin the Alpha tool for nif optimization, not everything is there yet but more is planned!) for the toasters, including multithreading and a workaround for Python leaking memory when running the toaster for a long time (you'll need --jobs 2 or more for this to work), skipping files based on regular expressions, skipping nifs that have a .tri or .egm attached to them, and saving nifs in a destination folder different from the source folder.

@Tommy_H: I'll take you up on your offer to help testing in the not so distant future. :) At the moment, the optimizer simply skips hair nifs - maybe some day we can process them too (first need to finish up .tri file support) - but for the time being, don't bother trying them since it won't make any difference.
User avatar
+++CAZZY
 
Posts: 3403
Joined: Wed Sep 13, 2006 1:04 pm

Post » Sun Apr 25, 2010 5:36 pm

Oh great, a new version! Sounds interesting, I think I'll do a full re-pyffi of the vanilla .bsa meshes with it.
User avatar
Ria dell
 
Posts: 3430
Joined: Sun Jun 25, 2006 4:03 pm

Post » Mon Apr 26, 2010 1:13 am

Well, nothing has changed in the optimizer, so I don't think it's worth yet redoing all your meshes.

I did some initial testing with my new vertex cache algorithm in vanilla Oblivion (not yet in 2.1.0 or 2.1.1!) but I didn't see any difference (yet). My machine is probably too powerful to notice the difference. Can anyone recommend a good mod for stress testing geometry rendering? :) I.e. something that really kills framerates?
User avatar
~Amy~
 
Posts: 3478
Joined: Sat Aug 12, 2006 5:38 am

Post » Sun Apr 25, 2010 4:55 pm

@amorilia
Is it possible that the spell modify_makefarnif has some problems (not every far- nif) with geometry optimizing (opt_geometry), with the usual optimize spell there is no problem.

pyffi.toaster:INFO:      === V:\test2\Better Cities Resources BSA 4.3.2\00 Core\meshes\architecture\bettercitiesresources\obcnewbravilwatergate01_far.nif ===pyffi.toaster:INFO:        --- modify_delvertexcolor & modify_delalphaprop & modify_delspecularprop & modify_delbsxflags & modify_delstringextradatas & fix_deltangentspace & modify_delcollision & modify_delanimation & modify_disableparallax & modify_texturepathlowres ---pyffi.toaster:INFO:          ~~~ NiNode [BravilWaterGate01_far] ~~~pyffi.toaster:INFO:            ~~~ NiTriShape [BravilWaterGate01_far:3] ~~~pyffi.toaster:INFO:              ~~~ NiTexturingProperty [] ~~~pyffi.toaster:INFO:                ~~~ NiSourceTexture [] ~~~pyffi.toaster:INFO:                  textures\architecture\castle\bravilCastleWall01.dds -> textures\lowres\architecture\castle\bravilCastleWall01.ddspyffi.toaster:INFO:              ~~~ NiTriShapeData [] ~~~pyffi.toaster:INFO:                removing vertex colorspyffi.toaster:INFO:        --- opt_cleanreflists & fix_detachhavoktristripsdata & fix_texturepath & fix_clampmaterialalpha ---pyffi.toaster:INFO:          ~~~ NiNode [BravilWaterGate01_far] ~~~pyffi.toaster:INFO:            ~~~ NiTriShape [BravilWaterGate01_far:3] ~~~pyffi.toaster:INFO:              ~~~ NiTexturingProperty [] ~~~pyffi.toaster:INFO:                ~~~ NiSourceTexture [] ~~~pyffi.toaster:INFO:              ~~~ NiMaterialProperty [Material #47430] ~~~pyffi.toaster:INFO:        --- opt_mergeduplicates ---pyffi.toaster:INFO:          ~~~ NiNode [BravilWaterGate01_far] ~~~pyffi.toaster:INFO:            ~~~ NiTriShape [BravilWaterGate01_far:3] ~~~pyffi.toaster:INFO:              ~~~ NiTexturingProperty [] ~~~pyffi.toaster:INFO:                ~~~ NiSourceTexture [] ~~~pyffi.toaster:INFO:              ~~~ NiMaterialProperty [Material #47430] ~~~pyffi.toaster:INFO:              ~~~ NiTriShapeData [] ~~~pyffi.toaster:INFO:        --- opt_geometry ---pyffi.toaster:INFO:          ~~~ NiNode [BravilWaterGate01_far] ~~~pyffi.toaster:INFO:            ~~~ NiTriShape [BravilWaterGate01_far:3] ~~~pyffi.toaster:INFO:              removing duplicate verticespyffi.toaster:INFO:              (num vertices was 1180 and is now 1104)pyffi.toaster:ERROR:TEST FAILED ON V:\test2\Better Cities Resources BSA 4.3.2\00 Core\meshes\architecture\bettercitiesresources\obcnewbravilwatergate01_far.nifpyffi.toaster:ERROR:If you were running a spell that came with PyFFI, thenpyffi.toaster:ERROR:please report this as a bug (include the file) onpyffi.toaster:ERROR:http://sourceforge.net/tracker/?group_id=199269


pyffi.toaster:INFO:                        === V:\test2\Better Cities Resources BSA 4.3.2\00 Core\meshes\architecture\bettercitiesresources\castle\anvil\ancastlegate01_far.nif ===pyffi.toaster:INFO:                          --- modify_delvertexcolor & modify_delalphaprop & modify_delspecularprop & modify_delbsxflags & modify_delstringextradatas & fix_deltangentspace & modify_delcollision & modify_delanimation & modify_disableparallax & modify_texturepathlowres ---pyffi.toaster:INFO:                            ~~~ NiNode [CastleGate01_far] ~~~pyffi.toaster:INFO:                              ~~~ NiTriShape [CastleGate01_far:1] ~~~pyffi.toaster:INFO:                                ~~~ NiTexturingProperty [] ~~~pyffi.toaster:INFO:                                  ~~~ NiSourceTexture [] ~~~pyffi.toaster:INFO:                                    textures\architecture\castle\CastleTowerLOD01.dds -> textures\lowres\architecture\castle\CastleTowerLOD01.ddspyffi.toaster:INFO:                                ~~~ NiTriShapeData [] ~~~pyffi.toaster:INFO:                                  removing vertex colorspyffi.toaster:INFO:                          --- opt_cleanreflists & fix_detachhavoktristripsdata & fix_texturepath & fix_clampmaterialalpha ---pyffi.toaster:INFO:                            ~~~ NiNode [CastleGate01_far] ~~~pyffi.toaster:INFO:                              ~~~ NiTriShape [CastleGate01_far:1] ~~~pyffi.toaster:INFO:                                ~~~ NiTexturingProperty [] ~~~pyffi.toaster:INFO:                                  ~~~ NiSourceTexture [] ~~~pyffi.toaster:INFO:                                ~~~ NiMaterialProperty [Material #616] ~~~pyffi.toaster:INFO:                          --- opt_mergeduplicates ---pyffi.toaster:INFO:                            ~~~ NiNode [CastleGate01_far] ~~~pyffi.toaster:INFO:                              ~~~ NiTriShape [CastleGate01_far:1] ~~~pyffi.toaster:INFO:                                ~~~ NiTexturingProperty [] ~~~pyffi.toaster:INFO:                                  ~~~ NiSourceTexture [] ~~~pyffi.toaster:INFO:                                ~~~ NiMaterialProperty [Material #616] ~~~pyffi.toaster:INFO:                                ~~~ NiTriShapeData [] ~~~pyffi.toaster:INFO:                          --- opt_geometry ---pyffi.toaster:INFO:                            ~~~ NiNode [CastleGate01_far] ~~~pyffi.toaster:INFO:                              ~~~ NiTriShape [CastleGate01_far:1] ~~~pyffi.toaster:INFO:                                removing duplicate verticespyffi.toaster:INFO:                                (num vertices was 482 and is now 378)pyffi.toaster:ERROR:TEST FAILED ON V:\test2\Better Cities Resources BSA 4.3.2\00 Core\meshes\architecture\bettercitiesresources\castle\anvil\ancastlegate01_far.nifpyffi.toaster:ERROR:If you were running a spell that came with PyFFI, thenpyffi.toaster:ERROR:please report this as a bug (include the file) onpyffi.toaster:ERROR:http://sourceforge.net/tracker/?group_id=199269



Frameratekiller -> Better Cities?
Download: http://www.tesnexus.com/downloads/file.php?id=16513
Interesting place: Imperial City Market District
Additional: buy some items (vegetables, clothes...) place them on the ground
User avatar
sarah simon-rogaume
 
Posts: 3383
Joined: Thu Mar 15, 2007 4:41 am

Post » Sun Apr 25, 2010 1:00 pm

@amorilia
Is it possible that the spell modify_makefarnif has some problems (not every far- nif) with geometry optimizing (opt_geometry), with the usual optimize spell there is no problem.

it should be identical for the opt_geometry section (it just zips over and grabs those spells)... could you pm/link/email (pacificmorrowindgmail[dot]com me to those files ( I know their in BC but BC is too large for my horrible dialup to so far download (and that been trying for months))
Pacific Morrowind
User avatar
Kevin Jay
 
Posts: 3431
Joined: Sun Apr 29, 2007 4:29 am

Post » Mon Apr 26, 2010 1:28 am

@PacificMorrowind
There should be a mail in your forum inbox, but I am not sure.
User avatar
hannah sillery
 
Posts: 3354
Joined: Sun Nov 26, 2006 3:13 pm

Post » Mon Apr 26, 2010 12:37 am

@PacificMorrowind
There should be a mail in your forum inbox, but I am not sure.

step 1: files that script is choking on received: done (thanks)
step 2: identify what part of the script is causing the error: done (vertex colours)
step 3: fix: in progress... should be up in a while
Pacific Morrowind
User avatar
Danger Mouse
 
Posts: 3393
Joined: Sat Oct 07, 2006 9:55 am

Post » Sun Apr 25, 2010 10:25 pm

Will this fix be something we should wait for if we are using Better Cities, PacificMorrowind?

Thanks for all your hard work!
User avatar
Kara Payne
 
Posts: 3415
Joined: Thu Oct 26, 2006 12:47 am

Post » Sun Apr 25, 2010 10:32 pm

@baphometal: I think we have now fixed it, could you test this one? http://www.sendspace.com/file/mo3msx

@swyfter: I would not think that it affects regular users of better cities, since we're fixing an issue here with a spell which is mostly useful to modders only
User avatar
Richard
 
Posts: 3371
Joined: Sat Oct 13, 2007 2:50 pm

Next

Return to IV - Oblivion