Github is where the bleeding edge lives. I think the 2.1.10 beta only deals with a corner case bug for exporting from Blender that I found. Not sure if anything else was changed so it should work just like 2.1.9.

I linked a set of updated files in the optimization kit thread, but maybe they'll be more visible here. http://www.4shared.com/file/gvbLWTgs/OptKitFixes.html

The updated files correct behavior when handling _far.nif files.

Ok, I did it but I see no performance improvement over the vanilla meshes.


That's the set I've been using to optimize everything, thanks.

Could someone provide binary patches of meshes for vanilla and SI that were optimized with the latest version of pyffi?

After running PyFFI v2.1.9, I looked at the log files. In some cases they can be very large.
How do I interpret the log files?
Warnings: unused vertex
What does it mean? Should I ignore?

Hi,

I've run into a strange issue while PyFFIng things. When I give it a large group of meshes to work on, it continually grows a larger and larger pool of processes and threads until it eats all of my memory and swap and my PC grinds to a halt. This is troublesome because ideally I'd like to have it run overnight on everything, rather than feeding it small groups of meshes every hour or so.

I'm using the 2.1.10 dev version of PyFFI along with the optimization kit v7 and Python 2.7.2. I'm just trying to verify if this is normal PyFFI behaviour, or if I'm using known bad versions of PyFFI/Python, or if there's something I need to configure differently?

Thanks for any advice .

Edit: Hmm, looking at the logs the v7 optimization kit provides (previously had same issue without the kit) I'm repeatedly seeing a message that says: No handlers could be found for logger "pyffi.nif.nitribasedgeom"

Not sure if that helps..?

I'm now going through and re-pyffing my vanilla meshes piecemeal with the optimization kit v7 + Arthmoor's posted fixes. I had previously pyffied them just using the INI that came with PyFFI, so I wanted to re-do it to make sure I'm getting well-optimized but non-buggy meshes. I'll afterwards proceed through to do the same with other meshes from mods.

However, while I can still PyFFI things piecemeal to avoid the above problem, it does leave me slightly concerned that perhaps I'm working with a buggy PyFFI / Python version or config, and will end up with buggy meshes. If the above described PyFFI behaviour is normal and expected, please just let me know .

That error message only means that PyFFI was unable to write something to the log for some reason. It's harmless.

Can't really say anything about your problem, though. Hardware usage issues are Amorilia's domain, who is currently MIA.

I've never had PyFFI begin consuming every ounce of memory until the system is exhausted of resources. Something is definitely not right with that. You may want to stick with PyFFI 2.1.9 in this case since 2.1.10 is still early beta.

I've tried uninstalling PyFFI 2.1.10 dev and installing 2.1.9, but unfortunately I'm seeing the same issue . Basically what's actually happening to eat the memory is that PyFFI is leaving many, many active python processes unterminated in the background, each of which uses 10-30MB of memory. During the time it took me to optimize just the vanilla clutter and creatures folders, it has left about 30 orphaned processes (more threads). These processes are terminated when PyFFI finishes (presumably at the end of each "pass" of the optimization kit, since each is a separate invocation of PyFFI).

I've tried various settings for "JOBS", with little effect.

I'm not sure what the issue is here exactly, and I'm not sure whether it affects the final result. It's a pain to only PyFFI small groups of meshes to avoid flooding my RAM, but I can work around that. I'm more concerned with whether this issue is indicative of other potential problems, i.e. PyFFI not doing its job correctly.

Should I downgrade Python? I'm using 2.7.2, maybe I should try a 2.6-series...

Edit: I've uninstalled Python 2.7.2 and installed the 2.7.1 that came with the Wrye Python package, and then I uninstalled and re-installed PyFFI 2.1.9, this time without even changing the install folder of PyFFI. Making another attempt now, it'll be a while before I know if there's any change...

I'm now up to 50+ separate python processes using the latest setup described below . I'm using JOBS=3, to give me enough processor room on my quad core to use Firefox to post this kind of stuff.

Since I'm now using the standard stable versions of everything, I'm at a bit of a loss. I'll try going through the log more thoroughly and hope to find something useful there....

Edit: In the time it took me to write that post, 30+ additional python processes spawned. I'm now over 80 .

Edit 2: Terminated PyFFI to check the log:

pyffi.toaster:4052:INFO:=== ./in\Vanilla\meshes\creatures\boxtest\skeleton.nif ===
pyffi.toaster:4052:ERROR:TEST FAILED ON ./in\Vanilla\meshes\creatures\boxtest\skeleton.nif
pyffi.toaster:4052:ERROR:If you were runniBip01 R Foot] ~~~
pyffi.toaster:3776:INFO: ~~~ NiNode [Bip01 R Toe0] ~~~

pyffi.toaster:3780:INFO:=== ./in\Vanilla\meshes\creatures\minotaur\eyelidslord.nif ===
pyffi.toaster:3780:ERROR:TEST FAILED ON ./in\Vanilla\meshes\creatures\minotaur\eyelidslord.nif
pyffi.toaster:3780:ERROR:If you were running a spell that came with PyFFI, then
pyffi.toaster:3780:ERROR:please report this as a bug (include the file) on
pyffi.toaster:3780:ERROR:http://sourceforge.net/tracker/?group_id=199269

pyffi.toaster:3500:INFO:=== ./in\Vanilla\meshes\creatures\mudcrab\mud crbeye l00.nif ===
pyffi.toaster:3500:ERROR:TEST FAILED ON ./in\Vanilla\meshes\creatures\mudcrab\mud crbeye l00.nif
pyffi.toaster:3500:ERROR:If you were running a spell that came with PyFFI, then
pyffi.toaster:3500:ERROR:please report this as a bug (include the file) on
pyffi.toaster:3500:ERROR:http://sourceforge.net/tracker/?group_id=199269

pyffi.toaster:3032:INFO:=== ./in\Vanilla\meshes\creatures\mudcrab\mud crbeye r00.nif ===
pyffi.toaster:3032:ERROR:TEST FAILED ON ./in\Vanilla\meshes\creatures\mudcrab\mud crbeye r00.nif
pyffi.toaster:3032:ERROR:If you were running a spell that came with PyFFI, then
pyffi.toaster:3032:ERROR:please report this as a bug (include the file) on
pyffi.toaster:3032:ERROR:http://sourceforge.net/tracker/?group_id=199269

First one got cut off by another thread writing to the log, it seems.
These are certainly something to ask about, but I don't think the number of errors I found is enough to explain it spawning 80+ processes within a few minutes.

Are the above errors anything to worry about, or is that something that's taken care of by later passes of the optimization kit?

Also, still seeking any advice about the process spawning issue. Anything specific I should look for in my log?

Just a little question. Is there a spell which converts TriStripData to TriShapeData?

I'm running Python 2.7.2 here myself so that's not likely to be the issue. I have no explanation for why PyFFI is invoking that many zombie processes (as they're called in linux) so you'd probably need to try and dredge up Amorilia to ask about it.

I've opened a bug report on the Sourceforge project page for this issue, with additional details and log files. If anyone is interested you can find it https://sourceforge.net/tracker/?func=detail&aid=3390826&group_id=199269&atid=968813.

Thanks for the feedback on this

Hey Amorilia,

I'm in the planning stages for a few new features for Wrye Bash, and I was wondering if I could add a dependency on pyffi for some functionality. Specifically, I'd like to:

• get the list of textures used by a mesh
  
• modify texture paths in a mesh so it can be previewed (in an embedded nif viewer, probably) without being installed
  
• convert the data in dds files into something that can be viewed in a wx widget
  
• walk .bsa files and extract their contents
  
• pack new .bsa files

I am also exploring ideas about integrating pyffi mesh optimization into the Wrye Bash installation process. Is this something you would be interested in seeing happen?

Also, I note in the documentation that some care has been taken to make the code cross-platform. Has it been tested on Linux? Is the functionality listed above available on Linux?

Finally, do you have plans to support Skyrim?

Thanks,
Myk

I'm using PyFFI 2.1.9 along with the optimization kit v7 and Python 2.6.5.
I've looked at the log files and many times they contain warnings:
pyffi.toaster:1636:WARNING:unused vertex

Can I ignore them?

From EVE_for_Oscuro_Oblivion_Overhaul_1_3_BAIN_installer-24078.7z
pyffi.toaster:5352:ERROR:TEST FAILED ON ./in\meshes\armor\BeastClaw\f\beastclawgauntlets.nif
pyffi.toaster:5352:ERROR:If you were running a spell that came with PyFFI, then
pyffi.toaster:5352:ERROR:please report this as a bug (include the file) on
pyffi.toaster:5352:ERROR:http://sourceforge.net/tracker/?group_id=199269

This one is pretty obvious.

Well, the best solution would be to do as PyFFI tells you and report the error on the bug tracker, which is linked in the OP.

Dunno if you'll get a response, though, Amorilia's been MIA for a while now.

Ok, Should I ignore the warnings in the first example?

Myk, I would love to see these proposed features in Bash! pyffi and nif previewing? Wow.

Alphanos, had same problem with zombie python processes, using: pyffi opti kit + Arthmoor's ini's + pyffi 2.1.9 stable + python 2.7 on winXP machine.

As far as I can tell, cause of problem is last .ini which has ?Out? folder for both source and destination; when i have whole oblivion meshes bsa in ?out? folder amd when it comes to the last .ini optimisation (somethig with _far nifs) program effectivly takes all my ram pretty fast, crashing stuff in background, producing some popup errors, etc etc.... My workaround has been to optimize few folders at time and while i've been extremely pedantic about it I''m still stuck with CTD after doing all the work, as http://www.gamesas.com/index.php?/topic/1172029-relz-oblivion-pyffi-optimization-kit/page__view__findpost__p__18369806revelting tale will describe you if you are interested, but you probably aren't.

Great irony lies here as we, who would benefit from optimizing meshes most having weakish (for today standards) PCs, have hardest time pyffing, crunching numberz, waiting endlessly... While otoh I read about quad cpu guy who runs optimization in 1 hour, like he can anything possibly gain from this. Oblivion must be flying by default on such machine. ~

A couple of quick Python/PyFFI questions. (I'm doing this from memory since I'm at work right now, so might have some of the details wrong.)

If I remember correctly, there's several versions of Python: 2.7.x for 32-bit machines, 2.7.x for 64-bit machines, and 3.x.x.
I presume that the 3.x.x versions are incompatible with PyFFI. But what about the 64-bit version of 2.7.x? Are there any advantages, or disadvantages, to using it?
EDIT: I did find Arthmoor's May 3 comment on the x64 version being incompatible with Wrye Bash. Any problems besides that?

EDIT 2: If no one can answer this question, I might install x86 AND x64 and see what differences (if any) I come up with.

I noticed your other thread actually, and I've also now been experiencing CTDs all over the place after optimizing the meshes for all of my installed mods. I didn't have the issue with just the vanilla meshes though, so I don't know if I can help there. If/when I figure out how to fix it I'll be sure to share, and I'll be watching your thread in the meantime.

This zombie process problem I can help with at least a little - for my own use, I cobbled together an extra trio of batch files that will automatically process a queue of directories containing meshes one at a time. You still need to watch out to make sure that any individual folder doesn't contain enough meshes to eat all of your RAM, but once you've done that check it's possible to just set it up and let it process nearly unlimited numbers of meshes, like most people seem to be able to do without any extra trouble . I had assumed that I was the only one having this problem; it's somewhat reassuring actually that it's happening to someone else too.

I'll find somewhere to post the batch files so you can grab them if you'd like .

Edit: http://www.sendspace.com/file/rvz7rx.
Disclaimer: I made this for my own use, and if your computer explodes from using it I take no responsibility.

JOBS: batch_optimize_slave.bat is a minorly-edited copy of the original kit's batch file. If you want to change the JOBS setting from what I was last using (8), then just edit that line in the file before use.

Use: Place multiple directories containing meshes into the new "pending" folder, and run batch_optimize.bat. The batch files will, one at a time, move a folder from pending to in and run the normal 4-stage optimization process. Once complete, the folder will be moved from in to complete_in, and the output folder will be moved to complete_out. Then it will do the same thing to the next folder in pending. When it finishes it simply closes with no extra messages or anything.

Note that if you put a folder in pending with a huge number of meshes, it's still possible to eat all of your RAM with zombie processes. This is not a "fix" for the zombie process issue; it simply breaks up huge quantities of meshes into smaller chunks, which alleviates the problem somewhat since the zombie processes get killed after each stage in the process.

Personally, I've found best results from making sure that no more than 2,000 meshes are in any single folder within pending. I know other factors affect this, but I haven't a clue what exactly, so to be safe I'd try to stay under 1,000 if you can.

Additional benefit: If you do mis-estimate the number of meshes that can be handled and the machine crashes, while you should still start over with anything current in the in/out folders, you can rest assured that folders which have already been moved to complete_in and complete_out are 100% done. No more starting from scratch since you aren't totally sure if it finished everything in a folder .

I wrote a little application (with niflib) which converts TriStrips to TriShapes and also optimizes TriShapes (with Tootle). Anyone interested in trying it out to root out (possible) problems?

I don't actually know much about how PyFFI really works. What does this new program do, exactly?

It takes a NIF and restructures the mesh. First it converts triangle-strips to triangle-lists. Then it does some basic indexing of the mesh, which in turn will remove all unused and double points (duplication is given by pos+UVs+normal+color for the vertex). Then it reassignes all the indices of UV, weights of bones etc. to the new indices. After that it fires up a raytracer which optimizes the index-order for maximum overdraw (to take advantage of early-z from the graphics card) and vertex cache. After that in addition it reorders all the vertex-positions for maximized local vertex buffer fit. Then it writes the NIF out again.

This is an example from a original Ob-mesh:
anvilaltar01.nif, before "515.862", after "421.326"
25% removed vertices
50% removed indices (as an effect of less vertices, and removal of degenerate indices necessary for strips)

The file here is A-okay. The UVs are good, the normals are good, no holes ...
The tool runs on files and directories.

Does this tool log errors it encounters when it runs so that meshes that bug out can be examined to see why?

Also, will it cover the necessary additional cleanup for _far.nif files?