New Pyffi Automation Tool

Post » Sun May 08, 2011 4:33 am

With the Alpha 4, there's an error saying "Failed to copy test nif" when I launch Pyffi.
Was this supposed to happen or did I do something wrong?
User avatar
Eileen Collinson
 
Posts: 3208
Joined: Thu Dec 28, 2006 2:42 am

Post » Sat May 07, 2011 11:02 pm

Could I use this to PyFFI the UC Furniture retexture mod? It changes the texture paths of a number of the Harvest [Containers] meshes.
User avatar
Josh Sabatini
 
Posts: 3445
Joined: Wed Nov 14, 2007 9:47 pm

Post » Sat May 07, 2011 8:21 pm

I have to admit I got the program to run...but I have no idea which mods to PyFFI or not. So in essence I do not know what I am doing and why - point blank.

I stopped the program after about 50 of my 12,000 nifs and looked at the difference in size of each mesh. Almost cut in half. I know there is a wiki about mod cleaning and mods that are recommended to be cleaned - is there a wiki to do the same for PyFFI tp recommend nifs to be cleaned (if that is the right term)?

Ha ha - Im pretty good a following directions as my current 250 plus esp game works well with CTDs every hour or so - so I guess I shouldnt complain.
User avatar
benjamin corsini
 
Posts: 3411
Joined: Tue Jul 31, 2007 11:32 pm

Post » Sun May 08, 2011 12:09 am

With the Alpha 4, there's an error saying "Failed to copy test nif" when I launch Pyffi.
Was this supposed to happen or did I do something wrong?


Something is not right.
At startup, pyffi is run on this file and the output checked to make sure python and pyffi are set up properly on your computer.
In your case the test file could not even be copied to the right directory.
Is there a file "test.nif" in your in the install directory? It was a 229 byte nif file that was in the 7z file you downloaded.
What is python_folder in your ini file?


-dan
User avatar
Elizabeth Lysons
 
Posts: 3474
Joined: Fri Feb 02, 2007 7:16 am

Post » Sun May 08, 2011 5:31 am

Something is not right.
At startup, pyffi is run on this file and the output checked to make sure python and pyffi are set up properly on your computer.
In your case the test file could not even be copied to the right directory.
Is there a file "test.nif" in your in the install directory? It was a 229 byte nif file that was in the 7z file you downloaded.
What is python_folder in your ini file?

-dan


I must have deleted the Test.nif after I downloaded it
Just put the test.nif in my pyffi folder and everything is working fine again
Thanks
User avatar
Kat Ives
 
Posts: 3408
Joined: Tue Aug 28, 2007 2:11 pm

Post » Sat May 07, 2011 11:54 pm

does the default script ignore _far nifs as has been recommended?

if not, would adding "_far" to the regexp list do the trick?
User avatar
N3T4
 
Posts: 3428
Joined: Wed Aug 08, 2007 8:36 pm

Post » Sat May 07, 2011 7:02 pm

does the default script ignore _far nifs as has been recommended?

if not, would adding "_far" to the regexp list do the trick?


You need to add "_far" to the skip_list

-dan
User avatar
Ryan Lutz
 
Posts: 3465
Joined: Sun Sep 09, 2007 12:39 pm

Post » Sat May 07, 2011 6:30 pm

11/10! :foodndrink: Worked like a charm using default settings, save redirection to the Python folder. The only meshes optimized which exhibited issues with optimization were those in the Effects folder which were easy enough to replace. It took around three days to go through about 26,000 .NIF's of which about 18,000 were optimalized. Thankfully, they were already unpacked as I don't use archives. The result is a very noticeable increase in FPS along with quicker cell loads and the game also starts up faster. The models are all visually identical. Thank you for the automated tool, as I've long known of the process, yet was unwilling to do it manually.

Vista 32, Dual core

p.s. Would this work for Fallout 3's meshes?


You need to add "_far" to the skip_list

-dan
For whatever it's worth, it optimalized my _far.NIF's and everything looks great. :shrug:
User avatar
Nicole Kraus
 
Posts: 3432
Joined: Sat Apr 14, 2007 11:34 pm

Post » Sat May 07, 2011 5:40 pm

You need to add "_far" to the skip_list

-dan


Thanks dan, ran your tool while at work and it did its magic well.

It did lock up on one nif though - a Ren's hair nif from Origins of the Mages Guild...which is odd, I thought it would skip hair nifs anyway. I'll get the actual filename + path for you if there's any need.

I killed the python process doing that nif though and the script happily proceeded with the remaining meshes. In game results look promising.

Now I just have to extract the vanilla BSA and do them too!
User avatar
Eilidh Brian
 
Posts: 3504
Joined: Mon Jun 19, 2006 10:45 am

Post » Sat May 07, 2011 8:11 pm

Thanks dan, ran your tool while at work and it did its magic well.

It did lock up on one nif though - a Ren's hair nif from Origins of the Mages Guild...which is odd, I thought it would skip hair nifs anyway. I'll get the actual filename + path for you if there's any need.

I killed the python process doing that nif though and the script happily proceeded with the remaining meshes. In game results look promising.

Now I just have to extract the vanilla BSA and do them too!


Hmm yea, can you post your ini file and the file name that caused the problem.

-dan
User avatar
Charlie Sarson
 
Posts: 3445
Joined: Thu May 17, 2007 12:38 pm

Post » Sat May 07, 2011 9:23 pm

For whatever it's worth, it optimalized my _far.NIF's and everything looks great. :shrug:


There's unlikely to be any harm from re-optimizing ones that were already optimized. Just a waste of time is all. The stock _far.nif files need it though, so a blanket block on all of them is not advised. So long as this tool isn't forcably generating tangent spaces on _far.nif files, as they don't use that information and having it there leads to data bloat and visual anomalies.
User avatar
Silvia Gil
 
Posts: 3433
Joined: Mon Nov 20, 2006 9:31 pm

Post » Sun May 08, 2011 12:55 am

Hmm yea, can you post your ini file and the file name that caused the problem.

-dan


sure mate will get it for you when i get home from work :)
User avatar
Hilm Music
 
Posts: 3357
Joined: Wed Jun 06, 2007 9:36 pm

Post » Sun May 08, 2011 2:06 am

whoops, forgot yesterday...
anyway here is the file:

\Oblivion\Data\meshes\OriginoftheMagesGuild\renshair\OMGRen_Hair18F.nif

As I stated, PyFFI locked up on that particular file, though your script did continue with the subsequent files when i killed the locked python process.
User avatar
Karine laverre
 
Posts: 3439
Joined: Tue Mar 20, 2007 7:50 am

Post » Sun May 08, 2011 4:36 am

whoops, forgot yesterday...
anyway here is the file:

\Oblivion\Data\meshes\OriginoftheMagesGuild\renshair\OMGRen_Hair18F.nif

As I stated, PyFFI locked up on that particular file, though your script did continue with the subsequent files when i killed the locked python process.


Yea that makes sense. When you enabled skip_characters in the ini files it doesn't skip hair per se, it skips anything in the "characters\" folder which is where hair usually resides.
To fix this with this mod add "renshair\" to skip_list.
I would guess in general it would be smart to add "hair" to skip_list. That would skip any file name with "hair" in it

-dan
User avatar
Alyesha Neufeld
 
Posts: 3421
Joined: Fri Jan 19, 2007 10:45 am

Post » Sun May 08, 2011 3:38 am

yeah good point dan, and I think adding "hair" would be the way to go in the vast majority of cases.

I was still surprised that PyFFI actually choked on that file. I thought it might make a mess of it, sure, but lock out? surprising.
User avatar
Lizzie
 
Posts: 3476
Joined: Sun Nov 19, 2006 5:51 am

Post » Sun May 08, 2011 2:43 am

I asked this already in this thread http://www.gamesas.com/bgsforums/index.php?showtopic=1014947, but maybe this thread is more appropiate... I have some questions:

What does it mean if the file size after pyffiing is bigger? F.e. I get a huge difference with following file: "No Copy - F:\Oblivion meshes julij 2009 razpakirano\meshes\architecture\castle\anvil\ancastleroofbig01.nif 72865 92251
Is the mesh optimized, has different data, what's the reason for getting bigger?
Is still better (considering stability, smoothness, less CTDs...) to use bigger (maybe optimized mesh) or not?
If the answer is yes, why Alpha4 doesn't support these files?
Why is "test_size" set to .99 and to 1 or even more?
What is the real purpose of "test_size" after all?
User avatar
Nicholas
 
Posts: 3454
Joined: Wed Jul 04, 2007 12:05 am

Post » Sat May 07, 2011 9:31 pm

I have three nooby questions:

1) If I set it to send the optimized files to a separate folder (just to be safe that I don't modify the original files), then at the end I'll have to go back in and manually place those files back into the different directories, manually overwriting as I go. Or, I could make an OMOD, which as I'm typing now obviously sounds like the thing to do (although it will be a huge OMOD).
Wouldn't it be better just to make a back-up of my meshes folder, and then let the program just immediately overwrite the original files (so I don't have to do any work at the end)?
If so, how to I disable the "save optimized meshes to a different folder" feature? Do I just point it to save in my Oblivion/Data folder so it overwrites?

2) Should I run it on my entire Data folder (I have many mods), or just my Data/Meshes folder?

3) Do I have to manually set up the skip list before I run it? Or can I just run the program on default installation? When I look at the pyffi_script.config settings, the line for skip list at the bottom of the config doc reads:
python_folder 	= "C:\python25\"skip_armor   	= 0skip_weapons	= 0skip_characters	= 1skip_creatures	= 1skip_fx		= 1skip_clothes	= 1size_test       = .99num_core        = 2skip_list       ="custom"," example_two","example_three/"
Is that where I change the skip list? Like should I type "fx\" into that line so it also skips the fx folder? What about .egm? Does the program already know to skip those, or do I need to add that to skip list as well?

Thanks! -- BZ
User avatar
rebecca moody
 
Posts: 3430
Joined: Mon Mar 05, 2007 3:01 pm

Post » Sun May 08, 2011 8:10 am

Since unpacking a bsa takes just a moment I unpack bsa and define a new folder for output meshes and copythe files from the output folder back to the original unpacked folder. Then I make an uncompressed bsa, backup my original bsa and chamge name for pyffied bsa to match the original and before that I backup pyffied bsa as well , cause I don't want to go through the whole process again.

Choose only the meshes folder with .nif files, it won't have any effect on other files even if you try to pyffi them.

The original ini is fine, I added 6 or 7 waterfall files from the landscape folder, cause Arthmoore said pyffiing them makes the waterfalls ugly, I use this:

skip_list= "\landscapewaterfall01", "\landscapewaterfall02", "\landscapewaterfall03", "\landscapewaterfallfoam01", "\landscapewaterhoriz01", "\landscapewaterhoriz02", "\landscapewaterhoriz03", "\landscapewatervertbot", "\landscapewatervertmid", "\landscapewaterverttop", "helmet"

size_test compares the file size before and after pyffiing file. It means that if pyffied files gets bigger after processsing it (this is possible, but don't ask me why) it wont be copied to your output folder. I changed .99 to 1.1 and only one file was more than 10% bigger than original file (anvilroof01 or smthng in Oblivion and one file in SI.


Fx files will be skipped in every case, no need to add it again, but adding helmets won't hurt. Egm are skipped by defoult , read the readme and notes on tesnexus that came with the Alpha4.
User avatar
Bad News Rogers
 
Posts: 3356
Joined: Fri Sep 08, 2006 8:37 am

Post » Sat May 07, 2011 8:39 pm

I asked this already in this thread http://www.gamesas.com/bgsforums/index.php?showtopic=1014947, but maybe this thread is more appropiate... I have some questions:

What does it mean if the file size after pyffiing is bigger? F.e. I get a huge difference with following file: "No Copy - F:\Oblivion meshes julij 2009 razpakirano\meshes\architecture\castle\anvil\ancastleroofbig01.nif 72865 92251
Is the mesh optimized, has different data, what's the reason for getting bigger?
Is still better (considering stability, smoothness, less CTDs...) to use bigger (maybe optimized mesh) or not?
If the answer is yes, why Alpha4 doesn't support these files?
Why is "test_size" set to .99 and to 1 or even more?
What is the real purpose of "test_size" after all?

I have no idea what it means if this particular file has a larger size after pyffi process. Perhaps it means the file will make your machine run Uber fast or perhaps it means this file was corrupted by the pyffi process. If you really want to know check out the before and after in nifscope.

The "real purpse" pf test_size is to not copy a file unless the pyffi process actually made it smaller. Why? When running pyffi on a large group of files you aren't going to manual check every one, and there is a chance that any file could be corrupted. Do you want to risk having a corrupted file if it doesn't even get compressed by x amount? that is why I added test size. Set test_size it to 2 or 10 or 100 if you want every file to be copied over despite the output file size.

-dan
User avatar
Benito Martinez
 
Posts: 3470
Joined: Thu Aug 30, 2007 6:33 am

Post » Sat May 07, 2011 11:56 pm

yeah good point dan, and I think adding "hair" would be the way to go in the vast majority of cases.

I was still surprised that PyFFI actually choked on that file. I thought it might make a mess of it, sure, but lock out? surprising.


sadly "hair" will regexp "chair" which is common in files that should be pyffied . . . oh well. Better to not pyffi chairs than pyffi hair. Try saying that 10 times very fast!

-dan
User avatar
Nick Pryce
 
Posts: 3386
Joined: Sat Jul 14, 2007 8:36 pm

Post » Sun May 08, 2011 1:11 pm

Alpha 5.

Alpha 5 is ready for release (I think)
It has a greatly improved recovery ability (restarting after stoping the process or system shutdown/crash)
Also the ini file was changed to remove all the folder skip options. Everything is in "skip_list" now.
I have done fairly extensive testing with it, but since alpha 4 was the first "stable and happy" version, I don't want to release alpha 5 till more people have tested it.
If you wouldn't mind using alpha 5 and letting me know how it worked for you, I would appriciate it.

available at:

Edit:
Available at tesnexus now

-dan
User avatar
Farrah Lee
 
Posts: 3488
Joined: Fri Aug 17, 2007 10:32 pm

Post » Sun May 08, 2011 6:53 am

I optimized my meshes with this tool and I just want to say: thank you kandiedan. This tool is amazing, a lot better than the old pyffi automation tool.


My INI:

python_folder 	= "C:\python25\"skip_armor	= 1skip_weapons	= 1skip_characters	= 1skip_creatures	= 1skip_fx		= 1skip_clothes	= 1size_test	= .99num_core	= 2skip_list	="waterfall","triggers\","_far"


Optimized meshes: Vanilla, SI, Midas' Magic, Integration, The Lost Spires, Stirk, Operation Optimization, COBL (the meshes are 50% smaller now!), OOO, MMM.

I got a huge FPS increase and much lower loading times. Also stuttering is reduced to a minimum.
User avatar
Trista Jim
 
Posts: 3308
Joined: Sat Aug 25, 2007 10:39 pm

Post » Sun May 08, 2011 9:42 am

Since unpacking a bsa takes just a moment I unpack bsa and define a new folder for output meshes and copythe files from the output folder back to the original unpacked folder. Then I make an uncompressed bsa, backup my original bsa and chamge name for pyffied bsa to match the original and before that I backup pyffied bsa as well , cause I don't want to go through the whole process again.

Choose only the meshes folder with .nif files, it won't have any effect on other files even if you try to pyffi them.

The original ini is fine, I added 6 or 7 waterfall files from the landscape folder, cause Arthmoore said pyffiing them makes the waterfalls ugly, I use this:

skip_list= "\landscapewaterfall01", "\landscapewaterfall02", "\landscapewaterfall03", "\landscapewaterfallfoam01", "\landscapewaterhoriz01", "\landscapewaterhoriz02", "\landscapewaterhoriz03", "\landscapewatervertbot", "\landscapewatervertmid", "\landscapewaterverttop", "helmet"

size_test compares the file size before and after pyffiing file. It means that if pyffied files gets bigger after processsing it (this is possible, but don't ask me why) it wont be copied to your output folder. I changed .99 to 1.1 and only one file was more than 10% bigger than original file (anvilroof01 or smthng in Oblivion and one file in SI.


Fx files will be skipped in every case, no need to add it again, but adding helmets won't hurt. Egm are skipped by defoult , read the readme and notes on tesnexus that came with the Alpha4.


Thanks for a great reply. I'll now modify your sugestions to use with Alpha 5
User avatar
Jarrett Willis
 
Posts: 3409
Joined: Thu Jul 19, 2007 6:01 pm

Post » Sun May 08, 2011 8:33 am

Running Pyffi as we speak. Hopefully things will work as well as I've been reading. :D

When it's finished, I can just overwrite the old meshes with the new ones to use them, right?
User avatar
Rik Douglas
 
Posts: 3385
Joined: Sat Jul 07, 2007 1:40 pm

Post » Sat May 07, 2011 10:48 pm

Running Pyffi as we speak. Hopefully things will work as well as I've been reading. :D

When it's finished, I can just overwrite the old meshes with the new ones to use them, right?


Yes that is the simplest thing you can do (and what I imagine most people do).
Personally, I don't repack my BSAs (on my Solid State drive it is faster/less problematic to leave them unpacked) so I use Bain to keep the pyffi files as a seperate package. That way I can easily remove them/test frame rates/identify problem files.

-dan
User avatar
^~LIL B0NE5~^
 
Posts: 3449
Joined: Wed Oct 31, 2007 12:38 pm

PreviousNext

Return to IV - Oblivion