New Pyffi Automation Tool : IV - Oblivion

» Fri May 06, 2011 10:47 pm

Is the clothes skip necessary? The option yes, but the standard setting "1" I think no ("0").

I got the defaults from
http://cs.elderscrolls.com/constwiki/index.php/Nif_Optimization

Missing:

1. STOP- Button

2. BREAK- Button

Yea still trying to figure out how I want the stop/restart user interface to work. But good points.

3. lesser activity on drive/folder c:\python25 use the environment variables
TEMP
TMP

good idea. will get that in next version.

» Sat May 07, 2011 12:06 am

Can someone please tell me what Pyffi is? I have absolutely no idea.

» Sat May 07, 2011 1:41 pm

Can someone please tell me what Pyffi is? I have absolutely no idea.

Pyffi does lots of things. In this case I am using it to optomize nif files (make them smaller by removing unneeded data). Reduces the file size, vram usuage, and possibily GPU usuage.

typical data. i ran this on my data folder (oblivion, QTP3, FRAN, Warcry). Each nif was reduced by about 8.5% on average

-dan

» Sat May 07, 2011 9:47 am

ALPHA FOUR PLANS

1) Add quad support - sum one volunteer to be a tester!!!! You'll probably need to install autoit3 and spend a bit of time helping me debug.

2) Add support for general file name skip list a regexp to take out any file name that matches anything in list
Allows the user to edit ini to skip any files they want
Example :
skip_list = fx\, custom\, better cities, hair\
this would skip in file in a folder named fx, custom, or hair and skip any file with the string "better cities" in it. I know better cities is already pyffied, and that is part of my better cities path so they would all get weeded out.

3) Add .bsa support (bsa unpack, optomize, repack). Probably a second utility.

4) Resume a stopped conversion "kinda".
Plan to implement this by running normaly, but checking destination folder for file before the pyffi script is run.
If destionaltion file already exists, assume the file has already been pyffied and skip.
Can resume any sort of stop/shutdown/crash this way assuming you select the same folders.

Maybe someday . . . If i can get some help. The source is included with all the downloads so feel free to get started on these if you want to help

Add XML Black list support
Add XML File Completed support
This creates a list of files that have been pyffied. It can be used to resume a stopped download or distributed with a mod to let people know what has been pyffied already.

Add _far.nif optimization support

If there are any features not planned for alpha four that would make this a much better tool for you. Please let me know

-dan

» Sat May 07, 2011 12:23 am

Im hoping this tool will work, pyffi never really worked for me. Ill give it a try this evening. Also great job on creating this. :thumbsup:

» Sat May 07, 2011 5:58 am

Im hoping this tool will work, pyffi never really worked for me. Ill give it a try this evening. Also great job on creating this. :thumbsup:

well if you configure the ini file and it DOESN'T work, please let me know so it can get fixed

» Sat May 07, 2011 4:50 am

Need help!!!!

quad core test version is now available at tesnexus
http://www.tesnexus.com/downloads/file.php?id=25423

If you have a little time and can run this test for me, it would be very helpful.

1) Download the quad core test verison.

2) Create a folder with some test nifs (10-20 is more than enough)

3) Set num_core = 2 in ini file.
pyffi the test folder. Record processor usage and the time it takes to run the script.

4)set num_core = 3 in ini file.
Delete the output folder and repeat the test. Recod processor usage and time

5)set num_core = 4 in ini file.
repeat

If the program exits with a message box with hex error code, please let me know what the code is.

-dan

» Sat May 07, 2011 12:48 am

5)set num_core = 3 in ini file.

I would presume that to be "4 in ini file"
I would test this but unfortunately I only got a dual core.
re: the checking if already PYFFIED. (don't know what you can/can't do in this program haven't yet looked at the script, so you may/or may not be able to check this) but if the average strip less is less than 10 it has NOT been pyffied, more than 10 does not mean that it has but less than 10 always means that it hasn't.
Pacific Morrowind

» Sat May 07, 2011 5:51 am

Hi Kandiedan,

Thanks for working on this PyFFi Tool.
I do have a Quad Core, so ill give it a try and do a serie of tests.

In order to compare results with other users, we need to make sure every user PyFFiies the same files.
Here are the files i would suggest to PyFFi (since everyone has these files, they are from the Meshes bsa

)

Meshes.bsa --> architecture\anvil\meshes\architecture\anvil

Ive been PyFFiing the first 20 files from the Anvil folder:

altarcloth01.nifanvilbasemantdoorframe01.nif anvilchaindollholder01.nif anvilaltar01.nifanvilcirclebench01.nif anvildoorframe01.nif anvildoorframeint01.nif anvildoormc01.nif anvildooruc01.nif anvildoormcanim01.nifanvildooruc02.nif anvildoorucanim01.nif anvildoorucinteriorload01.nif anvilfgsecondfloor.nifanvilfgfirstfloor.nifanvilfgtopfloor.nif anvilhousecavein01.nifanvilhousegeneric01.nif anvilhousegeneric01_far.nif anvilfightersguild01.nif

My computer:

Q9450 @ stock speed (2.66GHz)
4 GB DDR3 1333 mhz (9-9-9-23 if im correct

)
Asus 4850 512 MB stock speed
OS: Windows Vista 64

Time to PyFFi => about 2 minutes
CPU ==> 50% when running 2 PyFFi scripts , maxing up to 70%

No problems however, it seems to run fine.
Used 2 cores first, will run the other tests either tonight or tomorrow night.

EDIT:

2e PyFFi test run, now PyFFied above files using 3x PyFFi via the ini configuration.

Time to PyFFi => 1 minute 20 seconds (the last mesh seems to be a problem here, its a big mesh i guess

)
CPU ==> 75% when running 3 PyFFi scripts , maxing up to 85%

» Sat May 07, 2011 6:57 am

In order to compare results with other users, we need to make sure every user PyFFiies the same files.
Here are the files i would suggest to PyFFi (since everyone has these files, they are from the Meshes bsa

)

Meshes.bsa --> architecture\anvil\meshes\architecture\anvil

Ive been PyFFiing the first 20 files from the Anvil folder:

That sounds like a very good idea, I'll run that through my system later today at both 1 and 2 core ini setting and report the results back.
Pacific Morrowind

» Sat May 07, 2011 4:06 pm

Out of curiosity... I know nothing about this other than that it is supposed to streamline meshes to improve performance. Would something/anything from this be necessary to include in NifScript's code to prevent the loss of such improvements? If you could give some details on what, exactly, this does, that would help.

Thanks.

» Sat May 07, 2011 3:36 am

Out of curiosity... I know nothing about this other than that it is supposed to streamline meshes to improve performance. Would something/anything from this be necessary to include in NifScript's code to prevent the loss of such improvements? If you could give some details on what, exactly, this does, that would help.

Thanks.

I dont have any programming knowledge but here is what PyFFi does (taken from the old http://www.gamesas.com/bgsforums/index.php?showtopic=890860, latests PyFFi thread http://www.gamesas.com/bgsforums/index.php?showtopic=976098&hl=)

What's PyFFI?
PyFFI stands for Python File Format Interface and was writen by amorilia.

PyFFI is a Python library for processing block structured binary files. Starting from a file format description written in XML, PyFFI creates customizable Python classes to read and write file blocks as described by the XML. Currently, PyFFI supports the NetImmerse/Gamebryo NIF and KFM formats, CryTek's CGF format, the TGA format, and the DDS format. Many tools for files used by 3D games, such as a stripifier, and a tangent space calculator, are included in PyFFI as well.

What is of interest here is what it can do with nif files:
It contains a script called niftoaster.py which is an optimization script that runs on files fed to it. Niftoaster does a whole host of things, but the important ones are:

Removal of Duplicate Verts
Removal of Duplicate Material Properties
Removal of Duplicate Texture Properties
Stripification of NifTriStrips
Triangulation of NifTriStrips with lengths less than 10
Recalculation of Tangent Space

In short, after PyFFiing your meshes, the filesize drops a bit, which gives less stress to your vid cards and in particular its VRAM.
However, there are a few cases in which the filesize actually grows in size.
I dont know if the original files/meshes arent optimized and are optimized after PyFFiing (although they grow in bites).
It still needs to be checked.

» Sat May 07, 2011 2:47 pm

Out of curiosity... I know nothing about this other than that it is supposed to streamline meshes to improve performance. Would something/anything from this be necessary to include in NifScript's code to prevent the loss of such improvements? If you could give some details on what, exactly, this does, that would help.

Thanks.

No you won't need to do anything, these changes will be invisible to nifscript (well I would think you could probably check for these changes in specific meshes but I think it would be rather difficult). and nifscript will not remove the improvement (unless you add back in junk data which I can't see any reason for nifscript to do unless there is a bad section of code).
Pacific Morrowind

» Sat May 07, 2011 3:16 am

NifScript, at least currently, only modifies data, doesn't add it.

Eventually I'll get functions to copy/paste branches from one mesh to another, hopefully, which may copy junk data if it's present, but that's not really my problem.

Thanks, that's exactly what I needed to know.

» Sat May 07, 2011 2:21 pm

Need help!!!!

quad core test version is now available at tesnexus
http://www.tesnexus.com/downloads/file.php?id=25423

If you have a little time and can run this test for me, it would be very helpful.
Snip...

Downloaded and will run tests tonight and post results.
Great Work! This is definitely an asset to modders and players alike.

» Sat May 07, 2011 6:20 pm

NifScript, at least currently, only modifies data, doesn't add it.

Eventually I'll get functions to copy/paste branches from one mesh to another, hopefully, which may copy junk data if it's present, but that's not really my problem.

Thanks, that's exactly what I needed to know.

as long as both are PYFFIed that won't introduce bad data and I totally agree it is not your problem if one isn't.
Pacific Morrowind

» Sat May 07, 2011 10:34 am

First, if this has been addressed since I first posted, sorry - I haven't read anything yet.

Either I didn't install the script automater to the correct file paths, or something is seriously bugged with the Quad alpha release.

With the absence of installation instructions in the Readme, I extracted the Quad alpha to a folder on my desktop.
Next I created a folder, again on the desktop and copied the first 20 files from the Architecture\Anvil folder. (My list is slightly different from the on posted for some reason).
Then I created 3 destination folders, one for each core test.

In all cases the following was the result:

[b]Pyffi Script Creator[/b]Working on 2 of 20C:\Documents and Settings\Andrew\Desktop\PyFFI TestFolder\anvilaltar01.nifFiles Copied = 0Files Skipped = 0Files Errors = 0Total Reduction (Bytes) = 0Total Bytes Processed = 0Total Size of Files Moved (before Pyffi) = 0Compression = 0[b]error[/b]ERROR 0x05[b]tmp_bat_a.bat[/b]@echo PROC A 1C:cd C:Program Files\python25\python.exe Scripts\niftoaster.py optimize --noninteractive "C:Program Files\python25\temp_a.nif" > C:Program Files\python25\tmp_a.log[b]file_list.txt[/b]C:\Documents and Settings\Andrew\Desktop\PyFFI TestFolder\altarcloth01.nifC:\Documents and Settings\Andrew\Desktop\PyFFI TestFolder\anvilaltar01.nifC:\Documents and Settings\Andrew\Desktop\PyFFI TestFolder\anvilbasemantdoorframe01.nifC:\Documents and Settings\Andrew\Desktop\PyFFI TestFolder\anvilchaindollholder01.nifC:\Documents and Settings\Andrew\Desktop\PyFFI TestFolder\anvilcirclebench01.nifC:\Documents and Settings\Andrew\Desktop\PyFFI TestFolder\anvildoorframe01.nifC:\Documents and Settings\Andrew\Desktop\PyFFI TestFolder\anvildoorframeint01.nifC:\Documents and Settings\Andrew\Desktop\PyFFI TestFolder\anvildoormc01.nifC:\Documents and Settings\Andrew\Desktop\PyFFI TestFolder\anvildoormcanim01.nifC:\Documents and Settings\Andrew\Desktop\PyFFI TestFolder\anvildooruc01.nifC:\Documents and Settings\Andrew\Desktop\PyFFI TestFolder\anvildooruc02.nifC:\Documents and Settings\Andrew\Desktop\PyFFI TestFolder\anvildoorucanim01.nifC:\Documents and Settings\Andrew\Desktop\PyFFI TestFolder\anvildoorucinteriorload01.nifC:\Documents and Settings\Andrew\Desktop\PyFFI TestFolder\anvilfgfirstfloor.nifC:\Documents and Settings\Andrew\Desktop\PyFFI TestFolder\anvilfgsecondfloor.nifC:\Documents and Settings\Andrew\Desktop\PyFFI TestFolder\anvilfgtopfloor.nifC:\Documents and Settings\Andrew\Desktop\PyFFI TestFolder\anvilfightersguild01.nifC:\Documents and Settings\Andrew\Desktop\PyFFI TestFolder\anvilhousecavein01.nifC:\Documents and Settings\Andrew\Desktop\PyFFI TestFolder\anvilhousegeneric01.nifC:\Documents and Settings\Andrew\Desktop\PyFFI TestFolder\anvilhousegeneric01_far.nif[b]files_skipped.txt and output_log.txt[/b] have 0 entries.

I had to change the ini path to Python.
Other than that, I left the it alone except for changing the num_core value.

Oh, I have a Intel Q6600 running 32 bit XP, SP3.

» Sat May 07, 2011 11:21 am

Quad Core- Test:
no problems
20 anvil files
2 files skiped
18 files optimized

round 50 seconds (18 files 4,5 MB -> 3,42 MB) running some big programs at the same time

full usage of four cores

good work :thumbsup:

» Sat May 07, 2011 4:04 pm

First, if this has been addressed since I first posted, sorry - I haven't read anything yet.

Either I didn't install the script automater to the correct file paths, or something is seriously bugged with the Quad alpha release.

With the absence of installation instructions in the Readme, I extracted the Quad alpha to a folder on my desktop.
Next I created a folder, again on the desktop and copied the first 20 files from the Architecture\Anvil folder. (My list is slightly different from the on posted for some reason).
Then I created 3 destination folders, one for each core test.

In all cases the following was the result:

[b]Pyffi Script Creator[/b]Working on 2 of 20C:\Documents and Settings\Andrew\Desktop\PyFFI TestFolder\anvilaltar01.nifFiles Copied = 0Files Skipped = 0Files Errors = 0Total Reduction (Bytes) = 0Total Bytes Processed = 0Total Size of Files Moved (before Pyffi) = 0Compression = 0[b]error[/b]ERROR 0x05[b]tmp_bat_a.bat[/b]@echo PROC A 1C:cd C:Program Files\python25\python.exe Scripts\niftoaster.py optimize --noninteractive "C:Program Files\python25\temp_a.nif" > C:Program Files\python25\tmp_a.log[b]file_list.txt[/b]C:\Documents and Settings\Andrew\Desktop\PyFFI TestFolder\altarcloth01.nifC:\Documents and Settings\Andrew\Desktop\PyFFI TestFolder\anvilaltar01.nifC:\Documents and Settings\Andrew\Desktop\PyFFI TestFolder\anvilbasemantdoorframe01.nifC:\Documents and Settings\Andrew\Desktop\PyFFI TestFolder\anvilchaindollholder01.nifC:\Documents and Settings\Andrew\Desktop\PyFFI TestFolder\anvilcirclebench01.nifC:\Documents and Settings\Andrew\Desktop\PyFFI TestFolder\anvildoorframe01.nifC:\Documents and Settings\Andrew\Desktop\PyFFI TestFolder\anvildoorframeint01.nifC:\Documents and Settings\Andrew\Desktop\PyFFI TestFolder\anvildoormc01.nifC:\Documents and Settings\Andrew\Desktop\PyFFI TestFolder\anvildoormcanim01.nifC:\Documents and Settings\Andrew\Desktop\PyFFI TestFolder\anvildooruc01.nifC:\Documents and Settings\Andrew\Desktop\PyFFI TestFolder\anvildooruc02.nifC:\Documents and Settings\Andrew\Desktop\PyFFI TestFolder\anvildoorucanim01.nifC:\Documents and Settings\Andrew\Desktop\PyFFI TestFolder\anvildoorucinteriorload01.nifC:\Documents and Settings\Andrew\Desktop\PyFFI TestFolder\anvilfgfirstfloor.nifC:\Documents and Settings\Andrew\Desktop\PyFFI TestFolder\anvilfgsecondfloor.nifC:\Documents and Settings\Andrew\Desktop\PyFFI TestFolder\anvilfgtopfloor.nifC:\Documents and Settings\Andrew\Desktop\PyFFI TestFolder\anvilfightersguild01.nifC:\Documents and Settings\Andrew\Desktop\PyFFI TestFolder\anvilhousecavein01.nifC:\Documents and Settings\Andrew\Desktop\PyFFI TestFolder\anvilhousegeneric01.nifC:\Documents and Settings\Andrew\Desktop\PyFFI TestFolder\anvilhousegeneric01_far.nif[b]files_skipped.txt and output_log.txt[/b] have 0 entries.

I had to change the ini path to Python.
Other than that, I left the it alone except for changing the num_core value.

Oh, I have a Intel Q6600 running 32 bit XP, SP3.

That error code is being generated because there is no python log file.
Without knowing your system setup I would say you have one of the following problems.
1) Python is not installed (possibly installed incorrectly)
2) Pyffi is not installed (possibly installed incorrectly or not in default path)

So . . .
Did you use the wryebash python install? If not, what version of python are you running?
What version of pyffi are you running? Is niftoaster.py located in "python_root"\scripts

-dan

-edit
yea I know that it should check for correct install of python/pyffi before running.
That's one of the things I am working on for the next update.

» Sat May 07, 2011 11:48 am

Quad Core- Test:
no problems
20 anvil files
2 files skiped
18 files optimized

round 50 seconds (18 files 4,5 MB -> 3,42 MB) running some big programs at the same time

full usage of four cores

good work :thumbsup:

Great. Thanks for the testing.

-dan

» Sat May 07, 2011 10:09 am

I just tested it quick on my quad core.

Got an error 0x01F:\ bla bla, file path to the folder to optimize.

It started up ok, generated a list and skipped alot of files, none were updated of the few that was processed before the error. It was mostly charcter or clothes stuff. EDIT --- Oh it says "Failed Regexp Test" on the skipped files. End Edit.

I'll do some more testing later this evening when I got some more time.

Robin.

» Sat May 07, 2011 2:56 pm

@Locksley

Look here pyffi_script.ini:

python_folder 	= "C:\python25\"skip_armor	= 0skip_weapons	= 0skip_characters	= 1skip_creatures	= 1skip_fx		= 1skip_clothes	= 1size_test	= .97num_core	= 4

If you want to optimize clothes, then set skip_clothes to "0".
If you want to optimize characters, then set skip_characters to "0".

Is your python drive and folder "c:\python25" ?

» Sat May 07, 2011 4:56 am

@ Baphometal:

Could you tell me what kind of CPU you're using ?

» Sat May 07, 2011 5:09 pm

@Locksley

Look here pyffi_script.ini:

python_folder 	= "C:\python25\"

Is your python drive and folder "c:\python25" ?

Ok, changed that ("C:\GameOS\Program Files (x86)\Python\") got a Error 0x05 wich is funny because I got an other error the first time when I changed the python folder.

I will check things a little bit better tonite'.

» Sat May 07, 2011 1:20 pm

Ok, changed that ("C:\GameOS\Program Files (x86)\Python\") got a Error 0x05 wich is funny because I got an other error the first time when I changed the python folder.

I will check things a little bit better tonite'.

Are you using Vista? There might be a problem with accessability when you install programs in Program Files.
Or did you change the folder from C:\GameOS\Program Files (x86)\Python\ to C:\Python25 ?