[RELZ/WIPZ]PyFFI: Python File Format Interface

Post » Tue May 11, 2010 2:53 pm

You copy those lines in the code windows, add them to a .TXT file, change the directories in the quotations to whatever they are on your computer, and save the .TXT. Afterward, you change the .TXT to .BAT to allow Windows to execute them as a batch script when you open them.

Using the windows installer will take care of everything, including removing the previous version, for you.

For your last question, not unless a major optimization change has been made for Oblivion's meshes. Between 2.1.7 and 2.1.9b6, no real new developments have been made for Oblivion optimization, so you don't need to worry for now.
User avatar
Sabrina Schwarz
 
Posts: 3538
Joined: Fri Jul 14, 2006 10:02 am

Post » Tue May 11, 2010 1:46 pm

Hi Thomas,

Thanks so much for taking the time to answer my questions! I have created the .txt files and turned them in .bat files by replacing the .txt to .bat. Do I need to place these anywhere specific or can I just run them from any location? Just to verify, by running these .bat files, does this mean I will NOT have to optimize using the normal methods (right clicking on the .inis in the "toaster" folder and clicking "Run PyFFI or just right clicking on a folder and selecting "Optimize using PyFFI)? All I would have to do is first put my meshes in the "in" folder and then run the .bat files, correct?
User avatar
Karl harris
 
Posts: 3423
Joined: Thu May 17, 2007 3:17 pm

Post » Tue May 11, 2010 7:54 am

I would keep them in your PyFFI folder to be safe. Nice and organized that way. Also, yes, those batch scripts replace the normal "Run PyFFI" and Optimize with PyFFI" options.
User avatar
Lisha Boo
 
Posts: 3378
Joined: Fri Aug 18, 2006 2:56 pm

Post » Tue May 11, 2010 1:30 pm

Very nice! Will do! Thanks again! :foodndrink:
User avatar
My blood
 
Posts: 3455
Joined: Fri Jun 16, 2006 8:09 am

Post » Tue May 11, 2010 4:58 am

I'm amazed you are doing all the work by hand. :ninja:
User avatar
MR.BIGG
 
Posts: 3373
Joined: Sat Sep 08, 2007 7:51 am

Post » Tue May 11, 2010 3:03 pm

@ulrim

Are you referring to me? If so, is there an easier way or something I am missing?
User avatar
Scott Clemmons
 
Posts: 3333
Joined: Sun Sep 16, 2007 5:35 pm

Post » Tue May 11, 2010 6:23 pm

*chuckles* I think he's referring to his new http://www.gamesas.com/index.php?/topic/1172029-relz-oblivion-pyffi-optimization-kit/ :D I wasn't sure what you were up to or I'd have recommended it earlier...

Hey Ulrim, can you stick that link in your sig? It'd make it easier to find :D
User avatar
Jake Easom
 
Posts: 3424
Joined: Sun Jul 29, 2007 4:33 am

Post » Tue May 11, 2010 2:22 pm

Oh wow! That would make things a bit easier. I think I am going to go give that a try right now. By the way, I read it can take a long time (like 24-48 hours) to opt. the entire vanilla OB meshes bsa. Is this correct? Could I just opt. a couple of the vanilla mesh folders at a time so my computer isn't bogged for so long or do I need to do the entire bsa in one sitting? Thanks again for all the help and advice!
User avatar
Deon Knight
 
Posts: 3363
Joined: Thu Sep 13, 2007 1:44 am

Post » Tue May 11, 2010 10:52 am

Oh wow! That would make things a bit easier. I think I am going to go give that a try right now. By the way, I read it can take a long time (like 24-48 hours) to opt. the entire vanilla OB meshes bsa. Is this correct? Could I just opt. a couple of the vanilla mesh folders at a time so my computer isn't bogged for so long or do I need to do the entire bsa in one sitting? Thanks again for all the help and advice!


Ulrim has also released this other thing called binary patches that appear to speed up the process. I will admit I'm not entirely clear on what these patches do, other than speed up the optimization process of the vanilla meshes. The thread should be on the first or second page of this forum.
User avatar
Dean Ashcroft
 
Posts: 3566
Joined: Wed Jul 25, 2007 1:20 am

Post » Tue May 11, 2010 7:20 am

PyFFI has support for binary patches but I think it is still experimental.
The way it works is like this: instead of saving an optimized mesh ready to use, it saves a binary difference relative to that mesh.

What do you do with a binary difference file?

Suppose you don't have any optimization and your machine is very slow to run PyFFI. If you have access to this binary difference file, you can update your original mesh very quickly. It becomes exactly the same as the optimized version you would have if running the full PyFFI optimization.

Since binary patch support in PyFFI is experimental, I've released http://www.tesnexus.com/downloads/file.php?id=37548 for those who don't have very powerful computers to run PyFFI.
User avatar
Matthew Aaron Evans
 
Posts: 3361
Joined: Wed Jul 25, 2007 2:59 am

Post » Tue May 11, 2010 11:33 am

PyFFI has support for binary patches but I think it is still experimental.
The way it works is like this: instead of saving an optimized mesh ready to use, it saves a binary difference relative to that mesh.

What do you do with a binary difference file?

Suppose you don't have any optimization and your machine is very slow to run PyFFI. If you have access to this binary difference file, you can update your original mesh very quickly. It becomes exactly the same as the optimized version you would have if running the full PyFFI optimization.

Since binary patch support in PyFFI is experimental, I've released http://www.tesnexus.com/downloads/file.php?id=37548 for those who don't have very powerful computers to run PyFFI.


It's essentially the same thing as the linked PyFFI patcher, then? You create the binary patches with PyFFI, and anyone can download and plug in those patch files to obtain the PyFFI optimized meshes?

If yes, I'll update the link on the OP to yours, since yours is the most recent version now.
User avatar
Fanny Rouyé
 
Posts: 3316
Joined: Sun Mar 25, 2007 9:47 am

Post » Tue May 11, 2010 6:23 am

Exactly. As it is right now it's the same as the PyFFI patcher.

My utility can create patches from the 'in' folder without having to run PyFFI (that's how I created the patch files but it's disabled by default - more on this below).
Since I wrote it in python it will be easier to turn it into a window application and ditch the command line batch script. But that's for when I have some spare time ;)

To create patches with PyFFI, pass the '--diff' command to niftoaster.py.

With my 'Oblivion - PyFFI Binary Patch Kit' you'd have to:
    place the original meshes inside 'in'
    place the optimized meshes inside 'out'
    open 'patcher.py' with a text editor and at the end change
    scan(s_dir, s_list)#scan(t_dir, t_list)scan(p_dir, p_list)#create_patch(s_list, t_list)apply_patch(s_list, p_list)
    to
    scan(s_dir, s_list)scan(t_dir, t_list)scan(p_dir, p_list)create_patch(s_list, t_list)#apply_patch(s_list, p_list)
    run the patcher and patch files are created inside 'patch' folder, recursively.

User avatar
cheryl wright
 
Posts: 3382
Joined: Sat Nov 25, 2006 4:43 am

Post » Tue May 11, 2010 4:17 pm

Sounds good, I'll be updating the post momentarily. :)
User avatar
Casey
 
Posts: 3376
Joined: Mon Nov 12, 2007 8:38 am

Post » Tue May 11, 2010 7:24 am

To create patches with PyFFI, pass the '--diff' command to niftoaster.py.


To clarify: the --diff argument will be deprecated starting from 2.1.9 and will be dropped from a future release. To create or apply patches with pyffi, simply use the patch_recursive_make/apply.bat files in PyFFI's utilities/toaster/ folder. Quick instructions:
  • patch_recursive_apply.bat: for each file in in/ (and in its subdirectories) checks if there is a similarly named file in patch/ and if so, writes a patched file to out/
  • patch_recursive_make.bat: for each file in in/ (and in its subdirectories) checks if there is a similarly named file in out/ and if so, writes a patch to patch/


Be sure to use the latest beta when trying this out (the first release of this feature had a bug, see http://sourceforge.net/tracker/?func=detail&aid=3193914&group_id=199269&atid=968813). But if you have 2.1.8 or earlier, yes, the only way to do with with PyFFI is to use --diff with the appropriate arguments - but it's a bit tricky to set it up correctly, whence the new .bat files.

As usual, please report any bugs with it on the tracker: http://sourceforge.net/tracker/?group_id=149157
User avatar
Queen Bitch
 
Posts: 3312
Joined: Fri Dec 15, 2006 2:43 pm

Post » Tue May 11, 2010 8:53 pm

That's great! Its the same thing my code does. Thanks for importing the improvement to PyFFI!

I suspect the difficulty in configuring PyFFI was the worst thing for new users but now it's getting easier and easier to use.


--

Edit: Actually, it's not. You might want to check the tracker for the bug report.
User avatar
Marine x
 
Posts: 3327
Joined: Thu Mar 29, 2007 4:54 am

Post » Tue May 11, 2010 6:13 am

Thanks for the feedback - let's continue the discussion of potential bugs on the tracker.

For clarity, the scripts for making and applying patches have been in PyFFI for ages, but as you say, obviously I'm apparently not the most sensible person if it comes to tools that embody the principle of ease of use. :) I hope that with the last generation of batch files, I finally got it right somehow - or will, with your help.

Thinking of this a bit more, what we really want is a standard and easy way for people to release their patch sets, including a "patch recursive apply" script with clear instructions so users have no additional dependencies to download (such as pyffi etc.). So, in addition to patch_recursive_make.bat, we also need patch_release.bat which bundles all the necessary files together, with instructions and all. There are already a few patch kits out there which don't have a Python dependency - one of these may be preferred over our scripts?
User avatar
Daniel Brown
 
Posts: 3463
Joined: Fri May 04, 2007 11:21 am

Post » Tue May 11, 2010 9:08 pm

I realize this is the not the New Vegas forum, but I haven't had any luck searching for the answer to this question. According to the sourceforge page PyFFI does support F:NV, however when I ran it and created a new bsa with the PyFFIed meshes the game would crash on load every time. I couldn't find a skip-list for NV so I used the default one.

Are there limitations as to what NV meshes I can PyFFI that can be causing the crashes?
User avatar
Amy Gibson
 
Posts: 3540
Joined: Wed Oct 04, 2006 2:11 pm

Post » Tue May 11, 2010 3:11 pm

krimhorn, pyffi-ing any game but Oblivion is completely untested. That said, there have been a few bug reports recently for Fallout 3 (which is similar to Fallout NV). Those have been addressed, but apparently from your experience it appears that there are still outstanding issues. What you can do to help is: indentify the troublesome nif(s) which don't get optimized properly, and post a report on the pyffi bug tracker here (no need to include the files, unless its from a mod, as I have the game): http://sourceforge.net/tracker/?atid=968813&group_id=199269&func=browse
User avatar
Dean Ashcroft
 
Posts: 3566
Joined: Wed Jul 25, 2007 1:20 am

Post » Tue May 11, 2010 12:51 pm

So, in addition to patch_recursive_make.bat, we also need patch_release.bat which bundles all the necessary files together, with instructions and all. There are already a few patch kits out there which don't have a Python dependency - one of these may be preferred over our scripts?

I gave it some thought lately, too. However, I still don't find it useful (at all) to invent a "bundle" (archive) file to use solely with some kind of program that processes it. (Quick example: you can't open omods with 7zip to check the contents. OBMM has to be used, always, to open the packed binary omod file format.)
Just let people use their preferred compressing utilities - like 7zip or zip or whatever - to pack their files and have the tool process the directory tree with xdelta.

For what is worth, you might want to know I'm aiming at making this myself already. Suggestions welcome, of course, and I'll notify about any progress.
User avatar
Anthony Diaz
 
Posts: 3474
Joined: Thu Aug 09, 2007 11:24 pm

Post » Tue May 11, 2010 1:37 pm

you can't open omods with 7zip to check the contents.

If you've installed the 7zip context menu, you can right-click on an omod, chose 7zip, open. Or you can open 7zip and open the omod from the open file menu. The only reason for the omod format I see is that it will open OBMM if you double-click.
User avatar
christelle047
 
Posts: 3407
Joined: Mon Apr 09, 2007 12:50 pm

Post » Tue May 11, 2010 8:37 am

However, I still don't find it useful (at all) to invent a "bundle" (archive) file to use solely with some kind of program that processes it.


Agreed. I had a 7zip (or zip) archive in mind containing all patches, with batch file + instructions in the root.
User avatar
Rob Smith
 
Posts: 3424
Joined: Wed Oct 03, 2007 5:30 pm

Post » Tue May 11, 2010 2:03 pm

All I see by opening some .omod in 7z is:
configdatadata.crcscript

How do you actually open "data" in 7z? The "script" is editable, but in "data" all I see is a packed binary format.

Anyway, I've got my application now. Since it is working very well in general patching I think it won't be related to PyFFI. It can process any file. No need to clutter the board with my ideas :)
User avatar
GPMG
 
Posts: 3507
Joined: Sat Sep 15, 2007 10:55 am

Post » Tue May 11, 2010 8:58 pm

All I see by opening some .omod in 7z is:
configdatadata.crcscript

How do you actually open "data" in 7z? The "script" is editable, but in "data" all I see is a packed binary format.

Anyway, I've got my application now. Since it is working very well in general patching I think it won't be related to PyFFI. It can process any file. No need to clutter the board with my ideas :)

:blush2: You're right, I should have checked before posting, I guess my memory is faulty (or else something has changed in 7zip or the omod format in the year since I last played Oblivion :whistling:).
User avatar
CxvIII
 
Posts: 3329
Joined: Wed Sep 06, 2006 10:35 pm

Post » Tue May 11, 2010 12:14 pm

I've just released PyFFI 2.1.9:

https://sourceforge.net/projects/pyffi/files/pyffi/2.1.9/

Oblivion relevant entries in the changelog:

  • Added new script to make and apply patches (functionality is identical to and OnmyojiOmn's and jonwd7's pyffi patcher scripts, but it is written in Python to work on all platforms).
  • New fix_emptyskeletonroots spell (automatically included in the optimize spell) to fix issues with nifs that do not have their NiSkinInstance skeleton root set (fixes #3174085, reported by xjdhdr).
  • Fixed logging issue on Windows platform with multithreading enabled (fixes issue #3174339, reported by xjdhdr).
  • Fixed QSkope shortcut issue when path contains spaces (reported by Brumbek).
  • Skip terminal chars in mopper output (fixes issues with running mopper under wine).
  • Bugfix in patch_recursive_make/apply scripts for "deep" folder layouts (fixes issue #3193914, reported by xjdhdr).
  • Do not pack collisions in OL_CLUTTER (fixes issue #3194017 reported by Gratis_monsta).
  • Fixed issue with skipping terminal chars in mopper output on Windows platform (fixes issue #3205569, reported and diagnosed by ulrim).
  • Disabled copy in patch_recursive_apply script (see issue #3219744, suggested by ulrim).
  • Pass additional arguments of patch_recursive_apply/make to the patch command (see issue #3219744, suggested by ulrim).
  • Sane default settings for bhkRigidBody unknowns to ensure that constraints behave properly (contributed by Koniption).


Do you need to pyffi again with this version? If you use any mods, then probably yes (crucial fix for clutter items that might be disappearing), but otherwise, it's probably not worth the extra effort.

Full changelog is here:

https://github.com/amorilia/pyffi/blob/python25/CHANGELOG.rst

For those interested in Fallout 3 pyffi optimization - there are many fixes for it as well - although I'd say it is still in a very experimental state.

As always, please report any bugs on the tracker:

https://sourceforge.net/tracker/?group_id=199269&atid=968813

Enjoy!
User avatar
Penny Wills
 
Posts: 3474
Joined: Wed Sep 27, 2006 6:16 pm

Post » Tue May 11, 2010 12:18 pm

I PyFFI'd the stock Shivering Isles meshes and I got an "wtf I'm missing mesh" exclamation mark for this mesh:

meshes/clutter/sefurniture/secountercurve01.NIF

the mesh was there. It was resolved loading the non-pyffied mesh instead.

Could PyFFI have broken the mesh somehow? o.o

PyFFI version.......... not the one just released above, but the last stable one. Not a beta or alpha if I recall correctly.

EDIT:
(crucial fix for clutter items that might be disappearing)


already covered?
User avatar
Danny Warner
 
Posts: 3400
Joined: Fri Jun 01, 2007 3:26 am

PreviousNext

Return to IV - Oblivion