[INFO][RELZ] PyFFI-Python File Format Interface

Post » Mon May 02, 2011 7:12 pm

If you want to integrate patching in some way, that sounds good to me. I don't know if my batch file is anything special. The old version by didn't work on anything but XP, and I still haven't gotten around to testing my batch file in XP Mode in Virtual PC. The thing is that I'm finding some people with GOTY version say their BSA differs, some say their BSA is the exact same. So I don't know who is right. I had somebody verify for me that their GOTY BSA had the exact same MD5/SHA-1 checksums as my original release BSA. The script also still relies on Forfiles.exe which not all Windows versions have, apparently. What this means is extracting the BSA and patching the files and recreating the BSA instead of PyFFI-ing is currently about as hard to get the head around as PyFFI-ing itself, for novice users. The only current benefits to patching is time.

I was possibly going to do a more robust EXE file... A commenter had some good ideas like BSA extraction/creation automation with BSA commander, which would help people with patching the vanilla BSAs, and I could verify checksums and what not even.

Of course, that all deals with just the vanilla meshes... Most PyFFI-ing doesn't take nearly as long, so what exactly is the need for patching in PyFFI? I'm not against it, just curious. :)

I could see it working in the In/Out environment. An additional Patches folder which generates the Patches after PyFFI has finished running (although the two-part INI process kind of mucks that up). The optimize-in-place PyFFI-ing seems more tricky. Of course the In/Out/Patches environment is identical to the way the patcher is currently set up, which is why it'd work well there.

Let me explain. PyFFI has an option to create patches directly (--diff), and also an option to update nifs using patch files (--patch) - actually this was supposed to address the issues with the original patcher batch files. But I never really documented it properly, and never included xdiff or similar. Also, it only uses in/out (confusingly) - an extra "patches" folder as you suggest, would be much better.

So I think I'll upgrade this feature to do essentially what your batch scripts are doing (but it wouldn't rely on forfiles.exe and so on).

Just remember to credit H2DOK instead since they're his improved skip lists, not mine :)


Yes, will do!!

Aside, as it looks now, these updates won't be for 2.1.8, but more likely for 2.1.9, as I'm compelled to ship 2.1.8 already this weekend to release some longstanding bug fixes for the blender nif scripts, which require an updated pyffi.
User avatar
Maeva
 
Posts: 3349
Joined: Mon Mar 26, 2007 11:27 pm

Post » Tue May 03, 2011 1:23 am

can someone explain to me how to repack the BSA. When I did it the first time, and it crashed the game, I used "Add Folders" and selected the toaster/out folder. Should I be using the "Add Files" option, and selecting all of the 1000's of meshes?

If I should use the "Add Folders" , how "deep" into the folders should I go? (toaster/out) or (toaster/out/meshes)

Also, will it actually screw anything up if I move the files from the toaster/out folder to say, a new folder on my desktop?
User avatar
Hilm Music
 
Posts: 3357
Joined: Wed Jun 06, 2007 9:36 pm

Post » Tue May 03, 2011 8:57 am

You don't want to add just the out folder, PyFFI only puts the meshes it modified into that folder, not the ones it skipped. This might also explain your missing mesh trouble in Nehrim.

When repacking the BSA in OBMM, the folder you want to select is one folder above the meshes folder (so select the in folder and if it contains the meshes folder, the meshes will be properly packed).

Don't forget to re-pack your trees and DistantLOD (if not using TES4LODGen) folders either.
User avatar
Trish
 
Posts: 3332
Joined: Fri Feb 23, 2007 9:00 am

Post » Mon May 02, 2011 10:40 pm

You don't want to add just the out folder, PyFFI only puts the meshes it modified into that folder, not the ones it skipped. This might also explain your missing mesh trouble in Nehrim.

When repacking the BSA in OBMM, the folder you want to select is one folder above the meshes folder (so select the in folder and if it contains the meshes folder, the meshes will be properly packed).

Don't forget to re-pack your trees and DistantLOD (if not using TES4LODGen) folders either.

Alright, I managed to get my Nehrim Meshes PyFFied! Everything appears to be working nicely...for now at least.

So, when I Pyffi the oblivion meshes, do I have still have to worry about those http://sites.google.com/site/oblivionpoinfo/optimization/pyffi that have the added space?
User avatar
Lauren Graves
 
Posts: 3343
Joined: Fri Aug 04, 2006 6:03 pm

Post » Tue May 03, 2011 9:33 am

Only if you're on Vista/7. Even then, just don't touch the folders after PyFFI is finished with them and you won't have to deal with the leading spaces being truncated.

EDIT: In other news, the OP overhaul has begun it's trial run. If anyone has any questions or comments or additions that could be made please don't hesitate to ask.
User avatar
Bad News Rogers
 
Posts: 3356
Joined: Fri Sep 08, 2006 8:37 am

Post » Tue May 03, 2011 5:42 am

What's the latest version of PyFFI I can use with Python 2.6.5?
User avatar
Kristian Perez
 
Posts: 3365
Joined: Thu Aug 23, 2007 3:03 am

Post » Tue May 03, 2011 9:30 am

Only if you're on Vista/7. Even then, just don't touch the folders after PyFFI is finished with them and you won't have to deal with the leading spaces being truncated.

EDIT: In other news, the OP overhaul has begun it's trial run. If anyone has any questions or comments or additions that could be made please don't hesitate to ask.

Well, I followed everything to the letter, and after Pyffi'ing the Oblivion Meshes, my game is crashing when it loads a save. Just like before.

You have any idea what the problem could be?

I reset the timestamps, so the Meshes.BSA says 1/1/2006. I copied the Meshes.BSA from the Oblivion\Data directory to the Nehrim\Data directory. Could it be because Nehrim pulled the original Meshes out from the Oblivion\Data, and since I've replaced them, it's causing it to crash?
User avatar
Princess Johnson
 
Posts: 3435
Joined: Wed Feb 07, 2007 5:44 pm

Post » Tue May 03, 2011 1:41 am

So, when I Pyffi the oblivion meshes, do I have still have to worry about those http://sites.google.com/site/oblivionpoinfo/optimization/pyffi that have the added space?

I personally just manually deleted the leading spaces from those meshes (using XP) as Arthmoor's UOP updates modify those cave objects to look for the meshes that don't have the leading spaces.
User avatar
Taylrea Teodor
 
Posts: 3378
Joined: Sat Nov 18, 2006 12:20 am

Post » Tue May 03, 2011 4:38 am

If you're doing any modding at all, you'll want to try and keep those leading space files from getting deleted. The CS will not recognize that the UOPS has fixed this. Unpacking your BSA somewhere, doing PyFFI on them there, then repacking them will work. Just don't copy/paste them or you'll lose those. Again, this should only be an issue with Vista/Win 7 64bit.
User avatar
SiLa
 
Posts: 3447
Joined: Tue Jun 13, 2006 7:52 am

Post » Mon May 02, 2011 11:05 pm

New beta version (2.1.9 beta1) including freshly written recursive patch scripts is available:

https://github.com/amorilia/pyffi/downloads

Thanks to OnmyojiOmn and jonwd7 for their initial patcher scripts - I did not use any of their code (since the new ones are written in python, not in batch) but my code is obviously inspired by them.

Beware: use the new patch scripts only for testing - they are not suitable for "real life" use until more properly tested. For now, stick with jonwd7's patcher if you want to do real life patching, and not just testing. Aside, note that the patch_recursive_apply script will copy files from in to out if no patch file is present, so that should mitigate any issues with missing files, at least when using patches. Probably I should provide an option to do something similar with the other scripts.

The patches use xdelta3 (version z), which is now also bundled with the windows installer, so earlier patch sets (including those just uploaded recently by jonwd7) should in theory keep working, although I haven't actually tested this yet.

Quick instructions:
  • to create patch sets: copy originals into in/, run optimizer so updated are in out/, run patch_recursive_make.bat - the patch/ folder will fill with patches
  • to apply a patch set: copy originals into in/, unpack patch set into patch/, and run patch_recursive_apply.bat - the out/ folder will fill with updated files


If you find any issues, please report them on the tracker as usual:

http://sourceforge.net/tracker/?group_id=199269
User avatar
Charity Hughes
 
Posts: 3408
Joined: Sat Mar 17, 2007 3:22 pm

Post » Tue May 03, 2011 7:27 am

Thanks to OnmyojiOmn and jonwd7 for their initial patcher scripts - I did not use any of their code (since the new ones are written in python, not in batch) but my code is obviously inspired by them.

Beware: use the new patch scripts only for testing - they are not suitable for "real life" use until more properly tested. For now, stick with jonwd7's patcher if you want to do real life patching, and not just testing. Aside, note that the patch_recursive_apply script will copy files from in to out if no patch file is present, so that should mitigate any issues with missing files, at least when using patches. Probably I should provide an option to do something similar with the other scripts.

The patches use xdelta3 (version z), which is now also bundled with the windows installer, so earlier patch sets (including those just uploaded recently by jonwd7) should in theory keep working, although I haven't actually tested this yet.

I used the latest xdelta as well, at the highest compression level. I'm not sure what the size difference is between midway and maximum, but I needed to get the vanilla mesh patches as small as possible. And there was a pretty big difference between the min and max compression. It's unfortunate Windows does not have built-in Python support (*cough* OS X *cough*), because Python is my preferred language. I still want a patching solution free of additional installers and dependencies, but not as primitive as a batch file. Though if I don't come up with an executable at some point, I'd rather point people to patching with PyFFI itself, for the people who don't mind having to install Python and PyFFI to do it.
User avatar
koumba
 
Posts: 3394
Joined: Thu Mar 22, 2007 8:39 pm

Post » Tue May 03, 2011 10:26 am

What's the latest version of PyFFI I can use with Python 2.6.5?
The OP says "Python 2.6.6 (required to Run PyFFI)". But when I go to the link for documentation and look at the requirements, I don't see anything about Python 2.6.6.
I also looked at the readme.rst for 2.1.5, 2.1.6 and 2.1.7 and saw no mention of this requirement.

So I'm still looking for an answer to my original question.
What's the latest version of PyFFI I can use with Python 2.6.5? Thanks in advance.
User avatar
Lalla Vu
 
Posts: 3411
Joined: Wed Jul 19, 2006 9:40 am

Post » Mon May 02, 2011 9:06 pm

Python 2.6.6 will work fine. That's what I'm using. Just be sure you don't use the 64bit Python.
User avatar
Nuno Castro
 
Posts: 3414
Joined: Sat Oct 13, 2007 1:40 am

Post » Mon May 02, 2011 10:12 pm

I'm using whatever comes with the Python package for Wrye Bash found on Nexus. I think it's 2.6.5 -- it may be 2.6.6 though.
User avatar
Jonathan Windmon
 
Posts: 3410
Joined: Wed Oct 10, 2007 12:23 pm

Post » Tue May 03, 2011 2:47 am

I'm using whatever comes with the Python package for Wrye Bash found on Nexus. I think it's 2.6.5 -- it may be 2.6.6 though.
It was 2.6.5. Thanks!
User avatar
Donald Richards
 
Posts: 3378
Joined: Sat Jun 30, 2007 3:59 am

Post » Tue May 03, 2011 8:14 am

Ok, after gaining a little confidence after pyffing all the meshes in CNRP with a blank skiplist and not getting any freakiness, I decided to go all out and pyffied my Oblivion - meshes.bsa without a skiplist.

2 hours and counting ... nothing abnormal :D W00t PyFFI!
User avatar
Alex Blacke
 
Posts: 3460
Joined: Sun Feb 18, 2007 10:46 pm

Post » Tue May 03, 2011 3:38 am

Ok, after gaining a little confidence after pyffing all the meshes in CNRP with a blank skiplist and not getting any freakiness, I decided to go all out and pyffied my Oblivion - meshes.bsa without a skiplist.

2 hours and counting ... nothing abnormal :D W00t PyFFI!

Why on Earth would you do this? There are skip lists for a reason...
User avatar
Music Show
 
Posts: 3512
Joined: Sun Sep 09, 2007 10:53 am

Post » Tue May 03, 2011 3:03 am

Hey just did my Oblivion - Meshes BSA, 11 hours!

Are there any areas and items I can go and check that have known issues? Im seeing that helmets and the roots and stuff on the first page, but is there anything else(specific dungeons or towns or whatnot)?
User avatar
Charles Weber
 
Posts: 3447
Joined: Wed Aug 08, 2007 5:14 pm

Post » Mon May 02, 2011 7:31 pm

Hey just did my Oblivion - Meshes BSA, 11 hours!

Are there any areas and items I can go and check that have known issues? Im seeing that helmets and the roots and stuff on the first page, but is there anything else(specific dungeons or towns or whatnot)?

Just "coc testinghall" in the console. Go to the creature grove and check out the creatures. Go to the other rooms and check out all the clothing and clutter.
User avatar
Heather beauchamp
 
Posts: 3456
Joined: Mon Aug 13, 2007 6:05 pm

Post » Mon May 02, 2011 7:07 pm

I used the latest xdelta as well, at the highest compression level.

Hmmm, I'm using the default (no extra settings), is that maximal compression?

(EDIT: ok, just did some testing, and it isn't; I'll update the next beta release to create patches with maximal compression)


I still want a patching solution free of additional installers and dependencies, but not as primitive as a batch file.

Excellent point - there are two types of users:
  • Those who want to *make* patches will want pyffi anyway and it is easy for them not to need an extra tool.
  • Those who want to *apply* patches. For these users, installing the whole toolchain is complete overkill, and having a simple batch file that does the trick is great (easy to maintain, and it's open source by definition :) ).


Ok, after gaining a little confidence after pyffing all the meshes in CNRP with a blank skiplist and not getting any freakiness, I decided to go all out and pyffied my Oblivion - meshes.bsa without a skiplist.

2 hours and counting ... nothing abnormal :D W00t PyFFI!


Wonderful! Be sure to check that mothernightstatuebase.nif, people keep reporting that this doesn't work (I've never actually observed the problem myself, but maybe I don't play the right factions... :) ). If you (or anyone) sees an error, and can track down the cause down to nif level, then that would be absolutely awesome.
User avatar
jasminε
 
Posts: 3511
Joined: Mon Jan 29, 2007 4:12 am

Post » Tue May 03, 2011 7:34 am

Wonderful! Be sure to check that mothernightstatuebase.nif, people keep reporting that this doesn't work (I've never actually observed the problem myself, but maybe I don't play the right factions... :) ). If you (or anyone) sees an error, and can track down the cause down to nif level, then that would be absolutely awesome.

That's great and all that he played a whole two hours with no issue, but those skip lists have been built up over many years of testing. Two hours of testing shouldn't negate the years of multiple input from other people who have had actual problems.

I would think about this person differently if he/she PyFFIed everything and then intentionally encountered the problem NIFs that everyone reports, in some kind of scientific manner. (Not just running around the game world, probably NOT encountering the problem NIFs)

I know PyFFI continues to improve, but I don't see why somebody should ignore a skip list. It's not like PyFFIing those things will bring any additional improvements.


Hmmm, I'm using the default (no extra settings), is that maximal compression?

No, you have to use -9 in place of whatever the letter is for the patch flag (-e maybe?)... It takes a bit longer to make patches this way, but not to apply them. You may want to try a middle-of-the-road setting like -5.
User avatar
Sun of Sammy
 
Posts: 3442
Joined: Mon Oct 22, 2007 3:38 pm

Post » Mon May 02, 2011 8:57 pm

That's great and all that he played a whole two hours with no issue, but those skip lists have been built up over many years of testing. Two hours of testing shouldn't negate the years of multiple input from other people who have had actual problems.

I would think about this person differently if he/she PyFFIed everything and then intentionally encountered the problem NIFs that everyone reports, in some kind of scientific manner. (Not just running around the game world, probably NOT encountering the problem NIFs)

I know PyFFI continues to improve, but I don't see why somebody should ignore a skip list. It's not like PyFFIing those things will bring any additional improvements.
My friend, I'd like to remind you that:
1) I'm not encouraging anyone to do this
2) I've seen that amorilla immediately fixes any specific problems reported to the sourceforge tracker

I do have my old (skiplisted) pyffi 2.1.7 files on backup - just in case. But I also DO intend to report anything specifically nasty that comes up in my un-skiplisted meshpack. Afaict, amorilla has repeatedly said that its perfectly safe to run pyffi wothout the skiplist provided any anomalies are reported - which I DO intend to follow :)

Thank you for your time!
User avatar
Kate Murrell
 
Posts: 3537
Joined: Mon Oct 16, 2006 4:02 am

Post » Tue May 03, 2011 2:18 am

No, you have to use -9 in place of whatever the letter is for the patch flag (-e maybe?)... It takes a bit longer to make patches this way, but not to apply them. You may want to try a middle-of-the-road setting like -5.


Thanks, I also just found out about that, and the update will be included in the next beta release.

Indeed, Gratis_monsta is one of the regular (and well appreciated!!!) testers - but uses another alias on sourceforge whence perhaps the cause of confusion. Anyway, the skip list serves two purposes really:
  • To prevent optimizations to go bonkers if run on mods that have nifs with egm or tri files in vanilla, but that don't ship it (so pyffi doesn't see it when optimizing that it cannot change the vertex ordering in these nifs).
  • To prevent optimizations to go bonkers on any other nifs.


In an ideal world, there wouldn't be any nifs in the latter category, so, in that world, you can safely pyffi without skip list, provided you are running it on a vanilla unpacked bsa. Of course, the world isn't ideal (yet).
User avatar
stevie critchley
 
Posts: 3404
Joined: Sat Oct 28, 2006 4:36 pm

Post » Tue May 03, 2011 4:23 am

Just "coc testinghall" in the console. Go to the creature grove and check out the creatures. Go to the other rooms and check out all the clothing and clutter.


Thanks, Ill check it out.
User avatar
Bellismydesi
 
Posts: 3360
Joined: Sun Jun 18, 2006 7:25 am

Post » Tue May 03, 2011 8:54 am

Took a quick look over Bravil's Lucky Old Lady and there seems to be nothing wrong with her base :) Some pics (cropped for bandwidth)
http://img163.imageshack.us/i/screenshot13mt.png/
http://img822.imageshack.us/i/screenshot14p.png/

I double checked with FormID finder to confirm that the mesh wasn't a different one (I have Blood&Mud installed)

Also took jonwd7's tip and "coc testinghall"ed and tried out some items - nothing extraordinary. Went to the creature farm and had a HUGE melee (maybe MMM/FCOM at work there) but nothing special :P Also confirmed that the rat underbelly seam is now almost gone (only shows up when you have the rat close to you and have a torch in your hand)
User avatar
Dalley hussain
 
Posts: 3480
Joined: Sun Jun 18, 2006 2:45 am

PreviousNext

Return to IV - Oblivion