[RELz] Oblivion - PyFFI Optimization Kit

Post » Wed Mar 30, 2011 9:28 am

@andalaybay

The advantage of this kit is
    to be able to apply PyFFI in a large set of files or complex directory structure,
    with the safety of the skip list,
    not bypassing meshes that could be optimized,
    not bypassing meshes that could be checked for other PyFFI fixes,
    and applying several optimization passes at once(case of _far.nif).


Lets see if I can understand your point of view:

When applying PyFFI over meshes I'm selecting individually, I don't really need the skip list unless I'm not sure if they should be skipped at all. Provided the mesh has a sensible name of its type to compare to the list - i.e. helmet.nif - it may not be worth optimizing but checked for some particular operation - like fix_texturepath or similar. PyFFI is getting better each version - hopefuly to allow me to stop worrying about such cases. Nevertheless I see the usefulness of a customized optimization here.

When applying PyFFI on a folder with a complex structure, what you'd like to do is to let the meshes be skipped appropriately on the spot. For this, the only difference being the place where the folders are, I can try to make a custom right-mouse button command.

However, there may be a time when you want to do some specific optimization and the standard PyFFI command is still preferred.

Don't forget about testing and how the PyFFI project can progress to help us mortal mod users let go of all this care. :foodndrink:


Actually in my case, the issue is that I don't have the first clue as to what PyFFI does, I just know that it's something I should do :D So I run the process on all my meshes when I release a new version of my mod. But often I just create a new mesh and want to optimize that specifically as I add it to my mod. So being able to right-click and select optimize is easier than copying to the in folder. Are you saying that I shouldn't be using your four-pass optimization against a single file? From my point of view, I would just switch to using your scripts for all my optimizations. I don't want to worry about the details. I'm not a modeller, so I don't really understand all the details of NIF's. I'm pretty good at collision and physics, so I understand those parts of a NIF, but telling me that I should remove unused verticals or something like that is meaningless :) I would say that if something is unused, it should be removed, but I wouldn't be in a position to judge the impact of doing that - other than if seeing the results in game. :lol:

Not sure if it helps any, but most of my work has been to take vanilla meshes, rescale them and convert the collision from static to havoked. Or to create a new static collision mesh for the smaller version of the object. So I want to optimize those meshes because by the time I'm done with them, they probably need a good cleaning :)
User avatar
Prohibited
 
Posts: 3293
Joined: Tue Jun 12, 2007 6:13 am

Post » Wed Mar 30, 2011 5:57 am

I'm CTDing after running the oblivion-meshes through V6
I'm sure it's something I'm doing wrong
Python2.6.6 - PyFFI 2.19b5, PFOK V6

1) unpacking the entire bsa to the "in" folder with OBMM
Got 3 folders: Distant Lod, Meshes, Trees
2) run run_oblivion_optimize.bat
3) Copy over the meshes folder inside the "out" folder back to the "in" folder
4) Opennig BSA Creator in OBMM
5) Adding the 3 folders Distant Lod, Meshes, Trees
6) Saving it uncompressed as oblivion-meshes.bsa

Wish there were a kind of logging so I can see what cause CTD
User avatar
Elizabeth Falvey
 
Posts: 3347
Joined: Fri Oct 26, 2007 1:37 am

Post » Wed Mar 30, 2011 11:24 am

4) Opennig BSA Creator in OBMM
5) Adding the 3 folders Distant Lod, Meshes, Trees


There's your problem, if I'm not wrong. In BSA Creator you need to add folder "In" that contains that three folders.

Edit: I chose Compress all: Medium. What's the difference when choosing compression level?

Edit2: I have optimised Oblivion Meshes BSA with version downloaded on March 8th. Took me 20 h to finish. (8 years old PC, yeah). Do I really need to repeat the process or do the latter updates just improve optimised meshes even further?

http://www.tesnexus.com/downloads/file.php?id=36882has nice detailed instructions of the process including options for creating BSA. Maybe you (Ulrim) could also include them modified into your readme?
User avatar
Rachel Eloise Getoutofmyface
 
Posts: 3445
Joined: Mon Oct 09, 2006 5:20 pm

Post » Wed Mar 30, 2011 10:41 am

Edit: I chose Compress all: Medium. What's the difference when choosing compression level?

Many choose to not compress BSAs on the sound theory that the game can load the meshes contained in them into memory faster since they don't go through a decompression process. The one caveat is that the game limits how big a BSA can be, so that in certain cases one must break what was one BSA into several BSAs. But that won't be an issue with stock Oblivion meshes BSA, unless you're doing exotic modifications in addition to PyFFi.
User avatar
Katharine Newton
 
Posts: 3318
Joined: Tue Jun 13, 2006 12:33 pm

Post » Wed Mar 30, 2011 1:43 pm

Seems logical. Thanks for the answer.
User avatar
Valerie Marie
 
Posts: 3451
Joined: Wed Aug 15, 2007 10:29 am

Post » Wed Mar 30, 2011 5:23 am

DOS box (16 bit MS-DOS Subsystem) saying: 'Oblivion - PyFFI Optimization KitC:\PROGRA~1\Symantec\S32EVNT1.DLL. An installable Virtual Device Driver failed Dll initialization. Choose 'Close' to terminate the application.'

It seems you are having an issue with Norton software.
Check out these links, maybe they can help:
http://service1.symantec.com/SUPPORT/sunset-c2002kb.nsf/f7d6cb402f0b760785256ee600549628/136bd9321cd5f66185256edd00478e08?OpenDocument&src=http://forums.bethsoft.com/index.php?/topic/1172029-relz-oblivion-pyffi-optimization-kit/bar_sch_nam from Symantec Support
http://forums.cnet.com/7723-6142_102-64327.html in CNET forum
http://www.computing.net/answers/windows-xp/16-bit-windows-subsystem/125464.html in Computing.net by tom's guide

Just measured time i takes optimizing with PyFFI 2.1.9b5 on my machine - it takes roughly 2 hours.

Actually in my case, the issue is that I don't have the first clue as to what PyFFI does, I just know that it's something I should do :D

Perfectly understood. This is why it matters to me to study what commands must be used so the optimization can be as straightforward as fire-and-forget. I'll see what I can do with the right mouse button menus.


I'm CTDing after running the oblivion-meshes through V6I'm sure it's something I'm doing wrongPython2.6.6 - PyFFI 2.19b5, PFOK V61) unpacking the entire bsa to the "in" folder with OBMMGot 3 folders: Distant Lod, Meshes, Trees2) run run_oblivion_optimize.bat3) Copy over the meshes folder inside the "out" folder back to the "in" folder

I'm about to start up my last bsa test with PyFFI 2.1.9b5 optimization. This time I'm going to document it thoroughly so I can report whatever happens in the PyFFI bugtracker. I was running with a bay horse arround and when I crossed the imperial city bridge we just passed through the bridge and fell in the water... That was odd! I was using a bsa with meshes optimized with 2.1.7 and then 2.1.9b5 on top. Now I have a clean default bsa only optimized with 2.1.9b5.

Also, you might want to check the end of my first post on this thread. There are some files with leading space in the file name, namely:
meshes\dungeons\caves\exterior\ centrancechighlandsm01.nifmeshes\dungeons\caves\exterior\ centrancegoldcoastsm01.nifmeshes\dungeons\caves\exterior\ centrancegreatforestsm02.nifmeshes\dungeons\caves\exterior\ centrancemtnsnowlg01.nifmeshes\dungeons\caves\exterior\ centrancemtnsnowlg02.nifmeshes\dungeons\caves\exterior\ centrancemtnsnowlg03.nifmeshes\dungeons\caves\exterior\ centrancemtnsnowsm01.nifmeshes\dungeons\caves\exterior\ centrancemtnsnowsm02.nifmeshes\dungeons\caves\exterior\ centrancemtnsnowsm03.nifmeshes\dungeons\caves\exterior\ centrancerockmosssm01.nifmeshes\dungeons\caves\exterior\ centrancerockybeachsm01.nifmeshes\dungeons\caves\exterior\ centrancewestwsm01.nifmeshes\dungeons\caves\exterior\ centrancewestwsm02.nif


If using windows 7, when you move these files from the "out" folder to the "in" folder, the leading space is pruned.
This means you will end up with two versions of the files with leading spaces inside the "in" folder - the original (unoptimized) files with leading spaces and the ones from the "out" folder (which do not overwrite the respective original because Windows prunes the leading space).

Took me 20 h to finish. (8 years old PC, yeah). Do I really need to repeat the process or do the latter updates just improve optimised meshes even further?http://www.tesnexus.com/downloads/file.php?id=36882has nice detailed instructions of the process including options for creating BSA. Maybe you (Ulrim) could also include them modified into your readme?

Optimization result is the same from version 3 to version 6 of the Kit. I may have to check for collision safeguards in next version because I suspect PyFFI is breaking it since 2.1.7.

About the PyFFI patcher, I may be able to share the patches so you do not have to build it for so long. About the bsa creation method - there's plenty of documentation namely on the site http://tesivpositive.animolious.com/?page=pyffi but I'll read that too.

The one caveat is that the game limits how big a BSA can be, so that in certain cases one must break what was one BSA into several BSAs. But that won't be an issue with stock Oblivion meshes BSA, unless you're doing exotic modifications in addition to PyFFi.

I don't understand the limits people tell about them. For what is worth, I'm using an uncompressed 'Oblivion - Textures - Compressed.bsa' with about 2GB without issues (perhaps worth taking into account that I'm also using a 64 bit system). There may be issues with larger BSAs - however, that is an unlikely amount of files to pack together!
User avatar
Crystal Birch
 
Posts: 3416
Joined: Sat Mar 03, 2007 3:34 pm

Post » Wed Mar 30, 2011 12:16 pm


I'm about to start up my last bsa test with PyFFI 2.1.9b5 optimization. This time I'm going to document it thoroughly so I can report whatever happens in the PyFFI bugtracker. I was running with a bay horse arround and when I crossed the imperial city bridge we just passed through the bridge and fell in the water... That was odd! I was using a bsa with meshes optimized with 2.1.7 and then 2.1.9b5 on top.

Sounds like my Imperial Prison issue.
User avatar
Killer McCracken
 
Posts: 3456
Joined: Wed Feb 14, 2007 9:57 pm

Post » Wed Mar 30, 2011 2:34 am


(...)
This means you will end up with two versions of the files with leading spaces inside the "in" folder - the original (unoptimized) files with leading spaces and the ones from the "out" folder (which do not overwrite the respective original because Windows prunes the leading space).


I saw the files with leading space but didn't check if they were some pruned doubles in the folder.
Seems it's a bad idea to do it in the morning before my second coffee :tongue:
User avatar
jenny goodwin
 
Posts: 3461
Joined: Wed Sep 13, 2006 4:57 am

Post » Wed Mar 30, 2011 2:52 am

Sounds like my Imperial Prison issue.

http://www.youtube.com/watch?v=q2o_V48IfQ0 - just dropped a message in the PyFFI bug tracker. Seems the latest versions do awkward things to mesh collision.

I'm going to test disabling the mesh collision optimizations while keeping the bug fixes using PyFFI 2.1.9b5.
User avatar
Romy Welsch
 
Posts: 3329
Joined: Wed Apr 25, 2007 10:36 pm

Post » Wed Mar 30, 2011 2:43 pm

I am using PyFFI 2.1.9b5 latest exe with Python 2.6.6. I PyFFIed Oblivion - Meshes.bsa and DLCShiveringIsles - Meshes.bsa and ran Oblivion to check how it worked out. As soon as I made my character and left the screen to the beginning prison, fell through the floor. I did delete the files with leading spaces.

Also, so that others might not make the same mistake I did, don't install the kit inside the Oblivion game folder. I had originally installed it under a 'Utilities' folder I created in the Oblivion\data folder. It will repath the meshes and other files improperly (Hence why the OP probably suggested to install at C: root folder). Is anyone else falling through floors right at the beginning or is there something else I need to configure?
User avatar
SUck MYdIck
 
Posts: 3378
Joined: Fri Nov 30, 2007 6:43 am

Post » Wed Mar 30, 2011 2:42 pm

http://www.youtube.com/watch?v=q2o_V48IfQ0 - just dropped a message in the PyFFI bug tracker. Seems the latest versions do awkward things to mesh collision.

I'm going to test disabling the mesh collision optimizations while keeping the bug fixes using PyFFI 2.1.9b5.


Yes, I'm having similar issues - I think you saw my post in the PyFFI thread. They suggested I add something to the bug tracker, so I'll do that. I wasn't sure where the problem lay, but it looks like it's with PyFFI.
User avatar
Chloe Mayo
 
Posts: 3404
Joined: Wed Jun 21, 2006 11:59 pm

Post » Wed Mar 30, 2011 4:21 am

Question: Why was the _far.nif ini file changed the way it's been changed? That spell as far as I know will not remove any of the things it needs to remove in order to produce a properly cleaned _far.nif file.

The function you're using now in v6 is meant to generate new ones when you don't have any rather than be used to optimize existing ones. I fail to see how it's going to do much of anything.
User avatar
liz barnes
 
Posts: 3387
Joined: Tue Oct 31, 2006 4:10 am

Post » Wed Mar 30, 2011 10:15 am

I fail to see how it's going to do much of anything.

I got confused at a comment stating PyFFI spell opt_cleanfarnif should be used alongside optimize for best results.

After digging into some python code I understand that redundancy now.

I still don't understand why opt_cleanfarnif has to call optimize list of spells afterwards to produce good _far.nif meshes (it's exactly what modify_makefarnif spell does anyway, without the texture modification). But that is another question.

Since I've used version 2.1.6 in the past with good results I am going to suggest PyFFI 2.1.6 for the moment or (with very few tests I've run myself so far) version 2.1.7.
User avatar
Victoria Bartel
 
Posts: 3325
Joined: Tue Apr 10, 2007 10:20 am

Post » Wed Mar 30, 2011 12:37 am

I'll report that *something* was wrong in my game after optimisation. First, I tried it in Anvil, both in city and around it, later I tried another save in Skingrad cathedral. Everything was fine.
But when I tried to exit in Skingrad - CTD!
Then, I tried fast travelling to Skingrad - CTD!
Then, I tried riding from Anvil toward Skingrad, but after I passed Brina's Cross inn - CTD.

It isn't mod related because I tried without mods and after I restore original BSA CTD was gone.


If no one has any clue, I'll wait for patches because optimising was took too long on my PC.
User avatar
Ells
 
Posts: 3430
Joined: Thu Aug 10, 2006 9:03 pm

Post » Wed Mar 30, 2011 2:02 am

I got confused at a comment stating PyFFI spell opt_cleanfarnif should be used alongside optimize for best results.

After digging into some python code I understand that redundancy now.

I still don't understand why opt_cleanfarnif has to call optimize list of spells afterwards to produce good _far.nif meshes (it's exactly what modify_makefarnif spell does anyway, without the texture modification). But that is another question.

Since I've used version 2.1.6 in the past with good results I am going to suggest PyFFI 2.1.6 for the moment or (with very few tests I've run myself so far) version 2.1.7.


Yeah, I was using 2.1.6 without any issues. So I can use 2.1.6 with your optimizations?

I'll report that *something* was wrong in my game after optimisation. First, I tried it in Anvil, both in city and around it, later I tried another save in Skingrad cathedral. Everything was fine.
But when I tried to exit in Skingrad - CTD!
Then, I tried fast travelling to Skingrad - CTD!
Then, I tried riding from Anvil toward Skingrad, but after I passed Brina's Cross inn - CTD.

It isn't mod related because I tried without mods and after I restore original BSA CTD was gone.


If no one has any clue, I'll wait for patches because optimising was took too long on my PC.


Washington, this sounds an awful lot like the problems Arthmoor was having with one of the releases of RAEVWD. I don't think he ever really found out what the problem was, but there did seem to be a corrupted mesh in there somewhere. Perhaps you and he can compare which version of PyFFI you're using...
User avatar
Leticia Hernandez
 
Posts: 3426
Joined: Tue Oct 23, 2007 9:46 am

Post » Tue Mar 29, 2011 11:22 pm

I never did figure out what was causing it. I can still fast travel anywhere I please. Same thing happened with one of the Elsweyr updates but then it stopped happening and nobody ever figured out why - only that it was somehow related to VWD information. The thing that made it most puzzling is that with Elsweyr there hadn't been any re-use of PyFFI on any of the meshes. Things just broke, then just as mysteriously started working again.

The last RAEVWD update that went out had all been reoptimized with 2.1.8. It's the only substantial use of it I've had and my game is perfectly stable with those meshes. Though again, with Brumbek's updated files, something "just broke" and was giving me crap. Then somehow it just worked again. So I don't know WTF is going on.

All I can say for sure is that the last major reopimization of the main meshes I did was with 2.1.7 and everything has been fine. I hadn't seen any reason to do it all again just because 2.1.8 was out.
User avatar
Nana Samboy
 
Posts: 3424
Joined: Thu Sep 14, 2006 4:29 pm

Post » Wed Mar 30, 2011 2:32 am

Similar issues for me. All my meshes were optimized with 2.1.6, which had no reports of problems. But I kept getting sporadic reports of problems with furniture falling through the floor, usually with mod added houses. With IFR, when you activate the marker that you've positioned on the floor, it creates a havoked version of the furniture item and drops it where the marker was. Well in my current game, I had a single bed fall through the floor in the Imperial City shack every time I tried to activated the marker. So I optimized the meshes with 2.1.9 b5. Well that stopped the bed from falling through the floor but broke the static meshes. So I reinstalled the meshes that had been optimized with 2.1.6 and you guessed it: things are fine again :shrug: Now the only thing was that I had a BSA with most of the meshes in it and a few loose meshes from the latest round of development. So when I reverted back to the previous version of the meshes, I actually re-installed the whole mod, so everything was in a BSA again. I wouldn't think it would matter, but something very odd is going on.
User avatar
Casey
 
Posts: 3376
Joined: Mon Nov 12, 2007 8:38 am

Post » Wed Mar 30, 2011 1:05 pm

So I optimized the meshes with 2.1.9 b5. Well that stopped the bed from falling through the floor but broke the static meshes. So I reinstalled the meshes that had been optimized with 2.1.6 and you guessed it: things are fine again

Testing shows the only problem so far is related to static mesh collision. The version 2.1.9b5 has a few bug fixes and Amorilia said only the mopper.exe is causing that problem.

This binary contains the havok code and she says it's closed source (for what I understood of her remark, cannot be modified). I know she used to compile it so at least she had to have access to the code. Maybe I'm wrong, but it spits out a BSD type license when you use:
C:\Python26\Lib\site-packages\pyffi\utils\mopper.exe --license


BTW, Arthmoor, I think I forgot to answer your question properly before. _far.nif meshes are being optimized in the first pass and then re-optimized in the second. There is a redundancy of optimization (2 passes) so that can be tweaked for the next version.
User avatar
Sarah Bishop
 
Posts: 3387
Joined: Wed Oct 04, 2006 9:59 pm

Post » Wed Mar 30, 2011 7:35 am

I'm a bit confused. I had originally thought that one phase of your PyFFi optimization process included the "spell" that strips out unneeded data from far NIFs. (Strips out Specular Flags?) Is that not the case? Also, right now I have PyFFi 2.1.9b5 installed. I take it that's not safe? What's the most recent stable version of PyFFi you recommend?
User avatar
Invasion's
 
Posts: 3546
Joined: Fri Aug 18, 2006 6:09 pm

Post » Wed Mar 30, 2011 4:16 am

I'm a bit confused. I had originally thought that one phase of your PyFFi optimization process included the "spell" that strips out unneeded data from far NIFs. (Strips out Specular Flags?) Is that not the case? Also, right now I have PyFFi 2.1.9b5 installed. I take it that's not safe? What's the most recent stable version of PyFFi you recommend?


The PyFFI Optimization Kit configuration files instruct PyFFI to process the _far.nif meshes in the following way:

1) In the first pass, all _far.nif meshes are optimized.
2) In the last pass, all _far.nif meshes are cleaned and re-optimized.

I later learned (by inspecting the python code) that this behaviour is redundant. I'll make the necessary changes with an update soon.

A problem was found in the invocation of mopper.exe (the code that is used related to havok).
The tests I've run so far indicate that a large chunk of the resulting meshes are being pruned with this bug - likely the part where mopper is doing collision processing.
I could upload a "fix" for this problem, so you can successfully use PyFFI 2.1.9b5 with all its updated bug fixes while keeping collision optimization intact.

However, Amorilia is still the ultimate authority here since it's all her work and I'm sure there will be an update to the official PyFFI soon. I'm a mere layman figuring out how to use this thing right.
User avatar
Ownie Zuliana
 
Posts: 3375
Joined: Thu Jun 15, 2006 4:31 am

Post » Wed Mar 30, 2011 1:02 pm

(...)
Skingrad - CTD!
(...)


That's interesting.
When I was reporting CTD after optmizing oblivion-meshes.bsa, I was actually doing my test around Skingrad west gate.
Didn't try fadt travel cause I thought I did something wrong


I had some fell through meshes at Frostcrag Spire entrance (the first stairs next to the doors).
I'm going to wait for a Pyffi fix
User avatar
Adriana Lenzo
 
Posts: 3446
Joined: Tue Apr 03, 2007 1:32 am

Post » Wed Mar 30, 2011 2:06 am

I was using this version of PyFFI: PyFFI-2.1.8.37ae115.win32.
User avatar
Catherine Harte
 
Posts: 3379
Joined: Sat Aug 26, 2006 12:58 pm

Post » Wed Mar 30, 2011 2:42 pm

PyFFI 2.1.8 and 2.1.9b5 are both affected by the collision mesh bug.

Updated the Optimization Kit and uploaded a possible fix.


--

PyFFI 2.1.9 beta 6 available now with the bug fix - check first post for the download link.
User avatar
Nicholas C
 
Posts: 3489
Joined: Tue Aug 07, 2007 8:20 am

Post » Wed Mar 30, 2011 2:24 am

PyFFI 2.1.8 and 2.1.9b5 are both affected by the collision mesh bug.

Updated the Optimization Kit and uploaded a possible fix.
--
PyFFI 2.1.9 beta 6 available now with the bug fix - check first post for the download link.

Thanks for the kit update and heads-up on PyFFi 2.1.9-beta6. I had just installed the kit's fix and ran a few meshes for testing when your PyFFi announce appeared. I assume upgrading to 2.1.9b6 does exactly what your fix did?

I used meshes from an old version of Castle Dunkerlore as my test guinea pig. Was surprised how much several of its far NIFs got reduced. Then again Dunkerlore might be one of those mods that simply renames .nif's as _far.nif's without much if any optimizing.

You say your Optimization batch file includes opt_cleanfarnif's ability to strip out specular flags from far_nif's? I ask because I have been told those flags might well be the source of visual anomalies encountered in my current Oblivion installation.

Thanks again!
User avatar
Sam Parker
 
Posts: 3358
Joined: Sat May 12, 2007 3:10 am

Post » Wed Mar 30, 2011 8:26 am

FYI - 2.1.7 is also safe and is preferred over using 2.1.6. My entire vanilla BSA got reprocessed using that and so far no collision issues have cropped up.

Good though that the new version is getting things sorted out better. Still not sure why Amorilia said the collision stuff wasn't active because I didn't do anything special to 2.1.7 and it optimized all the collision just fine.
User avatar
mimi_lys
 
Posts: 3514
Joined: Mon Apr 09, 2007 11:17 am

PreviousNext

Return to IV - Oblivion