TESIV:POSItive -Performance & Optimization Info : IV

» Tue Jan 25, 2011 7:16 pm

Previous threads: http://www.gamesas.com/bgsforums/index.php?showtopic=1014947
Next thread: http://www.gamesas.com/index.php?/topic/1091580-tesivpositive-performance-optimiation-stabilization-information/
http://www.gamesas.com/index.php?/topic/1083993-in-depth-guide-for-installing-fcom-and-non-fcom-setups-with-bain/page__view__findpost__p__15794917__fromsearch__1
http://www.gamesas.com/index.php?app=forums&module=forums§ion=findpost&pid=15859364

http://sites.google.com/site/oblivionpoinfo/ --Performance, Optimization & Stabilization Information

http://sites.google.com/site/oblivionpoinfo/install/pcspecs
http://sites.google.com/site/oblivionpoinfo/optimization
http://sites.google.com/site/oblivionpoinfo/stabilization

Welcome to TESIV:POSItive!
Welcome!!

About this site

I was working on a guide by Blade of Mercy in order to try to smooth the last one-to-a-couple-hours crash from my game. When I came across steps involving, PyFFI, a Python program, and checked it out and found myself hesitant to proceed. Using PyFFI to optimize meshes definitely helps improve FPS, so explored for more answers. I asked in the PYFFI Automation Tool thread, and the author, Dan (Kandiedan), warned against inexperienced users doing this themselves because you need to know how to check for bad meshes. Well, soon after, other experienced modders assuaged a number of my concerns. Now I want to spread information about PyFFI and other ways to optimize and stabilize Oblivion setups. This site covers topics from the basics of installing Oblivion to the significance of optimizing meshes.

If you have suggestions for improvements or content additions, please drop a post in the site's thread.
- Tomlong75210

What is here?

This site is for the mod user. It covers topics from the installation and uninstallation of Oblivion to PyFFI-optimization and BSA repacking. The whole site is a walk-through from pre-Oblivion to running fully modded Oblivion. There are libraries for common problem and real load orders. If there is an aspect of running Oblivion that is not covered here, please recommend the missing topic in the site's thread. This site also has guides for patching mods and importing characters faces. Although there are probably links to point you in the right direction for other mod-related subjects, the whole site is dedicated to building a stable and optimized Oblivion setup. That is all.

How to navigate...

Most of the spine, the main pages, of this side have a guide-like format, with previous and next steps. The sidebar mirrors this structure. I recommend following the guide in its entirety, starting from the uninstallation step for non-first-timers, if you are starting a new Oblivion setup. Experienced users should know which parts they can skip by the pages' titles and introductory paragraphs. There are also more compact guides available on this site and linked guides, which you can find at the bottom of this page. For those visiting already running a setup you want to improve or check against example setup, the "Tuning Oblivion" page and onward contain most of the more sophisticated information. The guide-like presentation of the guide stops at "Troubleshooting."

Site Structure

* Welcome to TESIV:Positive --introduction to the site
* Intro to Modded Oblivion --introduction to mods
* Installing Oblivion --steps for clean uninstallation and installation of Oblivion
* Preparing Oblivion for Mods --acquiring utilities and configuring them
* Downloading Mods --where and how to download mods
* Installing Mods --what are the installation options? how to repackage mods
* Tuning Modded Oblivion --plugin cleaning and other more advanced maintenance
* Finishing the Installation --load ordering, building the bashed patch
* Running Oblivion --first run guide, tips
* Troubleshooting --crashes & problems library, getting help, helping yourself
* Stabilization --methods, related mods
* Optimization --methods, related mods
* Walkthroughs --compact modded Oblivon setup guides, step-by-step patching
* Lists --collections of various resources on and off this site
* Links --all related resources on the site (on the page), direct links listed in the sidebar
* Acknowledgments --the not so little people who have made great contributions to this site
* Contacts --current and old threads, my contact info
* Sitemap --a tool to help the user navigate this site

http://sites.google.com/site/oblivionpoinfo/optimization/pyffi
http://pyffi.sourceforge.net/

First, make sure to download the latest version. After installing PyFFI, when you right-click there is now an "Optimize with PyFFI" option. A number modders have recommended avoiding the creature and character folder, and also files with EGM files (i.e., certain helmets and head meshes.) Bad meshes, generated by PyFFI or by hand, can cause problems, that is why it is important to check meshes after running PyFFI. The process (when done thoroughly) may be very time-consuming, but it has been sped up a bit with the Python 2.6. Why do this? ...for increased FPS and more stability!!!!! The later versions of PyFFI heed the skip list in the optimization INI and skip EGMs by default. The INIs are located in \Program Files\PyFFI\Toaster by default. Remember to set the path to your Python folder (usually first level on your local disk) in the default INI.

There are two ways to use PyFFI to optimize. The first method is to right-click on a folder and choose "Optimize with PyFFI," but that modifies the original files. That is fine when dealing with small numbers of meshes, or if you are not worried about backup. However, there is an a way to optimize with PyFFI such that it creates backups of the modified files for you. UPDATE: At this point, it is fairly safe to use the right-click method. However, it is safer to optimize mods individually as opposed to right-clicking and optimizing your entire installed Meshes folder.

"Run PyFFI" Steps

1. Drop folders with meshes to be PyFFI-optimized in the "in" folder.
>> notes: PyFFI preserves the file structure of the "in" folder, so you can PyFFI multiple mods and folders at once.
2. Right-click on the optimize INI and choose "Run PyFFI"
3. Wait until PyFFI finishes, then get the PyFFI-optimized meshes from the "out" folder.

\PyFFI\utilities\toaster

inoutdefault.inioblivion_optimize.ini

The Vanilla meshes with leading spaces (to watch out for when you repack your BSA)

meshes\dungeons\caves\exterior\ centrancechighlandsm01.nifmeshes\dungeons\caves\exterior\ centrancegoldcoastsm01.nifmeshes\dungeons\caves\exterior\ centrancegreatforestsm02.nifmeshes\dungeons\caves\exterior\ centrancemtnsnowlg01.nifmeshes\dungeons\caves\exterior\ centrancemtnsnowlg02.nifmeshes\dungeons\caves\exterior\ centrancemtnsnowlg03.nifmeshes\dungeons\caves\exterior\ centrancemtnsnowsm01.nifmeshes\dungeons\caves\exterior\ centrancemtnsnowsm02.nifmeshes\dungeons\caves\exterior\ centrancemtnsnowsm03.nifmeshes\dungeons\caves\exterior\ centrancerockmosssm01.nifmeshes\dungeons\caves\exterior\ centrancerockybeachsm01.nifmeshes\dungeons\caves\exterior\ centrancewestwsm01.nifmeshes\dungeons\caves\exterior\ centrancewestwsm02.nif

Files that PyFFI cannot handle. Add the part in quotes to your skip list.

\meshes\architecture\statue\nightmotherstatuebase.nif - "nightmotherstatuebase"

Further optimization with PyFFI for _FAR NIF files ONLY! --Vorians

"C:\Program Files\Python26\python.exe" "C:\Program Files\Python26\Scripts\niftoaster.py" fix_deltangentspace Meshes

See the http://www.gamesas.com/index.php?/topic/1082438-inforelz-pyffi-python-file-format-interface/ for more information and help.

--Acknowledgements--
...Thank you!!

Albertine --helped notice a lot of little mod bugs
Arthmoor --too much to list here, including: tons of load order help, INI suggestions, mod suggestions, a mod-specific list
Bethesda --need I say anything...Elderscrolls, Morrowind, Oblivion...Thank you...
Blade of Mercy --Inspiration to create this site, 50 Steps Crash-Free Setup Guideline and much, much more
bg2408 --a lot of troubleshooting help and conflict tips
CorePC -- too much to list here, including much crashes, solutions and problems information
Camaro-69_327 --helped a ton to make this site's content clearer and more friendly for new users
Dan (kandiedan) -- PyFFI Automation Utility
display name is already in use -- Helped me with countless load order issues
Peter ID --great reviewer
Phoshi --Oblivion INI contribution
Psymon --gave a lot of advice on the presentation of material, especially where BAIN is involved
retroredeye --super helpful reading through and newbie-proofing the BAIN install guide
Shinra88 --PyFFI-ing information
SoSway --PyFFI leading spaces meshes list, OSR info, other technical help
tcr2 --crash picture
ThumperZ --Additional Streamline and Oblivion INI changes information
Vegetabill --proof reading, caught a lot of little inconsistencies
Vorians --pointed out a few PyFFI-optimization features, offers lots of help around the mods forum
Modders who gave in put in the Plugin Cleaning discussion
Authors that take the time to optimize their meshes and clean their plugins
Users who contributed and volunteered their load orders

Thank you all for contributing to a smoother Oblivion experience!!
- Tomlong75210

» Wed Jan 26, 2011 3:03 am

*TODO*

==TESIV:POSItive==
make Wrye Bash plugin commands page
put up video settings INI tweaks
finish HowTo download page
OSR INI tweaks
Add Win7 and Vista users page
Program & Utilities individual pages
GIMP installation, normal map plugin, DDS plugin
How to navigate the site for BAIN (to replace "complete" guide)
~~Before asking for help checklist~~
Should not be in an XX Load order list
Optimization page checklist
~~OOO Installation Guide~~
fill out site glossary
update "Links"

» Wed Jan 26, 2011 2:54 am

What's the recommendation for PyFFIng clothing?

» Tue Jan 25, 2011 4:01 pm

What's the recommendation for PyFFIng clothing?

Caution for clothing is the same as armor. It is not that you can't do it, but it is more likely for something to go awry. As long as the meshes check out in-game, it is okay. Some of the released PyFFI optimization mods include armor and clothing. Check thoroughly afterward, though. Otherwise, just leave clothes alone.

I'll amend the head post, thanks!
- Tomlong75210

» Tue Jan 25, 2011 5:04 pm

What's the recommendation for PyFFIng clothing?

Well I don't know the current "official recommendation", but my "private recommendation" would be to definitely give it a try. I have been pyffi-ing with PyFFI 2.0.5 the Roberts Female v12 body meshes and the clothing meshes from OOO, and after some 30 hours of game playing I have not yet seen a defect or anomaly. I have made proper backups of all files though, just in case there should be a problem in the future. But so far, I can only report positive experience with pyffi-ing of clothes and armor.

» Wed Jan 26, 2011 3:21 am

Well I don't know the current "official recommendation", but my "private recommendation" would be to definitely give it a try. I have been pyffi-ing with PyFFI 2.0.5 the Roberts Female v12 body meshes and the clothing meshes from OOO, and after some 30 hours of game playing I have not yet seen a defect or anomaly. I have made proper backups of all files though, just in case there should be a problem in the future. But so far, I can only report positive experience with pyffi-ing of clothes and armor.

Armour with the avoidance of helmets?

» Tue Jan 25, 2011 7:16 pm

Armour with the avoidance of helmets?

Definitely avoid meshes with EGMs, which usually means helmets, not all but most, I think.

» Tue Jan 25, 2011 11:33 pm

http://sites.google.com/site/oblivionpoinfo/system/app/pages/subPages?path=/libraries/cps

Looking for more suggestions!
- Tomlong75210

» Wed Jan 26, 2011 5:13 am

http://sites.google.com/site/oblivionpoinfo/system/app/pages/subPages?path=/libraries/cps

Looking for more suggestions!
- Tomlong75210

You should add that corrupt shaders could make random purple textures ingame.

» Tue Jan 25, 2011 5:10 pm

You should add that corrupt shaders could make random purple textures ingame.

I thought that that description might be lacking, at first I could only think of "messed up landscape textures."

Nice to hear from you again!
- Tomlong75210

Edit: Do you have a picture of this particular problem? I want to try to distinguish it from the craziness that occurs when you break the ~400 cap.

» Wed Jan 26, 2011 2:07 am

Definitely avoid meshes with EGMs, which usually means helmets, not all but most, I think.

To DerekBaker and Tomlong: I think that the Alpha5 PyffI tool available on Tesnexus is probably the smartest tool for pyffi-ing your meshes. It is automatically skipping any meshes that have .egm relations, so it says in its readme and/or features list. I have used Alpha5 for all my pyffi-ing, and it has come out OK. But I will admit that I am very much a layman in this. In fact, I have no idea what the EGM association really is... but Alpha5 takes care of it, so it looks.

It is my understanding that right-click pyffi-ing of single files or folders WILL NOT check for EGM associations, hence it is more risky than Alpha5 I would think. At least to a layman as I am.

» Tue Jan 25, 2011 8:25 pm

To DerekBaker and Tomlong: I think that the Alpha5 PyffI tool available on Tesnexus is probably the smartest tool for pyffi-ing your meshes. It is automatically skipping any meshes that have .egm relations, so it says in its readme and/or features list. I have used Alpha5 for all my pyffi-ing, and it has come out OK. But I will admit that I am very much a layman in this. In fact, I have no idea what the EGM association really is... but Alpha5 takes care of it, so it looks.

It is my understanding that right-click pyffi-ing of single files or folders WILL NOT check for EGM associations, hence it is more risky than Alpha5 I would think. At least to a layman as I am.

I have that linked, but the link is wrong in the PyFFI introduction paragraph (here and on the website.) I am fixing those now. However, I just searched for "egm" files and "fx" files before hand. (That is "fx" and not .FX files.) I deleted the files that had EGMs and the fx files. It didn't take any time at all, really. Also, files with EGMs are usually helmets and hoods.

Edit: I didn't know your upload was PyFFI-ed version. If it is not, could you upload that version too?

» Tue Jan 25, 2011 10:27 pm

I didn't know your upload was PyFFI-ed version. If it is not, could you upload that version too?

Do you mean my upload of the "Roberts Female Body v12 - Fixed" archive? If you do mean this one, no, I only uploaded the un-pyffied version of that, because I was/am not sure if the pyffied version will always be faultless. This is a potential hassle I do not want to contend with. Period, please understand. I thought everybody who wants to, could pyffi it for themselves.

Now, I do use the pyffied version privately and so far, it has been playing flawlessly and glitch-free. But I am hesitant to "officially" upload it on a popular venue like TESN because should there be a glitch eventually, I will only be flamed because I supposedly have screwed up Roberts fine original... I am thinking of an alternative, please write me a short PM and we can work out something no problem. Unfortunately it's too big a file (some 15 MB) to just send by email. I'm thinking of something like Megaupload or so. Just drop me a very short PM. I have no problem at all sharing it with folks, I just want to make sure they know what they are doing.

» Wed Jan 26, 2011 5:22 am

Do you mean my upload of the "Roberts Female Body v12 - Fixed" archive? If you do mean this one, no, I only uploaded the un-pyffied version of that, because I was/am not sure if the pyffied version will always be faultless. This is a potential hassle I do not want to contend with. Period, please understand. I thought everybody who wants to, could pyffi it for themselves.

Now, I do use the pyffied version privately and so far, it has been playing flawlessly and glitch-free. But I am hesitant to "officially" upload it on a popular venue like TESN because should there be a glitch eventually, I will only be flamed because I supposedly have screwed up Roberts fine original... I am thinking of an alternative, please write me a short PM and we can work out something no problem. Unfortunately it's too big a file (some 15 MB) to just send by email. I'm thinking of something like Megaupload or so. Just drop me a very short PM. I have no problem at all sharing it with folks, I just want to make sure they know what they are doing.

I figured it would be noticeable if the PyFFI-optimization of a body went wrong, haha. I can just PyFFI it myself. 15BM is nothing compared to the Vanilla mesh BSA. However, I will probably PyFFI some of the bigger mods, before I bother with cosmetic mods (i.e., COBL.)

» Wed Jan 26, 2011 6:32 am

Snip...

http://sites.google.com/site/oblivionpoinfo/randompages
More References! More things to know!!

Be Aware...
2. There is also a range of about 300-400 for the total number of plugins that can be in your data folder.
>>> Going over the total plugin cap can result in randomly missing textures (marked by purple meshes)
>>> Going over over also results in random black meshes/textures.

Snip...

You should re-write the above. It's not just plugins. BSA's and ESM's are also part of the total and rather than say "300-400" I would say "about 400".
Besides the funky textures, you can get just plain random weirdness or unexplained crashes or freezes.

Refer to this http://www.gamesas.com/bgsforums/index.php?showtopic=941296.

» Tue Jan 25, 2011 11:07 pm

You should re-write the above. It's not just plugins. BSA's and ESM's are also part of the total and rather than say "300-400" I would say "about 400".
Besides the funky textures, you can get just plain random weirdness or unexplained crashes or freezes.

Refer to this http://www.gamesas.com/bgsforums/index.php?showtopic=941296.

Oh yeah, I know from experience. I thought I fixed that. I guess I only fixed it here: http://sites.google.com/site/oblivionpoinfo/randompages/facts. It is not 400. It happened at much lower level for me.

» Wed Jan 26, 2011 5:32 am

I figured it would be noticeable if the PyFFI-optimization of a body went wrong, haha.

Hey Tom, the bodies of all female NPC (and my own female PC) were/are always fine.

Yeah definitely give it a try. I used Pyffi 2.0.5 and Alpha5 tool, it works very quickly as well. You can set skipping criteria in the .ini file but I only set the keyword "Hair". Yeah, it's not the "recommended, safe" way in terms of skipping, but I tell you, the result *SO FAR* has been plain normal visuals, which is exactly what you want.

If for any reason you still want my file(s) sometimes, really, don't hesitate to drop me a PM! I also made pyffies of OOO 1.33 and 1.34 beta 5, Roberts Female for Mods (only for OOO though) - so far all flawless but then again, I have not yet seen all content in-game. So officially I would not yet declare that it's all totally safe.

» Wed Jan 26, 2011 6:37 am

I did not use the automated tool this last time, but I will keep that in mind when I take on Cobl and the quest mods...That will be a lifetime. FCOM is not PyFFI-optimized either. That would be nice to do as well. I will do a little (only one or two of these rather big mods) at a time.

» Wed Jan 26, 2011 8:11 am

A little question: can I easily pack my mod in the BSA archives? What problems could I have?

From the help page of OBMM: "You also get to choose your compression levels. Higher compression levels will shrink the BSA, but will slow down BSA creation and may increase in game loading times."

From the Construction Set Wiki: "It has been said, that .bsa files would slow down load times, indefinitely, and would therefore cause the game itslef to be a slower experience. This is most definitely NOT true. A compressed .bsa file is about as fast as its uncompressed, folder tree brother. So, there really isn't any reason not to use this method of packaging your mod."

Another question: can I pack a mod without an .esp file? I think not, would not work in game, right?

P.S.
Excuse me, I followed the link "TESIV POSitive" just now.

» Wed Jan 26, 2011 7:34 am

A little question: can I easily pack my mod in the BSA archives? What problems could I have?

From the help page of OBMM: "You also get to choose your compression levels. Higher compression levels will shrink the BSA, but will slow down BSA creation and may increase in game loading times."

From the Construction Set Wiki: "It has been said, that .bsa files would slow down load times, indefinitely, and would therefore cause the game itslef to be a slower experience. This is most definitely NOT true. A compressed .bsa file is about as fast as its uncompressed, folder tree brother. So, there really isn't any reason not to use this method of packaging your mod."

Another question: can I pack a mod without an .esp file? I think not, would not work in game, right?

P.S.
Excuse me, I followed the link "TESIV POSitive" just now.

Some people have recommended choosing no compression. That way you get the benefited of the files being collected in one spot, and resources do not have to be spent decompressing the archive. "Uncompressed" refers to the data files loose in the folder, whereas no compression means in the BSA archive but not compressed. You can definitely pack a mod with out an ESP file. Many texture replacers do not have ESP files. Mods without ESP files are always replacers or resources of something though. BSA archives only contain data files, not ESP files. They are not like OMODs.

» Tue Jan 25, 2011 5:33 pm

Ok, I have a question, if anyone is willing to answer it...

I just finished PyFFIing my vanilla meshes (including SI), and the PyFFI Automation Tool said that there were more than 50 "Files Errors." Does this mean that some of the files are corrupt? If so, how can I make sure that they are deleted and not included in the OMOD that I'm going to make? Thanks in advance!

» Tue Jan 25, 2011 9:10 pm

Ok, I have a question, if anyone is willing to answer it...

I just finished PyFFIing my vanilla meshes (including SI), and the PyFFI Automation Tool said that there were more than 50 "Files Errors." Does this mean that some of the files are corrupt? If so, how can I make sure that they are deleted and not included in the OMOD that I'm going to make? Thanks in advance!

PyFFI has been developed to a point where using that Automation Tool is unnecessary. Grab the latest version of PyFFI, and then read the http://www.gamesas.com/index.php?/topic/1061938-info-relzpyffi-python-file-format-interface/. I do not know the significance of those errors. Does the tool you used copy the files it PyFFI-Optimizes to a different location? If so, then it should not have copied the files that it gave the error about. In that case, you can try going over those files with the new version of PyFFI. However, I recommend starting over and PyFFI-optimizing those files with the latest version of PyFFI. Here is the direct link to the latest version of PyFFI: http://www.sendspace.com/file/do4ekz. You should read the pages around that post to learn about the PyFFI INI. I can give you further directions if you need them.

Edit: wording

» Wed Jan 26, 2011 7:23 am

PyFFI has been developed to a point where using that Automation Tool is unnecessary. Grab the latest version of PyFFI, and then read the http://www.gamesas.com/index.php?/topic/1061938-info-relzpyffi-python-file-format-interface/. I do not know the significance of those errors. Does the tool you used copy the files it PyFFI-Optimizes to a different location? If so, then it should not have copied the files that it gave the error about. In that case, you can try going over those files with the new version of PyFFI. However, I recommend starting over and PyFFI-optimizing those files with the latest version of PyFFI. Here is the direct link to the latest version of PyFFI: http://www.sendspace.com/file/do4ekz. You should read the pages around that post to learn about the PyFFI INI. I can give you further directions if you need them.

Edit: wording

Well, I'm pretty sure that I'm using the most recent version of PyFFI (2.1.4). Also, are you sure that the Automation Tool is no longer necessary? I have found it to be very helpful (For example, it allows you to pause the PyFFI script).

» Wed Jan 26, 2011 4:56 am

Well, I'm pretty sure that I'm using the most recent version of PyFFI (2.1.4). Also, are you sure that the Automation Tool is no longer necessary? I have found it to be very helpful (For example, it allows you to pause the PyFFI script).

I need to update the first post again, thanks. Also, PyFFI is now supposed to support pause and resume. It also has an INI file with a skip list. The other problem with the automated tools is the size check, but you should be able to disable it somehow I guess. It could be giving errors because of an increase in file size. Reading that PyFFI thread, I learned that PyFFI sometimes does cause an increase, so the size check does not do anything good. Since you are using 2.1.4, you could just target the 50 files that the errors were given about. Certain types of files should be skipped, as you know, but aside from those exception, you could go ahead and try PyFFI-optimizing them again (with the right-click method even.)

» Tue Jan 25, 2011 7:32 pm

Ok, so I looked in my PyFFI install directory, and I found the Toaster utility. Is this what I should use? There are two .ini's in the folder, plus "in" and "out" folders.