Output is minimal in the moment, I hacked it together in 48 hours. I am running it in this moment on the mesh.bsa, no missing files yet. Most files go down by between 1/5th and 1/6th of the original size (80%-90%).
You'll get the general exception-output from niflib, for everything else we have to start forming it from it's raw state.


Which is that? Ah, uhm, I think it means removing attributes and stuff? No, it has no knowledge of anything, it just eats up anything which resembles a NIF and spits another NIF out.
If someones explains me what I'd have to do I can put purging and reduction into the tool.

I was referring to this portion of the RAEVWD readme:

These extra steps would only be done on a _far.nif file.

Although, yes, reduction code would be nice too, but that's not what I'm after

Has that been done manually?

Yes, when I did that initially, PyFFI did not have the ability to do it. So it was all manually removed from each file. A process which took weeks. My fingers still cringe in fear of it ever happening again.

Oh man.
So I can look into PyFFi what it is doing to replicate that, assuming it now does it automatically?

Yes, specifically the clean_farnif spell.

And that work is certainly appreciated.

Wow, that's crazy . In my use of RAEVWD, I never realized the real volume of work you had to put into it. Thanks !

On an unrelated matter, after PyFFIng all of my mod-provided meshes, I was experiencing many CTDs. I eventually tracked down at least one sure-fire CTD to the PyFFIed meshes for Let the People Drink. Is this a known issue, and I just shouldn't try to PyFFI them, or is there a specific exception I should add to my Optimiztion Kit ini files? I assume this has something to do with LtPD's animated waterfalls/pools/fountains/etc.

There is a serious bug in one of LtPD's fountain meshes that can cause this. I'm not sure myself which one it is, though.

So long as it's not just me - I had been worried that memory-eating aside, perhaps something _else_ was wrong with my PyFFI setup . I'm fine with not optimizing LtPD's meshes.

Just to clarify, is this a bug such that I'd be better off uninstalling LtPD? Or is it just a bad interaction between LtPD and PyFFI?


That sounds very appealing... I could sure use a boost in my machine's capability to handle Oblivion's geometry. Unfortunately my setup is still pretty unstable for the moment. I think that until I get things working fairly solidly, I'd make a poor tester, since I could easily have you chasing down bugs that have nothing to do with the new tool .

Okay, implemented. There seems to be a few mistakes in it though I believe:
a ) If you remove the VertexColorProperty, you have to remove VertexColorData as well I think
b ) Removing SpecularProperty will prevent any shiny object to "glitter" far away, maybe it's not there yet, but you know what I do with OBGE
c ) Removing SpecularProperty & APPLY_HILIGHT2 (parallax mapping) will become odd looking when your LOD-kick-in-distance is actually low, I already got strange far geometry with my POM (far is visibly brighter overall)
d ) Removing TB from the TNB will kill any correct normal-mapping, then you could throw the normals and the normal-texture away as well, you do have effects from normal-maps even over half the continent (textures normally get visibly darker), and there isn't really any performance hit from it

Point d is pretty severe. Maybe you didn't see it because the textures aren't really distinguishable at far distances, but in effect the normal-map became just random-noise because the TNB-matrix which is required for normal-mapping in tangent-space just got non-sense. That the _fars are too bright is a thorn in my eyes since I use them.

I have to look which shaders are used when the normal-map is absent though, and how they behave.


I have cases on some original files where the maximum overdraw (some worst-case: polygon over polygon over polygon caused by the winding of the shape) got down by a factor of 3000. We're discussing how we can proof-case this optimizer, because the FRAPS-FPS in NifSkope is frozen (because there is no consecusive refresh), so I can't tell you real-world number right away.

I though about two manequin-rooms with lots of complex meshes, one with normal one with optimizes mesh-versions.

+-----+|     |#1    #2|     |+-----+

So you can walk from one to the other and check FPS instantly without replacing meshes and restarting and changing all what-not circumstances/conditions which make it very difficult to have comparable numbers.

Here's the tool for testing:
http://paradice-insight.us/stuff/oblivion/NIFopt.7z

Application crashes (x32 and x64) with a BEX error

error message (x64 and spell -stripstolists):
used file: SBlmCorsetFull.nif form the mod HGEC_Sinblood_LethalMajestyOutfit_fixed-39756-1-0

Give it an output name. I updated the download with a few fixes.
BTW there are no strips in the file to convert.

Replaces this nif- file optimizer the PyFFI- Optimizer (part triangles re-ordering)?

Usage with PyFFI optimize spell:
first PyFFI, second nifopt or ?

Yes.


Would have no effect. If you have a sane NIF you won't need PyFFI, removing duplicates and empty nodes is the only use for PyFFI in this case.

So far as I'm aware not addressing that block hasn't ever caused any problems visually.

Yes, I realize under OBGE you're probably going to do more than the vanilla game is capable. In the vanilla game, specular distance doesn't operate that far out, and all you end up with is yellow LOD as far as the eye can see if you don't take it off.

Similarly here, without OBGE involved in some way, Parallax data does not operate that far out. Or if it does, it's too far away to notice.

I've actually tried this with the complete removal of normal maps. LOD may be working differently, because there is a noticeable difference in quality if they're tossed out. The game won't crash either way, but it *IS* making use of the normal maps even without the tangent space data. Obviously you can't do this at near-distance because the game will often just crash.

I've also never noticed the _fars being any brighter or dimmer than their near distance counterparts - unless something is triggering an anomaly.

No I just mean there is (possibly) some then redundant data left in the file. I think the VertexColorProperty configures how to treat the VertexColor-array that may reside in the file. It was more adressed to the niftools-programmers, if the spell removed the property, the related data should go out of the window as well.


Lately there are real strange things I read, like parallax+glow supposely doesn't work. I'd like to see test-cases, because I'm pretty sure they should work and are possibly buggy, unintendedly. This is another of those. Of this at least I guess I can make a test-case quick myself. So you say if I put a spec-map on a IC-far and I look from Cloudruler it's going yellow?
BTW anyone every tried 16bit floating point spec-maps with spec-values above 1.0? I'm pretty sure you could make glitterring on far thing this way right now. Of course you won't get the star, but's a thing we may have one day in OBGE.


No, I said when the config says far-LOD kicks in at say 3000 units the model can still be so big you'll see every wrinkle and it'll look weird. On the other hand all of this is highly dependent of the specific far. I understand now, after looking over them, none of them has anything beyond base-textures, is that right?


Normal-mapping won't work as intended without the full TNB. It's not necessary to state the geometry empirically renders still "fine". It's okay for me, just sayin' there is probably more to come (discussion, fixes, ...). When we auto-reduce hi to lores and bake the geometry-differences into a normalmap we go back to TNB anyway.
What is possible though is reconstruct TB in the shader (I did that with the terrain-LODs, I added normals, and based on UV+N I reconstructed TB inside the shader). I have to fork the shaders inside the engine anyway for LOD (can you believe the shader for eyes and armor is the same as for the tarrein, arggg), so we can do special shaders for LOD-elements.
What does work is using only normals in the NIF without normap-mapping, of course.


Maybe it's a problem of the lowres not matching the hires textures. It's very apparent on fortruins, they stick out of the landscape like no good. Are the lores-textures in any way special? Do the UVs of lo and hi res meshes perfectly match? Because if not+yes maybe we should make a hires-to-lores converter to fix that up?

BTW, should we move to the RAVWD-thread for this. We can continue discussion the tool here though. Does it work? I have like 6 broken GeomMorpher here, all other files are apparently sane (have not checked if the bones survived ...). Converting O-meshes.bsa takes around 10-15 minutes.

Yeah, I'll respond over there...

Bump. Tool works? I found a few bailouts in Vvardenfell Imports, but 20GB of other meshes convert. I have to look at damaged bones and morphers though, visually. As in NifSkope you don't see that stuff.

Creatures and Clothes are a big problem (distorted upper clothes, distorted rats). Armor I haven't tried. Weapons, landscape, architecture (far- nif), rocks, dungeons (folder ...) seems to be no big problem (some cave roots are distorted).

Wow, I already filed two bugs on the Nif-bugtracker. Seems I have to fix the core library-code a little bit while I progress.
Vvardenfell Imports is a good test-set. I wonder though (just from a logic PoV), why do Adamantium helmets have vertex-deformers, I mean they aren't out of flexible Adamantium, right?

Have you looked at the exception list that PyFFI is not supposed to touch? I think some of the mentioned distortions are because things that shouldn't be getting touched are still being modified, like helms that have *.egm files etc. I think havok roots were part of that exclusion set.

I saw the the filter, but of course I have no idea about the roots. Though I'm going to try to fix up everything I find without the need for a filter. I have "The Conformulator"'s source code (hey scanti, walking in your footsteps ) so I expect you'll see .egm support.

Okay, progress:
http://paradice-insight.us/stuff/oblivion/NIFopt.7z

- EGM support
- fixed removal of faces in all cases (morphs, partitions, egms, etc.)
- fixed the entire SkinPartition thing, that didn't work at all
- more comprehensive output (tells you when repairs have been applied, or when there is some suspicious stuff in the file)
- converts SkinPartition-strips to lists as well now
- optimization for the SkinPartitions as well (though the vertices are in base-mesh order, no vertex-map reordering)

Now I want to know how can I see damaged morphers or [...] without actually applying it in Oblivion? NifSkope does not show those.

Edit: Side-question: Is there somewhere BSA source-code besides the OBMM source-code, so I may optimize directly from BSA-to-BSA?
Edit: Problematic meshes:


Better Cities - Meshes\meshes\architecture\bettercitiesresources\anvilbayexp\boatmod.nif:
- doesn't load vertices, NifSkope shows it corrupt, from 5.0.0
Better Cities - Meshes\meshes\bettercitiesresources\clothes\pregnantdress.nif
- contains a bone with 0 weights, from 5.0.0
Vvardenfell Imports\meshes\vvardenfellarmormod\high indoril\*.nif
Apachii Goddess Store\meshes\*somewhere*.nif
- several NIFs which have identically connected but differently percolated faces (I fixed that in the code)
Vvardenfell Imports\meshes\vvardenfellarmormod\hlaalu\hlaalucuirass.nif
- the file bugs out niflib, Upperbody:0 is loaded twice, even NifSkope goes all wonky
Apachii Goddess Store\meshes\*somewhere*.nif
EyeCandy Compilation\meshes\*somewhere*.nif
- some NULL extra-datas (I fixed that in the code)
TamRes - Complement\meshes\tamres\tamres\armor\legionmagerobe\m\robebmage.nif
- contains an illegal combination of TriShape+TriStripData
Oscuro's Oblivion Overhaul/meshes/armor/growlf/Fantasy/boots.nif
- contains a SkinPartition without VertexMap
Oscuro's Oblivion Overhaul/meshes/Weapons/throttle/tkIcxthSword.nif
- contains an unattached BinaryExtraData, this produces a fatal load error
Apachii Goddess Store\meshes\clothes\apachii\wigscorean\*
- most contain broken tangent-space vectors
CyrodiilUpgradeResourcePack/Meshes/architecture/CyrodiilUpgradeOverhaul/LRFarmHouses/houselower01.nif
- kaputt (missing 5 blocks)
CyrodiilUpgradeResourcePack/Meshes/architecture/CyrodiilUpgradeOverhaul/IceCavern/exterior/ELicicle01.nif
- kaputt (missing 2 blocks)
CyrodiilUpgradeResourcePack/Meshes/architecture/CyrodiilUpgradeOverhaul/IceCavern/interior/ELrmwalllong01.nif
CyrodiilUpgradeResourcePack/Meshes/architecture/CyrodiilUpgradeOverhaul/IceCavern/interior/ELwhalllong01.nif
- kaputt (missing 1 block)


I converted without hickups (besides the above):

Shivering Isles and it's RAEVWD

Final report: processed files: 1439 processed faces: 2266755 modified meshes: 3229 fixed meshes: 0 damaged meshes: 0 broken hierarchy: 72Final report: processed files: 132 processed faces: 237419 modified meshes: 2 fixed meshes: 0 damaged meshes: 0 broken hierarchy: 0

Knights
Oscuro's Oblivion Overhaul

Final report: processed files: 1809 processed faces: 3400849 modified meshes: 1557 fixed meshes: 9 damaged meshes: 0 broken hierarchy: 0

Better Daedric Shrines (all of them)

Final report: processed files: 1212 processed faces: 1937166 modified meshes: 987 fixed meshes: 0 damaged meshes: 0 broken hierarchy: 54

Sheogorad

Final report: processed files: 1166 processed faces: 1082148 modified meshes: 140 fixed meshes: 0 damaged meshes: 0 broken hierarchy: 4

Better Cities

Final report: processed files: 3577 processed faces: 6918121 modified meshes: 80 fixed meshes: 0 damaged meshes: 0 broken hierarchy: 2

All Natural

Final report: processed files: 394 processed faces: 2835151 modified meshes: 12 fixed meshes: 0 damaged meshes: 0 broken hierarchy: 0

Oblivion and it's RAEVWD

Final report: processed files: 8032 processed faces: 13674383 modified meshes: 17502 fixed meshes: 0 damaged meshes: 0 broken hierarchy: 162Final report: processed files: 630 processed faces: 841505 modified meshes: 19 fixed meshes: 0 damaged meshes: 0 broken hierarchy: 0

Artifacts

Final report: processed files: 1328 processed faces: 2390081 modified meshes: 546 fixed meshes: 0 damaged meshes: 0 broken hierarchy: 0

SentinelsLegendaryWeaponsAndArmors

Final report: processed files: 160 processed faces: 374195 modified meshes: 15 fixed meshes: 0 damaged meshes: 0 broken hierarchy: 0

Vvardenfell Imports

Final report: processed files: 782 processed faces: 4850839 modified meshes: 357 fixed meshes: 2 damaged meshes: 0 broken hierarchy: 3

Apachii Goddess Store

Final report: processed files: 1456 processed faces: 13625917 modified meshes: 923 fixed meshes: 52 damaged meshes: 0 broken hierarchy: 0

EyeCandy Compilation

Final report: processed files: 2069 processed faces: 12429420 modified meshes: 1446 fixed meshes: 14 damaged meshes: 0 broken hierarchy: 0

TamRes - Complement

Final report: processed files: 87 processed faces: 277182 modified meshes: 151 fixed meshes: 4 damaged meshes: 0 broken hierarchy: 1

TamRes - Complement II

Final report: processed files: 120 processed faces: 379964 modified meshes: 14 fixed meshes: 2 damaged meshes: 0 broken hierarchy: 0

TamRes - Meshes

Final report: processed files: 788 processed faces: 2134355 modified meshes: 150 fixed meshes: 3 damaged meshes: 0 broken hierarchy: 0

TamRes - Landscape

Final report: processed files: 1210 processed faces: 1295281 modified meshes: 1151 fixed meshes: 76 damaged meshes: 0 broken hierarchy: 0

TWMP Summerset - Meshes

Final report: processed files: 633 processed faces: 1490167 modified meshes: 11 fixed meshes: 0 damaged meshes: 0 broken hierarchy: 4

TWMP Valenwood Elsweyr

Final report: processed files: 151 processed faces: 199161 modified meshes: 117 fixed meshes: 1 damaged meshes: 0 broken hierarchy: 0

Elsweyr Anequina

Final report: processed files: 1439 processed faces: 2555803 modified meshes: 439 fixed meshes: 35 damaged meshes: 0 broken hierarchy: 4

I have optimized with nifopt_x64
DLCFrostcrag
DLCHorseArmor
DLCOrrery
DLCShiveringIsles - Meshes
DLCThievesDen
DLCVileLair
Knights
Oblivion - Meshes

I use four directories
1. original files (V:\test-1)
2. stripstolists files (V:\test-1a)
3. optimizelists files (V:\test-1b)
4. optimizeparts files (V:\test-1c)


and got one crash
error message:

console window:

part of the logfile:


Log files
- http://www.4shared.com/file/d-UYE_JS/test-1.html
- http://www.4shared.com/file/Ur3hyrjl/test-2.html



The PyFFI optimization process (spell optimize) calls some "sub"- spells
- fix_delunusedroots (execution parallel)
- opt_cleanreflists (execution parallel)
- fix_detachhavoktristripsdata (execution parallel)
- fix_texturepath (execution parallel)
- fix_clampmaterialalpha (execution parallel)
- fix_bhksubshapes (execution parallel)
- fix_emptyskeletonroots (execution parallel)
- opt_mergeduplicates (execution series)
- opt_geometry (execution series) <--- mesh optimization process (I think)
- opt_collisionbox (execution series)
- opt_collisiongeometry (execution series)

Which of these spells are a part of nifopt or could be a part?
[x] opt_geometry