[RELz] Really AEVWD - Thread #6

Post » Wed Mar 09, 2011 12:06 pm

REALLY Almost Everything Visible When Distant

http://www.gamesas.com/index.php?showtopic=1046403

A modest approach to distant viewable objects, aka LOD.

http://tesalliance.org/forums/index.php?/files/file/301-really-aevwd/
http://www.moddb.com/mods/really-aevwd
http://www.tesnexus.com/downloads/file.php?id=20053

You're all familiar with the various LOD packages, right? Some provide more than others. Some ar geared specifically toward super-high end machines. One isn't even designed to be played and will cripple anything currently on the market today. For the rest of us, myself included, we'd just like to enhance the visual experience without crippling the game. None of the available packages has been fully optimized, usually with large amounts of unnecessary data still attached.

In order to address these issues, I've decided to compile into one source as many useful LOD objects as can be reasonably applied to the game from the following sources:

AEVWD Large: Enhanced lowres textures, Imperial City meshes, Skingrad castle meshes, and the 3 daedric shrines it comes with.
AEVWD Fixes: Frostcrag Spire (requires the official DLC to be activated).
UOMP: Battlehorn Castle (requires the official DLC to be activated).
Oblivion: City wall meshes only.
Brumbek: LOD illuminated windows, optimized lowres texture pack, optional QTP3 lowres texture pack, as well as several mesh and texture fixes.
Me: Machine regenerated all architecture, Ayleid ruin, and Imperial fort meshes for better quality and lower polygon counts. UOP corrected versions were used for the reductions wherever possible.

The overall goal is to provide enhanced LOD visuals while also trying to preserve performance as much as possible. This all stems from my realization that even with a high end graphics card, full detail LOD meshes and textures are simply out of the question. The VRAM usage is causing large amounts of stuttering when trying to swap out the memory and not even Streamline 3.1 can keep up. This package has brought the VRAM usage under control and has had a huge impact on overall lag and stuttering.

All of the meshes have been hand checked and had the following done:

* Collision data removed. If actual collision grids were still present, the branches have been deleted entirely. It's impossible to collide with an LOD object.
* Animation controllers removed. It is impossible to animate an LOD object, so this data is bloat.
* Geometry morphers removed. If a mesh had these, they've been removed, as they are only useful in animation.
* All NiStringsixtraData blocks removed. These are nearly always associated with collision.
* All BSX Flags removed. These are nearly always used for collision and animation.
* All NiSpecularProperty blocks removed. Leaving these in place causes visual anomalies, most notably the "every LOD turned yellow" issue.
* All NiBinaryExtraData blocks (tangent/normal spaces) removed. This data is not required for LOD meshes and helps tremendously in reducing memory usage.
* All parallax flags turned off. The distances involved make this useless anyway.
* All meshes have been assigned lowres textures instead of hires ones.
* Nearly all NiAlphaProperty blocks removed. If the mesh required it for transparency, it was kept. If this is left on a mesh that does not require it, visual anomalies can result.
* All NiVertexColorProperty blocks removed. These lead to some cases of black LOD objects.
* Optimization using PyFFI after the above has been done.

The typical LOD mesh comes out the other end of this process losing up to 75% of the data it was carrying with no loss of function. A fully optimized and functional LOD mesh will have only the NiMaterialProperty, NiTexturingProperty, and NiTriShapeData/NiTriStripsData branches left in its NiTriShape/NiTriStrips branches, and will have only NiTriStrips or NiTriShape branches attached to the scene root.

Compared with all of the other LOD options available, this is the most optimized, game friendly set of LOD objects there is.

Performance

Since I can only draw one meaningful comparison here, I shall do so. I didn't think to take a before shot of what it looked like before this project began. I didn't think I was going to offer it up as a VWD mod until halfway through this.

Tamriel VWD Enhanced, mixed in with parts of AEVWD Large, majority of it in full detail meshes+textures: 892MB VRAM @ 15FPS
Really AEVWD, when installed with all of its options: 475MB VRAM @ 18FPS

Shot of the scene being tested, after everything was done: http://www.tesnexus.com/downloads/images/20053-1-1222633270.jpg

As you can see, your FPS may not shift much, but the VRAM usage is astounding. And at least from where the shots were taken there's only a minor difference in detail. It will become more noticeable up close, but that's one of the limitations.

The screenshots were taken with the following system:

AMD Phenom 9850 2.5Ghz (quad core)
AMD/ATI Radeon HD4870 w/ 1GB VRAM, Catalyst 8.12 drivers. Set to force 8xAA+16xAF.
QTP3 Redimized
QTP3 Reduced Lite - with only landscape and architecture used.
Brumbek's optional QTP3 lowres textures for LOD objects.
Zacharot's 2048x2048 distant land textures.
All Natural (weather mod with hires graphics)
Open Cities Full + Blood&Mud + Leyawiin Reborn, version 3.1.1
Optimized Distant Land MAX 1.1 - further optimized using PyFFI
Other graphics enhancers

Screenshot location is on top of White Gold Tower in the Tamriel worldspace, facing west toward Chorrol. This is an ideal location for LOD testing because there is next to nothing in the near view other than the tower mesh itself. Clear weather was chosen through the console to eliminate skew from All Natural's excellent weather graphics. So what you see is 98% LOD and 2% whatever else. The VRAM savings alone are quite impressive.

There are also extra LOD objects present due to the addition of several other mods with buildings and such, Verona Bay is visible in the shot in the middle right for instance.

I think on more normalized systems that aren't as completely bogged down by massive amounts of enhancers will do quite a bit better with this LOD package.

Possible solutions to FPS issues:

1. uGridDistantCount in the Oblivion.ini file is set to 25 by default. Lowering this to 20 or even 15 will provide a dramatic improvement.
2. If lowering uGridDistantCount is not enough, selectively removing parts of the architecture meshes RAEVWD provides will help the most.

Installation

Simply unpack the archive to your Data directory. Everything should fall into place. If prompted to overwrite files, answer yes.

Or unpack to a separate location and only put in what you want, you don't have to use it all.

You can also make an OMOD of this using OBMM if you wish, to make removal easier later if you want.

No DistandLOD folder is provided. In order to actually see the results of what you just did, you need to download TES4LODGen and run it. It's a wicked simple utility that processes LOD information based on _far.nif files provided and will tailor itself to whatever mods you have installed.

Uninstallation

Manually remove the files you installed, or deactivate the OMOD if you made one.

Options

An optional ESP is included which places copies of buildings and trees behind the city walls and deletes the stock vanilla LOD houses. Kvatch and the Imperial City are not covered by this ESP. If you want the IC, use the second optional ESP file.

The ESP is 100% module compatible with Open Cities, and Open Better Cities, in any combination thereof.

Cathedrals have not been placed where missing since the Unofficial Oblivion Patch and some other mods address this. If you're not using the UOP, uh.... why not?

The cloister in Anvil as well as Benirus Manor have also been left out since the UOP deals with these as well. The UOP places the full detail mesh, so don't worry :)

The optional ESP is *NOT* compatible with most of Better Cities. Bravil and Chorrol will show up in LOD. Anvil, Bruma, Cheydinhal, Leyawiin, and Skingrad will be empty behind the walls. That includes Bethesda's crappy LOD houses. The compatibility method used has no way to fully fit in with BC. So if you're using BC, it may be better to avoid using the RAEVWD Cities.esp file at all. If you install this and complain about missing LOD behind the walls of the mentioned cities, I'll know why and also know you didn't read this :)

A second optional ESP is included for use with the Imperial City. Be advised that activating this additional ESP is going to impose a steep hit on your FPS. Mine dropped 6-8 frames after activation viewing the IC at a distance. Lower powered rigs may struggle with this. So this ESP option is not for everyone. There may also be a large additional FPS drop as you approach the city because the full size meshes needed to be placed behind the walls in order to make the LOD work at all.

The Imperial City ESP file should not be used in combination with Better Cities as they provide their own LOD placements and this will cause duplication with it.

Compatibility

The only thing this will conflict with is another LOD mod that affects the same files this one includes. Otherwise there's no issue.

Known Issues

If you are not using the optional ESP, there will be no LOD objects behind city walls other than what Bethesda supplied. If you want that solved, activate the ESP, but see the compatibility notes above. This may also add an increased burden on your system since city architecture and trees aren't normally available.

Sometimes when viewing the world at odd or extreme angles, the LOD objects will turn bright yellow or completely black and then revert to normal when you move. One known reason for this is another LOD mesh that has specular properties enabled on it. Another known cause are LOD meshes with vertex color properties. Since all of the meshes supplied by RAEVWD have been cleaned of these problems, look to LOD meshes provided by other mods and contact their authors for assistance in getting those fixed as they know their meshes best. Not all causes of this problem have been identified yet, so there may not be anything to fix.

Sometimes you will see red, blue, or green reflections in the water where LOD should reflect instead. Possible causes for this are LOD meshes with collision data still attached, BSX flags still attached, or parallax flags which are still set. Since all of the RAEVWD meshes have had these properties removed, another mod which supplies LOD objects is likely to blame. It is also possible that some textures with RAEVWD could be causing this, so if you can identify which reflections are causing this, we can attempt to track down a way to fix it.

Occasionally only part of a mesh will go black. If this happens, that part of the mesh is missing a texture assignment. If this happens with an RAEVWD mesh, please report it in the official thread. If it happens with an LOD mesh provided by another mod, you'll need to take that up with its author.

When used with bloom lighting, the LOD objects may always show up as black. Provided you are *SURE* the lowres directory is in the right place in your textures directory, the only fix for this is to switch to a different lighting method. I'm not sure if normal lighting works, but switching to HDR will resolve this issue. There is no known way to fix this otherwise.

Credits

Silverglade for his work on AEVWD Fixes.
Bomret for compiling the work on Illuminated AEVWD.
Igge for all his work on AEVWD Large, and the normal AEVWD.
Kivan for all his hard work on the UOP and UOMP mesh corrections.
Brumbek for providing several new lowres textures, numerous mesh/texture fixes, new illuminated windows, and his optional QTP3 matched set.
The makers of Milkshape 3D, without that trial period none of the rebuilt mesh work could have been done.

Licensing and Legal

Redistribution is strictly forbidden without prior consent. This means do not upload it anywhere unless you have obtained permission from myself (Arthmoor).

Non-English translation versions may be uploaded without permission on the following conditions:

1. All mod files contained in this archive are retained with their current names.
2. No alterations are made to the contents of the plugins or master file other than those necessary to translate the English text.
3. No alterations or additions are made to the meshes and textures other than those necessary to translate English text.
4. This readme is included, in ENGLISH, exactly as it is written in the version it was downloaded with.
User avatar
Monika
 
Posts: 3469
Joined: Wed Jan 10, 2007 7:50 pm

Post » Wed Mar 09, 2011 7:24 am

I fixed my issue with stones and trees not showing up in Anequina. It fixed itself at the same time I got All Natural interiors working as intended. I am not sure how; I will try to break it at a later date (will finally play the game instead of fixing it) to be able to give some info on the issue if someone else comes along with a similar problem.

Cheers!
User avatar
christelle047
 
Posts: 3407
Joined: Mon Apr 09, 2007 12:50 pm

Post » Tue Mar 08, 2011 8:49 pm

I finally got this to work for me, however, whenever I look at the Imperial City after just coming out of the sewers, the framerate will drop to about 10fps. I can look anywhere else and run about 60fps it seems. Again, if I just happen to get any bit of the Imperial City in view, it goes to crap.

I am running a Q6600 at 3.0 Ghz, 4GB DDR2, GTX 285 1GB of VRAM. I have QTP3 Redimzed with some other graphical mods. I only get a framerate drop when I have RAEVWD enabled.

Any idea of what I may be able to do to help?
User avatar
Lil'.KiiDD
 
Posts: 3566
Joined: Mon Nov 26, 2007 11:41 am

Post » Wed Mar 09, 2011 11:04 am

Arthmoor,

I started working through the fort ruins nifs to prepare them for being visible when distant in the Mesogea mod as I am using them for town walls that will be visible from afar. Given that you have already optimized them it will save me hours of work. Would you mind me using the "Fort Ruins" distant nifs to save me time re-doing the work you have already done?

For some reason PM isnt working very well.
User avatar
willow
 
Posts: 3414
Joined: Wed Jul 26, 2006 9:43 pm

Post » Wed Mar 09, 2011 5:53 am

Sure, that's no problem.
User avatar
Big mike
 
Posts: 3423
Joined: Fri Sep 21, 2007 6:38 pm

Post » Wed Mar 09, 2011 11:59 am

I'm having a slight issue with RAEVWD or rather with it in conjunction with the game settings I'd guess. When closing in on, say, Bruma or a Fort the RAEVWD objects vanish but I still don't have the chapel or Fort drawn by the game. In short, buildings seem to pop out of existence for a short while when closing in on them. I have View Distance at 100% and Object fade at max.
User avatar
Jade
 
Posts: 3520
Joined: Mon Jul 10, 2006 6:42 am

Post » Wed Mar 09, 2011 1:47 am

Good mod but still this is FPS killer for me (tested some hours ago); cannot use it.
User avatar
Prisca Lacour
 
Posts: 3375
Joined: Thu Mar 15, 2007 9:25 am

Post » Wed Mar 09, 2011 6:00 am

I'm having a slight issue with RAEVWD or rather with it in conjunction with the game settings I'd guess. When closing in on, say, Bruma or a Fort the RAEVWD objects vanish but I still don't have the chapel or Fort drawn by the game. In short, buildings seem to pop out of existence for a short while when closing in on them. I have View Distance at 100% and Object fade at max.


I get that occasionally too. I don't think there's anything that can be done about it. There's sometimes a delay between when the game pops the VWD objects out and then loads the near distance versions in. Happens for me most often if I've been running around a lot without stopping. I guess even with OSR installed you can cause the loader tasks to back up in the queue.
User avatar
Chloe Botham
 
Posts: 3537
Joined: Wed Aug 30, 2006 12:11 am

Post » Tue Mar 08, 2011 9:03 pm

I get that occasionally too. I don't think there's anything that can be done about it. There's sometimes a delay between when the game pops the VWD objects out and then loads the near distance versions in. Happens for me most often if I've been running around a lot without stopping. I guess even with OSR installed you can cause the loader tasks to back up in the queue.

Wow, I was just thinking about this and reminding myself I need to ask about it at some point. How fortuitous! It is good to know at least a potential reason for it... Thanks, Arthmoor.
User avatar
Sudah mati ini Keparat
 
Posts: 3605
Joined: Mon Jul 23, 2007 6:14 pm

Post » Wed Mar 09, 2011 5:38 am

I get that occasionally too. I don't think there's anything that can be done about it. There's sometimes a delay between when the game pops the VWD objects out and then loads the near distance versions in. Happens for me most often if I've been running around a lot without stopping. I guess even with OSR installed you can cause the loader tasks to back up in the queue.


You're right - it seems to only happen occasionally. I've been wondering for a while if it was me having wrong settings or something.
User avatar
CxvIII
 
Posts: 3329
Joined: Wed Sep 06, 2006 10:35 pm

Post » Wed Mar 09, 2011 4:09 am

Arthmoor, I have the yellow glow problem on the distant meshes for the fort walls. Whats your advice?
User avatar
Luna Lovegood
 
Posts: 3325
Joined: Thu Sep 14, 2006 6:45 pm

Post » Wed Mar 09, 2011 6:41 am

One or more of your other _far.nif files has a specular data node in its branch. The fort ruin meshes are clean. The readme file has the gory details of what data needs to be removed. PyFFI also has a mode where you can have it clean _far.nifs of unwanted data automatically.
User avatar
^~LIL B0NE5~^
 
Posts: 3449
Joined: Wed Oct 31, 2007 12:38 pm

Post » Wed Mar 09, 2011 2:17 am

What's the compatibility situation with the current version of Open Cities Reborn?
User avatar
Jason White
 
Posts: 3531
Joined: Fri Jul 27, 2007 12:54 pm

Post » Wed Mar 09, 2011 5:00 am

100%. This doesn't do anything that would conflict unless you're trying to use the cities plugin with it, which you probably shouldn't do.
User avatar
Matthew Warren
 
Posts: 3463
Joined: Fri Oct 19, 2007 11:37 pm

Post » Wed Mar 09, 2011 3:55 am

What's the compatibility situation with the current version of Open Cities Reborn?


Just re-run TES4LODGen and you are set.
User avatar
{Richies Mommy}
 
Posts: 3398
Joined: Wed Jun 21, 2006 2:40 pm

Post » Wed Mar 09, 2011 1:05 pm

i reported an issue with black LOD objects a few weeks ago. it only appears in elsweyr, i have HDR enabled and everything installed correctly. all LOD objects that belong to the RAEVWD anequina package turn black and back to normal even without moving, kind of flickering, but mainly black. original anequina, main RAEVWD and vanilla LOD shows correct.

now i narrowed it down to this file: ulldrockdesertsand02_far.nif

if remove it the rest of the LOD object show correct. (i still get that fancy colors sometimes, but also only in elsweyr.)
i used the regular mesh that comes with anequina and added _far for testing and then only the top of the stone and all RAEVWD anequina LOD was black.
User avatar
Andrea Pratt
 
Posts: 3396
Joined: Mon Jul 31, 2006 4:49 am

Post » Wed Mar 09, 2011 4:55 am

That probably has something to do with the alpha channels on the textures more than the meshes themselves since the meshes are clean. I'll probably have to fix the textures and update the package. Getting rid of alpha channels on textures that don't need it seems to be beyond the capabilities of Paint.NET or GIMP so one needs to pull some hackish tricks to get around that.
User avatar
Killah Bee
 
Posts: 3484
Joined: Sat Oct 06, 2007 12:23 pm

Post » Wed Mar 09, 2011 10:06 am

Getting rid of alpha channels on textures that don't need it seems to be beyond the capabilities of Paint.NET or GIMP so one needs to pull some hackish tricks to get around that.


Gimp

You should be able to make the alpha channel transparecny visible. and remove it with Gimp

Solution 1

Open texture, go to, menu bar

Layer
Transparency select Threshhold Alpha
Move slider all the way left, should make the alpha channel visible in texture now, ok
Then resave has dxt1 that will remove the alpha channel.

Solution 2

under the same Menu is Remove Alpha Channel has well

Paint.net does not have the feature ( you can extract the alpha channel but you cannot remove it directly, unless you save in dxt1 format)
User avatar
Yvonne
 
Posts: 3577
Joined: Sat Sep 23, 2006 3:05 am

Post » Wed Mar 09, 2011 1:05 am

That probably has something to do with the alpha channels on the textures...


i removed the alpha channel on the normal map. now it works. even the fancy colors seem to be gone, but i have to test this further...

the alpha channel on the normal map is used as specular map and useless for distant objects anyway, so it's wise to remove it to save memory at least.

i work with photoshop and could easily run a batch on all the files if you need help... :)
User avatar
Andrew Tarango
 
Posts: 3454
Joined: Wed Oct 17, 2007 10:07 am

Post » Wed Mar 09, 2011 8:33 am

So I tried the GIMP method outlined. GIMP was indeed able to remove the alpha channels, but there were large swaths of white left behind even when the transparency threshold was 0. So that didn't work.

Did what I usually do in these situations. Fired up WTV, toggled off the alpha, screenshotted that, pasted to Paint.NET, then copied the square I needed over the top of the previous texture. It sounds like a lot, but it takes maybe 2 minutes for each and produces perfect results.

@side777: Are you telling me then that there's absolutely zero purpose in having an alpha channel on a normal map for VWD use? If that's the case, then absolutely yes, if you can get them stripped without the process destroying the remaining image I'm all for it. For the main RAEVWD and SI packages. the QTP3 texture pack could probably use that too. Not sure how significant the memory savings will be but I imagine it may help with framerates too?

I'll have the Elsweyr ones manually stripped pretty quickly here, so no need to do those.
User avatar
jodie
 
Posts: 3494
Joined: Wed Jun 14, 2006 8:42 pm

Post » Wed Mar 09, 2011 4:56 am

Are you telling me then that there's absolutely zero purpose in having an alpha channel on a normal map for VWD use?

Not sure how significant the memory savings will be but I imagine it may help with framerates too?


yes. the alpha channel on the normal map is used as the specular map only. there's no specular lighting in the distance, so...

i don't think it affects framerate, the specular map gets likely just ignored. i don't know if you always use full size (i.e. same size as the color map) normal maps, which is also not necesary for VWD... but a 256x256 alpha channel is big enough to be worth the deleting, because these textures are almost always loaded.

i will have a look at the packages, with photoshop it takes 2 minutes to process ALL textures...
User avatar
Rachel Cafferty
 
Posts: 3442
Joined: Thu Jun 22, 2006 1:48 am

Post » Wed Mar 09, 2011 4:16 am

Hello Arthmoor,

I have just one question about the village/Town, which can be seen on the right side of your picture. Which Mod is it . Is it one of yours?

Keep up the good work.

Thanks.
User avatar
Zosia Cetnar
 
Posts: 3476
Joined: Thu Aug 03, 2006 6:35 am

Post » Wed Mar 09, 2011 12:18 pm

That's Verona Bay, from the Verona House Bloodlines mod.

Also for those keeping track, the Anequina extension for RAEVWD was updated last night. I didn't realize that a bunch of the textures necessary for it to work are not included with the January 2010+July 2010 updates for Anequina. Also stripped alpha channels from the normal maps, which eliminated the sparkly rock problem.
User avatar
JERMAINE VIDAURRI
 
Posts: 3382
Joined: Tue Dec 04, 2007 9:06 am

Post » Wed Mar 09, 2011 9:45 am

hI !
I just installed RAEWD and must run TES4LOD I guess - only I want to be able to backup my current install in case something goes wrong - so which folders should I backup (in which folders does TES4LOD place its files) ?

Thanks :)
User avatar
Tanika O'Connell
 
Posts: 3412
Joined: Fri Jan 26, 2007 1:34 am

Post » Wed Mar 09, 2011 2:21 am

hI !
I just installed RAEWD and must run TES4LOD I guess - only I want to be able to backup my current install in case something goes wrong - so which folders should I backup (in which folders does TES4LOD place its files) ?

Thanks :)


Nothing much to fear regarding that, but the files go into Data\DistantLOD so that's what you should back up in that case.
User avatar
Katie Pollard
 
Posts: 3460
Joined: Thu Nov 09, 2006 11:23 pm

Next

Return to IV - Oblivion