[RELz]Anti Aliasing White Line Remover

Post » Wed Sep 01, 2010 10:02 pm

Suggested by RebelMax, made by me, tested by Rebelmax, unknown to basketball players.
Removes the white line some people get when using forced Anti Aliasing.
See screenshot with download on tesnexus (screenshot proposed, prepared and uploaded by RebelMax) :

http://www.tesnexus.com/downloads/file.php?id=36156

Readme, same as on nexus for now :
Requires OBSE (whichver works with OBGE version that you use) and OBGE v2 or later.

This HLSL shader for OBGE replaces the top most row and leftmost column with black so that it blends in with the screen and is less noticeable than the white line. I can't test it much on my machine since I don't get the problem.

Load in the usual way using shaderlist.txt, the support plugin doesn't support it right now but may in future versions allow loading of any shaders.

I guess it will work best when loading very last, though it probably won't hurt much elsewhere.

Thanks very much to RebelMax who requested this and tested it quickly and thoroughly. Here's what he said :

"I've used Testing Hall for... erm, testing Turned off Vsync and removed Oblivion Stutter Remover to remove FPS cap completely, and got in the corner to maximize FPS. No shaders were used except yours.

"Test 1" shader -> 195 FPS.
"Test 3" shader -> 195 FPS.
No shaders -> 222 FPS.

In normal conditions, I run Vsync with OSR capped at 30 FPS, so there's no chance one can notice such a tiny FPS hit. I think the same applies to those who use 60 FPS cap"


I tested in the same way on my Nvidia 9800GT :
No mods loaded, vysnc off, looking at the wall in a new game in the prison cell to maximize speed :
No shader : ~490 FPS
Shader aawr.fx : ~413 fps

User avatar
Tyrel
 
Posts: 3304
Joined: Tue Oct 30, 2007 4:52 am

Post » Wed Sep 01, 2010 1:10 pm

Yay! Thank you again for getting rid of that bloody frame thing! :D
User avatar
Josh Lozier
 
Posts: 3490
Joined: Tue Nov 27, 2007 5:20 pm

Post » Wed Sep 01, 2010 5:01 pm

Finally a fix for this. Glad to know someone else has the same issue.
User avatar
Anthony Santillan
 
Posts: 3461
Joined: Sun Jul 01, 2007 6:42 am

Post » Wed Sep 01, 2010 12:15 pm

I think you should mention that in order to get OGE working together with AA, one must change their bUseDepthBuffer=1 to bUseDepthBuffer=0 in OBGE.ini (Win7: C:\Users\\Documents\My Games\Oblivion. WinXP: C:\Documents and Settings\\My Documents). Of course this leaves you with only some of the OGE shaders that will work, because most of them use depth buffer to do their magic, but you can't make those shaders work together with AA anyway.
User avatar
Len swann
 
Posts: 3466
Joined: Mon Jun 18, 2007 5:02 pm

Post » Wed Sep 01, 2010 7:10 pm

I think you should mention that in order to get OGE working together with AA, one must change their bUseDepthBuffer=1 to bUseDepthBuffer=0 in OBGE.ini (Win7: C:\Users\\Documents\My Games\Oblivion. WinXP: C:\Documents and Settings\\My Documents). Of course this leaves you with only some of the OGE shaders that will work, because most of them use depth buffer to do their magic, but you can't make those shaders work together with AA anyway.


Do you know exactly which shaders DO work with bUseDepthBuffer=0? I know SSAO, NormaFilterAA, CelShader+EdgeAA, DoF and Godrays doesn't - but do all the others? Probably a question for the OBGE thread, but it's quite on topic here aswell.
User avatar
liz barnes
 
Posts: 3387
Joined: Tue Oct 31, 2006 4:10 am

Post » Wed Sep 01, 2010 4:25 pm

Do you know exactly which shaders DO work with bUseDepthBuffer=0? I know SSAO, NormaFilterAA, CelShader+EdgeAA, DoF and Godrays doesn't - but do all the others? Probably a question for the OBGE thread, but it's quite on topic here aswell.


Shaders that DO work:

ColorEffects.fx
ColorMood.fx
ENBColorEffect.fx
HLSLbleachbypass.fx
HLSLColorGrading02.fx
HLSLvignette.fx
NormalFilterAA.fx
obsharpen.fx
And presumably Retro Shaders by localhosed

Others in the official shader pack don't.
User avatar
Trevi
 
Posts: 3404
Joined: Fri Apr 06, 2007 8:26 pm

Post » Thu Sep 02, 2010 12:18 am

Don't you just have to disable the depth buffer for the in game antialiasing setting, not when it is forced through the drivers?
User avatar
Thomas LEON
 
Posts: 3420
Joined: Mon Nov 26, 2007 8:01 am

Post » Wed Sep 01, 2010 9:29 pm

Don't you just have to disable the depth buffer for the in game antialiasing setting, not when it is forced through the drivers?

No, you have to disable HDR to use the in-game AA setting, not depth buffer. Regardless, I've seen reports of color lines being visible even with in-game AA.
User avatar
Chloe Lou
 
Posts: 3476
Joined: Sat Nov 04, 2006 2:08 am

Post » Wed Sep 01, 2010 11:36 am

Sweet. At least something can be done about it. ATI doesn't seem the slightest bit interested in fixing their buggy drivers that caused this to begin with. I'm not about to throw out AA since the game looks too blocky without it.

Seems a shame that none of the really good shaders will work if you're forcing AA at the driver level. I just can't see how anyone tolerates playing with blockfest.
User avatar
Michelle davies
 
Posts: 3509
Joined: Wed Sep 27, 2006 3:59 am

Post » Wed Sep 01, 2010 11:18 pm

Seems a shame that none of the really good shaders will work if you're forcing AA at the driver level. I just can't see how anyone tolerates playing with blockfest.

I really never cared whether I have AA or not. I don't understand the extreme some people go with complaints because they cannot use AA. Not saying you Arthmoor are doing so, just saying in general i've seen some really absurd comments made...

With that said, OBGE has a very nice AA shader alternative. It looks just as good in my opinion. ENB also has it's own AA shader so it seems as well.
User avatar
Ezekiel Macallister
 
Posts: 3493
Joined: Fri Jun 22, 2007 12:08 pm

Post » Wed Sep 01, 2010 10:14 am

I suppose I should try out the OBGE version of AA. I've wanted to try out the godrays shader too but it's always been said that it won't work with AA, and the blockfest bugs me too much to give up AA. I suppose the irony in using the OBGE version of AA is that with the driver forced option off, the line problem would go away and I'd not need this nifty little shader :P
User avatar
roxxii lenaghan
 
Posts: 3388
Joined: Wed Jul 05, 2006 11:53 am

Post » Wed Sep 01, 2010 5:42 pm

I suppose I should try out the OBGE version of AA. I've wanted to try out the godrays shader too but it's always been said that it won't work with AA, and the blockfest bugs me too much to give up AA. I suppose the irony in using the OBGE version of AA is that with the driver forced option off, the line problem would go away and I'd not need this nifty little shader :P

Heh, yeah that is true.

It's not so much Godrays, but the way the OBGE team exposed the depth buffer, which is causing a conflict of forced AA users and OBGE. If I recall, I believe Scanti mentioned before he vanished that he was almost on the brink of fixing the AA conflict. Sadly thus, no one has seen Scanti in the OBGE threads for some time now.
User avatar
victoria gillis
 
Posts: 3329
Joined: Wed Jan 10, 2007 7:50 pm

Post » Wed Sep 01, 2010 4:01 pm

I don't really know much about using AA in oblivion, this discussion has been educational for me. Yes though, all my retro shaders work without the depth buffer so they'll work with forced AA if I understand correctly.
User avatar
Tiff Clark
 
Posts: 3297
Joined: Wed Aug 09, 2006 2:23 am

Post » Wed Sep 01, 2010 5:11 pm

At least SSAA should work from my experiences. Because it is just about resolution and it is compatible with almost anything in theory. I hope Scanti comes back with a fix.

PS. I used NFAA in these shots, I don't think you can see any jaggies.
Spoiler

http://i51.photobucket.com/albums/f386/vtastek/oblivion/ss01/Oblivion2010-12-2002-45-19-21.jpg

http://i51.photobucket.com/albums/f386/vtastek/oblivion/ss01/Oblivion2010-12-1807-35-50-75.jpg

http://i51.photobucket.com/albums/f386/vtastek/oblivion/ss01/Oblivion2010-12-2002-55-59-26.jpg

http://i51.photobucket.com/albums/f386/vtastek/oblivion/ss01/Oblivion2010-12-2002-55-13-62.jpg

http://i51.photobucket.com/albums/f386/vtastek/oblivion/ss01/Oblivion2010-12-2002-53-22-23.jpg

http://i51.photobucket.com/albums/f386/vtastek/oblivion/ss01/Oblivion2010-12-2002-51-37-81.jpg

http://i51.photobucket.com/albums/f386/vtastek/oblivion/ss01/Oblivion2010-12-2002-50-34-14.jpg

http://i51.photobucket.com/albums/f386/vtastek/oblivion/ss01/Oblivion2010-12-1821-20-32-81.jpg

http://i51.photobucket.com/albums/f386/vtastek/oblivion/ss01/Oblivion2010-12-1807-35-38-54.jpg

http://i51.photobucket.com/albums/f386/vtastek/oblivion/ss01/Oblivion2010-12-1802-30-57-82.jpg


It is not perfect but it makes the problem highly tolerable. I turned DoF off in these except the last one so it should give a better idea about the AA effect. They are all in original resolutions and direct ingame screenshots.
User avatar
bimsy
 
Posts: 3541
Joined: Wed Oct 11, 2006 3:04 pm

Post » Thu Sep 02, 2010 12:49 am

I suppose the irony in using the OBGE version of AA is that with the driver forced option off, the line problem would go away and I'd not need this nifty little shader :P

Heh, I use forced 2xAA with transparency supersampling + NFAA and it looks and runs http://img218.imageshack.us/img218/5092/screenshot29.png. Although plain NFAA looks really good on static pictures, in motion it looks not so great (common shader AA downside, I suppose). Plus it doesn't do well with smoothing tree leaves. So that's where 2xAA+STAA comes in handy, and applying NFAA afterwards makes everything, especially foliage, look like a quality CG.

Off-topic alert :tongue:
User avatar
Veronica Flores
 
Posts: 3308
Joined: Mon Sep 11, 2006 5:26 pm

Post » Wed Sep 01, 2010 5:34 pm

MLAA is another candidate, worthy of at least a try, IMO. If you have an ATI (AMD) card, that is. Probably 5xxx or above.

My HD5870 can do it just fine, with minimal impact, and to good effect. However, NFAA is also nice. And sometimes I've simply gone with forced "hardware" AA. Never once thought of combining types though... interesting!

But anyway, although I'm not getting said issue these days, I have done at various times, and if I ever do again, I'll be sure to use this shader. Thanks! :goodjob:
User avatar
Auguste Bartholdi
 
Posts: 3521
Joined: Tue Jun 13, 2006 11:20 am

Post » Wed Sep 01, 2010 3:45 pm

Out of curiosity would it be possible to sample neighboring pixels that aren't on the screen border and generate a smooth replacement instead of just using black?
User avatar
Chloe Mayo
 
Posts: 3404
Joined: Wed Jun 21, 2006 11:59 pm

Post » Wed Sep 01, 2010 2:42 pm

I just can't see how anyone tolerates playing with blockfest.


FPS issues on slower rigs? Giving you that for free.
User avatar
Samantha Mitchell
 
Posts: 3459
Joined: Mon Nov 13, 2006 8:33 pm

Post » Wed Sep 01, 2010 6:02 pm

Out of curiosity would it be possible to sample neighboring pixels that aren't on the screen border and generate a smooth replacement instead of just using black?


It's possible to sample the nearest pixels in corresponding rows and columns and use those instead of black. I guess that would look okay too, but maybe you mean something else that I don't understand? The thing I describe is what Rebelmax suggested already but I hadn't programmed it yet.
User avatar
-__^
 
Posts: 3420
Joined: Mon Nov 20, 2006 4:48 pm

Post » Wed Sep 01, 2010 9:04 pm

It's possible to sample the nearest pixels in corresponding rows and columns and use those instead of black. I guess that would look okay too, but maybe you mean something else that I don't understand? The thing I describe is what Rebelmax suggested already but I hadn't programmed it yet.

Duplicating adjacent pixels is one way to do it. I suggested sampling and blending so the duplication effect is less obvious, but in all honesty it's probably not necessary since a 1px edge is tiny at high resolution.
User avatar
Jessie Butterfield
 
Posts: 3453
Joined: Wed Jun 21, 2006 5:59 pm


Return to IV - Oblivion