Really short story (after two months of playing with consistent CTDs): I repacked the Dawnguard.bsa file to include the USKP-DB files with BSAopt set at '10(quick)' compression. Two days ago, I noticed that the vanilla bsa is not compressed. So, I repacked mine without compression. CTDs have stopped.

Before I go into detail of how I discovered this I have to ask... Does this make any sense? I have also noticed that some vanilla BSAs are not compressed and some are. The game never had a problem loading the assets from the Dawnguard.bsa (when it didn't CTD). So, it could possibly be a game engine flaw handling the memory for BSA decompression.

I would not be asking this if I have not already spent several days playing normally and even reloading much older saves to test the situations that always caused the CTDs. I am reasonably sure this is not just a coincidence.

These are:

1) A dragon spawns when I fast travel to or leave the interior of Falkreath.

2) Vampires spawn in Whiterun.

3) I approach the Whitewatch Tower for the first time, 3 bandits spawn.

4) Less frequent: I enter the Dawnguard HQ.

Thanks.

Yes, compressing certain types of resources causes a crash. Just look at what's in each of those original game BSAs that aren't compressed and you should be able to figure out which ones. Since the Dawnguard BSA isn't segregated by type of file they left the whole thing uncompressed.

The core problem seems to be that certain resources must be accessed very quickly when they are needed and the delay (and CPU power) it takes to uncompress them is the problem. Interestingly if you uncompress all of those resources onto a slow hard drive you'll can also get the same type of crashing behaviors and this whole issue has been a problem since Morrowind.

Depending on your system you might be able use compression at one of the other settings, but it's probably not worth the time and effort it would take to see which ones might work for you. Just having the resources in a BSA file gives you the biggest performance boost. Any performance increase from using compression is entirely dependent on just how powerful your CPU is compared to the rest of your system.

It looks like anything that has voice files (data\sound\voice\), because the vanilla files that have them are not compressed. Those with music and fx can be compressed.

In fact, the Dawnguard.bsa files has voices for dragons in it. Finally I have an explanation of how it caused the CTDs for dragons, which I thought would only use Skyrim base game assets.

It also explains how my packaging of many other mods into packed BSAs has not caused a problem. None of them have voice files.

I only stumbled on this myself trying to repack the vanilla BSAs.

There's probably not much to be gained by merging BSA contents anyway.

Probably so, but I did it as a last ditch desperate effort to improve performance and reduce CTDs. I also optimized all textures with DDSopt. Afterwards though I learned some things.

Using SPM I found that my GPU is my weak point. With ETaC installed my FPS suffered majorly in Morthal and Dawnstar, even after I resized the ETaC textures to 1K or less. I decided to see what would happen if I removed the high-res texture pack (which I optimized but did not resize). My FPS returned to normal levels. I found that at 1440x900 I did not need most of the larger textures. For the first time, I was also able to install SMIM (after reducing most of its textures way down).

The only problem remaining was the consistent CTDs. Now that I figure them out (redid the Dawnguard and Dragonborn BSAs without compression), my game is running extremely smoothly and I have only gotten one CTD.

So I don't know if the BSA boogie I did helped or not.

I had no idea, thanks for the information. I'll have to be more careful in what I do with my BSA's in future.

A lot of people don't know this. I just found that the Wyrmstooth.bsa file has voices and is packed.

EDIT: BTW, I did check the BSAs of the USKPs. They are not packed. No surprise really, since Arthmoor knows all.

EDIT2: The Wyrmstooth author says it's a known issue with music files (which are also in the BSAs I repacked and in Skyrim - Sounds.bsa, which is not packed).

So, is it music and voices then?? Fx files are in Skyrim - misc.bsa, which is packed.

BSAopt has been broken for a long time.

Heh, that's because our team was one of the ones who discovered this the hard way

Really ?

I have used it for everything to do with BSAs over the last four years and not experienced any problems

If you forget to select the right game in the Game menu you will have problems ( wrong BSA format results, it defaults to Oblivion format BSAs )

If you use too high a compression level for the files you are including you will have problems ( as being discussed )

But thats all I know of, and are user end lack of knowledge problems, not BSAOpt

http://obge.paradice-insight.us/wiki/BSAopt where all the documentation was, is definitely broken

Edit : Here's the wayback machine last snapshot https://web.archive.org/web/20141122031851/http://obge.paradice-insight.us/wiki/BSAopt

THANK YOU, I've been looking for some documentation for a long time.

With my old slow pc compressed bsa's which contained any papyrus scripts also resulted in massive script lag. Where uncompressed didn't have this issue. Even with my new pc I still repackage most bsas to be uncompressed just to avoid any headache, and I also hate having hundreds of loose files cluttering my data folder.

I also use BSAopt and have never encountered any issues that where not a result of user error.

There is nothing wrong with BSAopt!

I would like to know if the compression applies to voices (anything in the data\sound\Voice directory) or music (anything in the data\music\ directory) or both. It would be cool if others didn't have to learn about it the hard way too.

Both.

Compression cannot cause script lag. Scripts are decompressed/loaded once (topic info fragments being the one exception in order to reduce memory load) and just run in memory past that.

Compression and the archives generally improve performance because your hard drive is far slower than your CPU. The one exception would be mp3 audio which is already compressed and adding zlib compression on top doesn't make the file any smaller, thereby slowing down the load for no advantage.

If your CPU is particularly slow the added load to do the decompression can be a problem. I once also had a really old machine that had a very slow CPU and would get better performance with uncompressed BSAs. But that was because I had upgraded the RAM enough that the most used BSAs were actually buffered in RAM by the OS so I wasn't actually going all the way to the hard disk to get the data. On every other (reasonably balanced) system I've had since, the compressed BSAs give better performance, but only if the compressed BSA is smaller than the uncompressed BSA.

If you try to compress various types of resources you'll see that some of them pack together much better than others. As SmkViper says, compressing audio and video using general algorithms like zlib is a waste of resources. It can even make the resulting "compressed" file bigger than the equivalent "uncompressed" one if you're really unlucky. One way to approach the problem is to create both a compressed and an uncompressed BSA then compare them. If they are reasonably close in size you're probably better off using the uncompressed version. But except for audio and video (and JPEG images which Skyrim doesn't use) there's almost always enough difference to justify using the compressed version.

I have had better luck with compressing wav files to xwm (except \sound\fx) and then packing with no compression. I have had great luck packing .dds, .hkx, .nif, .pex, and even .swf. I had some problems once with \interface folder but I never comfirmed whether it was user error.

BTW, you can create a mixed archive with BSAOpt by choosing the same output file for multiple iterations. I usually put everything less than 1kb and all sound files into a separate folder. I use BSAOpt to compress the first, then choose the second folder as source but leave the output file the same, change the compression to none and hit pack.

Scripts would start later than they should have and sometimes not at all if they where compressed. Whereas loose files or uncompressed bsa's had no issues like that. Might not have been script lag per se, but it did affect performance and playability. Most likely it was my old pc.

No, they actually wouldn't. The game would hitch instead of seeing a delay, and even then that's only the very first time said script was encountered in the game. For things like quests they load in the main menu before you'd even be in game.

Whatever you were seeing was either due to low framerate or poorly written scripts, not compression or archives.