[Concept] Going a step beyond merging

Post » Wed Mar 09, 2011 11:26 am

It seems that all the major overhauls have a huge impact on performance, due to the sheer amount of replacement scripts and such added to, modified, or removed from the core of Oblivion. It's no surprise FCOM can bring Oblivion to its knees, considering it is likely to have more altered logic than new logic. Bash has helped the problem significantly by merging stats and modifying leveled lists, but I feel there may be a way to eliminate much of the overhead that remains.

Would importing large mods into Oblivion.esm be plausible or practical? I would imagine the lack of on-the-fly modification of game logic would considerably cut back on the amount of resources required. Why would Oblivion suffer like it does now when any given operation does not require that the location be looked up, anolyzed of NPCs that would spawn, interrupt the spawning of said NPCs, and spawn other NPCs from a dynamically built list in their place?

If the game logic was recompiled to natively implement the features of a major modification, would the game not run a lot smoother and with fewer resources? I know mods that rely on OBSE will likely not work this way, and mods that tweak small amounts of data or just add something new wouldn't be necessary, but importing the backbone of major game changing mods like OOO (and especially FCOM) into Oblivion.esm would yield great results.

No, I don't expect this to happen. It's just a thought, but curiosity still remains. How much work would be required for something like this? Would it be worth it?



What do you guys think?
User avatar
Kaylee Campbell
 
Posts: 3463
Joined: Mon Mar 05, 2007 11:17 am

Post » Wed Mar 09, 2011 11:06 am

Short answer: Just install less. We are very well past the point of having a game that can handle anything thrown at it.

Longer answer: Bad idea. By altering the Oblivion.esm you may void its functioning in relation the exe (or not) and would absolutely break the functioning of many mods (most mods) that depend on the Oblivion.esm. This in turn would necessitate more and more mods integrated into it and who would be responsible for doing this merging and what mods would then get included in such a way that it pleases most people.

So say you do just an FCOM version. Then mods that come out after that (and the incredibly large back catalog) would have to be released as both Oblivion.esm and Oblivion-FCOM.esm compatible.

Plus it would mean re-engineering how most mods are even designed to function meaning rewrites galore.

It would remove an essential component of mods - modularity.

And probably many other reasons that didn't immediately fall out of my broken head.
User avatar
Javaun Thompson
 
Posts: 3397
Joined: Fri Sep 21, 2007 10:28 am

Post » Wed Mar 09, 2011 8:00 am

Can't be done. I've attempted similar things only with the UOP/USIP as the candidates for merging. The mere act of writing Oblivion.esm back out using TES4Edit broke it beyond any possibility of functioning. Yes, I had to rename it to trick it into even letting me try. But when the game loaded ( and I was surprised it did ) there was not a single cell of landscape anywhere to be found. Somehow the process of writing the data back out blew up the entire Tamriel worldspace. All of the objects, trees, etc. loaded and static items were where they belonged, but all the NPCs and havok enabled items fell immediately into the water.

In short, don't waste your time. I told someone else just recently about deep rabbit holes and wanting to dive down them, this would be the mother of all deep rabbit holes and really doesn't have an end.
User avatar
Felix Walde
 
Posts: 3333
Joined: Sat Jun 02, 2007 4:50 pm

Post » Tue Mar 08, 2011 11:31 pm

From what I know of the engine's inner workings I doubt such task will have the benefits you're proposing.

All plugins and masterfiles are loaded right at the start, even before you load or start any game. That's why it throws you back to desktop immediately after the intro menu when a master for one or more plugins is missing. As far as I know there is no applying of mods' changes or anything done while you're playing the game.
The slowdowns you're experiencing with big overhauls don't come from the amount of changes in them but from the amount of added content and quality increases.

Let's say you have an overhaul which adds more creatures/NPCs to the game world. Now suddenly the number of actors the game engine has to process is 10x the usual amount. You're wondering why this slows down processing? Let's merge those changes into the Oblivion.esm (if it could be done that is) and we have what? Still 10x the number of actors to process. What did we win? Nothing.

The resolutions of all items and landscape textures was increased 4x for better visual quality. This makes your graphics card stress its ram more, of course, and also the system ram because it can't always have it all loaded in its own. Will this change even the slightest when you include these changes into Oblivion.esm? Of course not.

Let's say there are for example 10.000 new scripts executed per frame, whether they are running on objects, quests or spells. Can this slow down your game? Of course it can. Will merging those scripts into Oblivion.esm reduce this number any? Nope.

Simple changes like to stats of items or even game settings are merged by the engine already when it starts up. This does not happen while ingame.
Now you merging those changes into Oblivion.esm by hand might make the game start up faster (technically it has little to no mods to merge anymore)... but that's it already.

...I hope you get what I'm aiming at.

edit: Maybe there are such things as "changes to game logic" when you add or remove any mods and the internal base data structure as created by the initial start-up merge of plugins and masterfiles is changed, but still these would not happen over and over every time you load your saves, and especially not any time "after" loading of your saves. Once the changes were applied, the game stores the new structure into your savegames. That's why savegame size increases continually the more mods you install and the more places you visit/actions you undertake/etc.
User avatar
Amy Masters
 
Posts: 3277
Joined: Thu Jun 22, 2006 10:26 am

Post » Wed Mar 09, 2011 8:04 am

I was going to say mostly the same thing Drake but there is one more thing and that if you really want to increase performance (besides OSR) one of the biggest things you can do is decrease spawns. Since FCOM adds many spawn points you would need to decrease spawns further then what is in default Oblivion to equal Oblivions spawn count but you could also delete some spawn points to help if you can use the Construction set.
User avatar
Kevin Jay
 
Posts: 3431
Joined: Sun Apr 29, 2007 4:29 am

Post » Tue Mar 08, 2011 9:43 pm

To be honest you'd be better off praying that someday, someone decided to just completely remake Oblivions engine like certain people are doing for Morrowind(CrystalScrolls and OpenMW). The method your suggesting and which Arthmoor seems to have experimented with is just flogging a dead horse, alot of effort for no gain and massive headaches(in general and all the mods your tinkering with the ESM could end up breaking).


A long, long time ago i had started coding a replacement engine for Oblivion in my spare time from the ground up but stuff happened IRL, other projects caught my attention and just fading interest for Oblivion caused me to drop it. The only advice i can offer anyone after having a good dig through the Gamebryo Engines' innards is to go for a complete replacement, there really is only so much user made patches such as WEOCPS and OSR can do. The best you can hope for is to make it stable and even that means very little as long as Oblivion uses just a single core. People will throw silly modlists at it and overload the engine/hardware making it crashes a matter of when rather than if.

Just my two cents.
User avatar
Josee Leach
 
Posts: 3371
Joined: Tue Dec 26, 2006 10:50 pm

Post » Wed Mar 09, 2011 7:30 am

The game does not load changes "on the fly", but rather during the main load at the beginning of the game (the "big O" screen). Doing this would shorten that one single load, but would not improve performance nor influence any other load times. It would also be a compatibility nightmare, a hellish amount of work, and also would almost certainly be nixed by Bethesda.
User avatar
Lexy Corpsey
 
Posts: 3448
Joined: Tue Jun 27, 2006 12:39 am

Post » Wed Mar 09, 2011 1:30 am

Actually you could create a project to do this by creating a program that would merge the changes with the Oblivion.esm and not have problems with Bethesda like they do with Weidu and the baldur's gate games.
User avatar
Steven Hardman
 
Posts: 3323
Joined: Sun Jun 10, 2007 5:12 pm

Post » Wed Mar 09, 2011 2:06 am

Except for what I pointed out above. You'll break it by trying.
User avatar
Nick Tyler
 
Posts: 3437
Joined: Thu Aug 30, 2007 8:57 am

Post » Wed Mar 09, 2011 6:43 am

Assuming what you guys say is correct, why is it that an uncleaned AFK_Weye brings the system to its knees while in the area? If the stuff is compiled at launch time, why would it affect the runtime performance so much? Furthermore, Oblivion performance still suffers greatly while using FCOM reduced spawns. Counting the amount of entities, you would actually have fewer in memory, yet have much higher CPU load. I know added content is a burden upon the game, but it's not just about memory usage.

On the issue of Oblivion.esm being unmoddable, is this one of the explanations why there hasn't been a tool to run Oblivion on Fallout 3's less horrible engine? That'd be a sight, but if there's a master ESM issue, then I can see it being a problem.
User avatar
Sunnii Bebiieh
 
Posts: 3454
Joined: Wed Apr 11, 2007 7:57 pm

Post » Wed Mar 09, 2011 4:24 am

What do you mean by an "uncleaned" AFK_Weye?

Adding that content to the master will not change the reality of the fact that it's content that wasn't there before. Weye was one house and an inn. AFK_Weye transforms it into a proper sized village. There isn't anywhere for the memory requirements to go but up. There's also nowhere for the relative performance of the area to go but down, even if for MOST of us that downward drop is barely noticeable.

There hasn't been a tool to run Oblivion.esm on Fallout 3 (or F:NV for that matter) because in all likelihood it would be illegal for such a thing to exist and be distributed. Also pretty sure that Oblivion.esm would be missing numerous chunks of critical data the FO3/F:NV engines would need from it.

You appear bent on trying, despite the warnings, so all I'll say at this point is make backups and try it. When it explodes, we'll watch for the flash on the horizon :)
User avatar
christelle047
 
Posts: 3407
Joined: Mon Apr 09, 2007 12:50 pm

Post » Wed Mar 09, 2011 9:12 am

There hasn't been a tool to run Oblivion.esm on Fallout 3 (or F:NV for that matter) because in all likelihood it would be illegal for such a thing to exist and be distributed.
Not exactly. Conversion of assets from game to game is prohibited, but the actual game data should be fine for personal use.
Also pretty sure that Oblivion.esm would be missing numerous chunks of critical data the FO3/F:NV engines would need from it.
This, however, would be an issue - but a "tool to run Oblivion.esm on Fallout 3" would, by definition, include some way of fixing that. The main problem, however, is that Fallout 3 has guns instead of magic. Sure, things are still referred to as magic effects and alchemy internally, but the game's been recoded to use guns instead of fireballs, and no shields at all. Plus, there's the issues of body slots (armour in F3 comes as suits), the entire levelling-up system, the generalisation of melee weapons, the lack of Fallout 3 equivalents to popular Oblivion plugins, and much more.
User avatar
Josh Trembly
 
Posts: 3381
Joined: Fri Nov 02, 2007 9:25 am

Post » Wed Mar 09, 2011 10:12 am

Furthermore, Oblivion performance still suffers greatly while using FCOM reduced spawns. Counting the amount of entities, you would actually have fewer in memory, yet have much higher CPU load. I know added content is a burden upon the game, but it's not just about memory usage.

On the issue of Oblivion.esm being unmoddable, is this one of the explanations why there hasn't been a tool to run Oblivion on Fallout 3's less horrible engine? That'd be a sight, but if there's a master ESM issue, then I can see it being a problem.

Well reduced spawns will help but only slightly with FCOM. What would help FCOM greatly is reduced spawn points. It greatly increases the spawn points (areas where things spawn). I recall reviewing this with regard to Shezries town and the fort near Pells gate ... CorePC reported I think 5 spawn points near the fort next to pells gate. Normally there is one. This means an increase in some areas of 4-5 times as many areas that are spawning, so even if you use a reduced spawns plugin with FCOM that takes the number that spawn down there is still an great increase over vanilla - even if each point was set at vanilla numbers.

Your thinking seems to hinge on the conception that all the mods add extra processing by the virtue of them being more to read via being in separate containers (esp/esm) - not so. Designed to work this way. That is how they do their DLC too.

With F3 it is generally far more stable than Oblivion, but recently I've added mods to my fallout game that are bringing it over 210 active plugins (over 250 if counting all) and it is showing some sluggishness finally. Getting me to rethink some about my Oblivion installs.

I doubt they will raise the level of plugins that skyrim can take. I doubt that skyrim will be that sturdy either once it has 255 mods as well.
User avatar
Lynette Wilson
 
Posts: 3424
Joined: Fri Jul 14, 2006 4:20 pm

Post » Wed Mar 09, 2011 4:47 am

Not exactly. Conversion of assets from game to game is prohibited, but the actual game data should be fine for personal use.


You'll note I specifically mentioned distribution, and there's precedent for distribution of the tool which enables conversion being illegal under the EULA. Just go look for that M.... one for converting Morrowind assets to Oblivion.

I doubt they will raise the level of plugins that skyrim can take. I doubt that skyrim will be that sturdy either once it has 255 mods as well.


Given that Skyrim is on an entirely new engine, we have no idea how the mod system will work. They may try to stick with what they have, or come up with something entirely new. Maybe it won't even have a limit. Even if all they did was add one digit so FFF is the new max, that's 4094 (you lose 2 immediately - master file + save) plugins which I doubt anyone would ever see happen.

We might even get lucky and find Skyrim simply doesn't need as many mods to flesh the game out and running into limits won't be as much of an issue to start with.
User avatar
Manuela Ribeiro Pereira
 
Posts: 3423
Joined: Fri Nov 17, 2006 10:24 pm

Post » Tue Mar 08, 2011 10:42 pm

Actually that isn't how hexadecimal works and if they went further it would probably be ff ff. There will always be reasons to go past 255 mods and for me I have over 400 mods I'm planning on using for my game and more then 255 of them are not gameplay fixes and most are content mods.
User avatar
Penny Flame
 
Posts: 3336
Joined: Sat Aug 12, 2006 1:53 am

Post » Wed Mar 09, 2011 5:58 am

Given that Skyrim is on an entirely new engine, we have no idea how the mod system will work. They may try to stick with what they have, or come up with something entirely new. Maybe it won't even have a limit. Even if all they did was add one digit so FFF is the new max, that's 4094 (you lose 2 immediately - master file + save) plugins which I doubt anyone would ever see happen.

We might even get lucky and find Skyrim simply doesn't need as many mods to flesh the game out and running into limits won't be as much of an issue to start with.

All information is stored in bytes, a third hexadecimal digit would be 12 bits - or a byte and a nybble - and thus not something you can store. If it does up at all, it'll be to 2 bytes, or 65536 (-2 for master and save). I can't see a reason to not use 2 bytes, but then, I can't see much of a reason to go beyond one. The RAM hit might become significant on the memory starved consoles depending on how detailed the world is, but at literally an extra byte per thing, it's barely worth talking about.

As for the concept in the OP, I can't honestly see the advantage, and it sounds like a right hassle. Imagine rebuilding your bashed patch for every mod you add or change, then add the overhead of having to load the master file and edit the edited components and find places for the new components, then merge the .bsas because they won't load without a .es? with the right name, for little real benefit. Madness.
User avatar
Darren Chandler
 
Posts: 3361
Joined: Mon Jun 25, 2007 9:03 am

Post » Tue Mar 08, 2011 11:02 pm

You'll note I specifically mentioned distribution, and there's precedent for distribution of the tool which enables conversion being illegal under the EULA. Just go look for that M.... one for converting Morrowind assets to Oblivion.



Given that Skyrim is on an entirely new engine, we have no idea how the mod system will work. They may try to stick with what they have, or come up with something entirely new. Maybe it won't even have a limit. Even if all they did was add one digit so FFF is the new max, that's 4094 (you lose 2 immediately - master file + save) plugins which I doubt anyone would ever see happen.

We might even get lucky and find Skyrim simply doesn't need as many mods to flesh the game out and running into limits won't be as much of an issue to start with.



I really doubt Bethesda ever thought people would really download and install over 255 mods at any given point in time. I'm sure they wouldn't even think that the Script Extender or Graphics Extender would come out and do the things they do, but they have and as such I'm sure some of the most popular mods have really made Bethesda rethink many aspects of their own games and good for them.

Can't wait for Skyrim but I for one hope it's just as moddable as Oblivion. My one wish is that the engine is more stable and that it doesn't need many mods to begin with, hopefully most of the mods will simply be extra content since Skyrim will hopefully surpass a fully modded Oblivion.
User avatar
kristy dunn
 
Posts: 3410
Joined: Thu Mar 01, 2007 2:08 am


Return to IV - Oblivion