Version Control: A Primer : IV

» Tue Aug 25, 2009 8:44 pm

The Oblivion Construction Set has a functional version control system built-in. And yet a lot of people don't know that it exists, nor how to get it working. As more and more people look towards creating their own stand-alone world with the Oblivion Construction Set, many will find such knowledge useful. There's an article up on the http://cs.elderscrolls.com/constwiki/index.php/Version_Control which has been there for some time now, but it appears to be poorly linked and might not be straightforward to some. So here's a basic primer. This will concentrate on setting things up to work on a local computer, rather than a network.

To start with, open the ConstructionSet.ini file in C:\Documents and Settings\\My Documents\My Games\Oblivion\

Find the following and modify them to suit. This depends on the full network name of your computer, and the Oblivion install path. Don't just copy them verbatim. I'll pretend my computers full network name is PathFinder, and Oblivion is installed to C:\Games\ElderScrolls\Oblivion. The dollar sign ($) you see is supposed to be there.

bUseVersionControl=1SNetwork Path=\\PathFinder\C$\Games\ElderScrolls\OblivionSNetworkMasterPath=\\PathFinder\C$\Games\ElderScrolls\Oblivion\Data\SNewVersionBackupPath=\\PathFinder\C$\Games\ElderScrolls\Oblivion\VersionBackup\SLocalBackupPath=Data\CheckInBackup

Next, append the following to the end of the same file. This is needed to unlock some of the functions we need to use. Substitute your local computer login name for .

[WhoCanMerge]<username>=1

Now create two folders. Using the install path above, they would be C:\Games\ElderScrolls\Oblivion\VersionBackup and C:\Games\ElderScrolls\Oblivion\Data\CheckInBackup

And before you go any further, copy Oblivion.esm somewhere as a backup, just in case.

Now start up the CS and select Oblivion.esm from the Data Files list and load it. You will see the following ...

---------------------------Warning---------------------------TESBitArrayFile::OpenAndMapBitArrayFile could not find file 'Oblivion.fud'.Continue Running Executable?"Yes to all" will disable all Warnings this session.---------------------------Yes   No   Cancel---------------------------

Now if you see something different, then you have no active network connections. You need to rectify this. A dial-up connection is sufficient. Don't ask me why, it's just the way it is. Assuming you see what we wanted, click "Yes". Once loaded, you will see the button on the far left of the toolbar is now accessible. Leave it alone for now.

Open the Data Files window again. Oblivion.esm should be highlighted. Click on the Details button. Two confirmations will pop up. Click "Yes" for both.

The CS will parse the master file and then the File Details window will open. Enter CONTROL + SHIFT + B. You will see the following ...

---------------------------Confirm Bit Array File Update.---------------------------You have chosen to create/update the bit array files for the current file. This operation cannot be undone and may take a long time to complete.Do you wish to continue?---------------------------Yes   No---------------------------

We need these files. Select "Yes" to create them. Close the File Detail window, and select "OK" in the Date Files window to reload the master file.

Now your set. All the functions available can be found either in the Version Control screen, accessible via the button on the far left of the toolbar, or in the File Details window where we created the Oblivion.fid and Oblivion.fud files. The Version Control screen, accessible via the button on the far left of the toolbar is the one most useful. The functions found in the File Details window can be dangerous (and time consuming).

What's useful here, is merging to a master. If your creating plugins to run with Oblivion.esm, this is pointless. If your creating your own stand-alone master file, this is priceless. So lets test it on Oblivion.esm

Save a plugin file first. Always do this. Now find the player character in the NPC list, and delete everything from the players inventory. These are the items you start with. Add whatever you want to replace them and save again. I've added Mithril Boots, Mithril Greaves, and a huntsmans vest. The point here is to change something that is instantly visible when you start a new game.

If you now have another a look at the player character in the Object Window, you will see that it is highlighted in green. this means you have that form checked out. Click the Version Control button and you will see the same thing in the Version Control window. Highlight the player character in the Version Control window and click on "Check In."

Once that's done, close the Version Control window, open the Data Files window, and reload the master file. Now start a new game with only Oblivion.esm (and DLCShiveringIsles.esp if SI is installed) and look at what your character wears. Groovy huh? :dancing:

In the VersionBackup folder you created earlier you will find a copy of the pre-merge master file and plugin. In the CheckInBackup folder you created earlier you will find a second copy of the pre-merge plugin. Your plugin will still exist in the Data folder, and will still be loaded by the CS, but it will now be empty.

You can use the Version Control system to create a non-dependent master file from scratch as well.

The functions available from within the File Details window that I am aware of are as follows. Some of them will render your master file un-usable. Some require selecting specific forms or groups in the File Details window. I suspect that a lot of them are a left over from a time when the master file format was still being finalized, but the game world was well and truely under construction.

Ctrl+Shift+T

New form type string

Ctrl+Shift+U

---------------------------Confirm Size Update.---------------------------You have chosen to update the size of the selected group. This operation cannot be undone.Do you wish to continue?---------------------------Yes   No---------------------------

Ctrl+Shift+I

---------------------------Confirm ID Update.---------------------------You have chosen to update the form ID for one or more forms from the current file. This operation cannot be undone. Additionally, this will not remove any links in other forms to the old ID. Errors may be encountered on load.Do you wish to continue?---------------------------Yes   No---------------------------

Ctrl+Shift+O (letter O)

---------------------------Confirm ESM Conversion.---------------------------You have chosen to create a version of the current ESM file optimized for XBox loading. This operation cannot be undone.Do you wish to continue?---------------------------Yes   No---------------------------

Ctrl+Shift+P

---------------------------Confirm Child Sort.---------------------------You have chosen to sort all child sub groups in the file. This operation cannot be undone.Do you wish to continue?---------------------------Yes   No---------------------------

Ctrl+Shift+A

---------------------------Confirm Bit Array File Validation.---------------------------You have chosen to validate the bit array files for the current file. This operation may take a long time to complete.Do you wish to continue?---------------------------Yes   No---------------------------

Ctrl+Shift+S

---------------------------Confirm Form Sort.---------------------------You have chosen to re-sort the forms in the current file. This operation cannot be undone and may take a long time to complete.Do you wish to continue?---------------------------Yes   No---------------------------

Ctrl+Shift+F

---------------------------Confirm FORM Size Update.---------------------------You have chosen to update the size of the selected form. This operation cannot be undone.Do you wish to continue?---------------------------Yes   No---------------------------

Ctrl+Shift+G (Don't use this. You'll regret it.)

---------------------------Confirm Group Update.---------------------------You have chosen to update the form groups in the current file. This operation cannot be undone and may take a long time to complete.Do you wish to continue?---------------------------Yes   No---------------------------

Ctrl+Shift+L

---------------------------Confirm ESM Conversion.---------------------------You have chosen to create a version of the current ESM file with Tamriel land only. This operation cannot be undone.Do you wish to continue?---------------------------Yes   No---------------------------

Ctrl+Shift+Z

---------------------------Confirm Version Update.---------------------------You have chosen to update the file version of the master file. This operation cannot be undone.Do you wish to continue?---------------------------Yes   No---------------------------

Ctrl+Shift+C

---------------------------Confirm Group Update.---------------------------You have chosen to update the child group of the selected form. This operation cannot be undone.Do you wish to continue?---------------------------Yes   No---------------------------

Ctrl+Shift+V

---------------------------Confirm File Validation.---------------------------You have chosen to validate the loaded forms against the current file. This operation may take a long time to complete.Do you wish to continue?---------------------------Yes   No---------------------------

Ctrl+Shift+B

---------------------------Confirm Bit Array File Update.---------------------------You have chosen to create/update the bit array files for the current file. This operation cannot be undone and may take a long time to complete.Do you wish to continue?---------------------------Yes   No---------------------------

Ctrl+Shift+0 (number 0)

---------------------------Confirm Group Removal.---------------------------You have chosen to remove any empty groups in the current file. This operation may take a long time to complete.Do you wish to continue?---------------------------Yes   No---------------------------

Ctrl+Shift+Delete

---------------------------Confirm Delete.---------------------------You have chosen to remove one or more forms from the current file. Unlike setting the ignore flag, this operation cannot be undone. Additionally, this will not remove any links in remaining forms to the deleted ones. Errors may be encountered on load.Do you wish to continue?---------------------------Yes   No---------------------------

» Tue Aug 25, 2009 9:48 pm

Very good info :goodjob:

» Wed Aug 26, 2009 8:13 am

Thank Rocket...never player around with version control...I new about it..but could never get it quite work right...Now it may be possible...thank you once again...

» Wed Aug 26, 2009 2:27 am

[bump] for more exposure

Also, would it be possible to use Version Control to handle updating scripts? That is, I have a mod full of scripts and "functions" that I'd like to inject into other mods whenever I update it.

» Tue Aug 25, 2009 8:39 pm

With Corepc - I've known about this (and the Wiki article) but never messed with it much at all - which is also why the article is poorly linked to. I would strongly encourage you to get involved with the Wiki and help us figure out how to bring more exposure to this topic.

» Tue Aug 25, 2009 10:38 pm

This is most useful when working with a team like on a huge worldspace...Has there is a master file that is put into place and you can prevent formid from getting overwritten by other team members working on same worldspace...

I have just not had the time to play around with this once again..to see what potentioial this has..but from my understanding this is how bethesda built the game using Version Control...for each region...and worldspace...

Corepc

» Wed Aug 26, 2009 7:53 am

Anyone have any ideas as to why my master file doesn't change? I followed the steps above to the letter, but everytime I check in some forms my master file remains exactly the same (except the date modified). I start out loading my master file, which is dependent on Oblivion.esm, and then make some changes to a world space (unique to my master file), then I check in my world space. Is there something I am missing? I know I have the directories set up correctly... and the FID and FUD files generated... please, if you have any ideas I must get this merging working.

Thanks!

» Tue Aug 25, 2009 10:52 pm

Odd that this popped up in a search for "unlock", but

[bump]

» Wed Aug 26, 2009 4:24 am

[bump] from the jaws of death

» Tue Aug 25, 2009 10:57 pm

The first step (editing the ini file) seems to have a logical bug. The author assumes that the share C$ is enabled. This is only true on windows 2000. Windows XP and Vista don't have this option enabled. Home versions of these operating systems might not work at all since their networking capabilities are limited.
You need to DISABLE "simple file sharing" under tools -> folder option (available in any windows explorer window. e.g. my computer). If you have a limited version of windows e.g. "Home version" this option is not available. In this case you need to enable sharing of drive C (right click on the drive) giving it the share name C$ and giving full write permissions (Note that this is a major security hole and you need to take steps to disable file and printer sharing with a firewall. Windows firewall may be sufficient to do so). If your username doesn't work then you can try using "Guest" as username. A reboot isn't necessary but you can try it if the changes don't register.
The above procedure assumes you have installed the game in your C: drive, If not you need to replace it with the drive letter you installed the game e.g. D$, E$ and of course enable sharing of that drive instead of drive C:
I haven't tested this yet but it is a logical assumption that you need to follow the above procedure for version control to work. I don't use Vista. I tried to be as simple and foolproof as possible.

» Tue Aug 25, 2009 10:42 pm

Whew ! Great job on the tutorial :goodjob: This really needs to be on the wiki.

» Wed Aug 26, 2009 11:25 am

Deserves another bump.

» Wed Aug 26, 2009 3:17 am

I mainly work on scripted stuff and what I most want is branch, merge, and diff of scripts. It sounds like this is not supported at all in the CS version control stuff. Its unclear from the description that it supports any real sort of version control at all and is really just a backup of last save and formID locking system. How do you restore back to a version earlier than just the last save?

So I'm back to cut-n-pasting of scripts back and forth from the CS to get them in/out of mercurial... unless someone else has a better idea?

One idea I have is to use something like 'strings' to pull the script text out of the *.esp files. It'll probably take a bit of fiddling to get it so it can extract individual scripts into separate files, and I would still need to paste it back into the CS after doing things like merge, but at least I should be able to automate checking in versions from the *.esp file. When I get something working I'll followup back here.

» Tue Aug 25, 2009 10:45 pm

So I had a quick dig inside an esp and the following Python code fragment will extract the named scripts from a *.esp so you can check them in to a SCM like mercurial.

#!/usr/bin/pythonimport reimport sysscn_re = re.compile(r'\x00SCTX..(.+?)SLSD\x18', re.S)name_re = re.compile(r'(?:scriptname|scn)\s+(\S+)', re.I)def getScripts(data):  """Gets a list of scripts from an esp/esm file."""  return scn_re.findall(data)def getScriptName(data, index=0):  """Gets the name of a script"""  return name_re.search(data).group(1)def main(argv):  data = http://forums.bethsoft.com/index.php?/topic/793590-version-control-a-primer/open(argv[1]).read()  for script in getScripts(data):	name ='%s.scn' % getScriptName(script)	print 'Writing %s' % name	open(name, 'wb').write(script)if __name__ == '__main__':  main(sys.argv)

This will only work for named scripts and not result scripts. I'm not sure if result scripts will just be skipped or will make it crash. It's just a quick and dirty solution for my needs right now.

I also had a look inside Wrye Bash code and it looks like it would not be too hard to use/extend it to have cmdline diff and patch, and even three way merge of *.esp and *.esm files. It would even be possible to do 3 way merges of scripts inside mods, but you would have to open and recompile the scripts inside the CS afterwards. This would mean you could do proper branch/merge of mods using distributed SCM's like mercurial or git. This would make development of large mods by teams much more manageable.

Is anyone else interested in this? It's a slightly larger project than I originally intended to tackle, and probably more than I personally need right now, but if there was lots of interest I'd give it a go.

» Tue Aug 25, 2009 9:19 pm

So it looks like no interest... either because no one really wants it, or no-one quite understands what this means :-)

What I'm proposing allow a cygwin session like this;

$ # get a textual diff of two mods$ moddiff old.esp new.esp---- old.esp	2009-07-28 17:52:33.031377000 +1000++++ new.esp	2009-07-28 17:45:24.887444000 +1000@@@ GMST @@@- fFatigueReturnBase = 2.0- fFatigueRunBase = 10.0+ fFatigueRunBase = 2.0+ fFatigueRunMult = 5.7@@@ aaRFQuestScript @@@@@ -1,7 +1,6 @@ scn aaRFQuestScript short foo-short bar+float bar float baz begin@@ -23,6 +22,3 @@   set foo to 1.0-  set bar to  10+  set bar to 10.5   set baz to 13.5$$ # get a binary patch.esp of differences between two mods$ moddiff -b old.esp new.esp > patch.esp$ # apply differences in a patch.esp to a mod$ modpatch base.esp patch.esp >newbase.esp$$ # Note mercurial is configured in ~/.hgrc to use moddiff and modmerge as the diff and merge tools$ # add and checkin the base version of a mod$ hg addA mymod.esp$ hg commit$ # create an experiment branch$ hg branch experiment$ # now edit mymod.esp in the CS$ hg diff---- a/mymod.esp	2009-07-28 17:52:33.031377000 +1000++++ b/mymod.esp	2009-07-28 17:45:24.887444000 +1000@@@ GMST @@@+ fFatigueRunMult = 5.7- fFatigueReturnMult = 0.06+ fFatigueReturnMult = 0.08$ # now submit it and change back to the default branch$ hg commit$ hg update -C default$ # now edit mymod.esp in the CS again and check and submit the changes$ hg diff---- a/mymod.esp	2009-07-28 17:52:33.031377000 +1000++++ b/mymod.esp	2009-07-28 17:45:24.887444000 +1000@@@ GMST @@@+ fFatigueRunBase = 2.0+ fFatigueRunMult = 4.0- fFatigueReturnBase = 2.0$ hg commit$# now merge in the changes from the experimental branch$ hg merge -r experimentmerging mymod.espfFatigueRunMult is modified in both local and other versions;base:undefined local: 4.0 other: 5.7Select which one to use (L)ocal (O)ther (B)ase (default: Local): L$ hg diff---- a/mymod.esp	2009-07-28 17:52:33.031377000 +1000++++ b/mymod.esp	2009-07-28 17:45:24.887444000 +1000@@@ GMST @@@- fFatigueReturnMult = 0.06+ fFatigueReturnMult = 0.08$ hg commit$ # now look at the combined diff since the original submit$ hg diff -r 0---- a/mymod.esp	2009-07-28 17:52:33.031377000 +1000++++ b/mymod.esp	2009-07-28 17:45:24.887444000 +1000@@@ GMST @@@+ fFatigueRunBase = 2.0+ fFatigueRunMult = 4.0- fFatigueReturnBase = 2.0- fFatigueReturnMult = 0.06+ fFatigueReturnMult = 0.08

This example is pretty simple, in that it only shows local branch and patch stuff and only changes involving game settings, but in theory these changes can be passed around and pulled between developer's distributed repos like normal mercurial changesets, and any type of record inside the *.esp's can be correctly 3-way-merged between different developers branches. The three way merge handling of conflicts in changes to records can always be simply handled via the interactive decision approach, but for things like leveled lists or scripts more advanced merges can be done using wrye bash or text merges. A text merge would also prompt you to recompile the script in the CS after the merge.

» Wed Aug 26, 2009 6:47 am

So it looks like no interest... either because no one really wants it, or no-one quite understands what this means :-)

What I'm proposing allow a cygwin session like this;
SNIP
This example is pretty simple, in that it only shows local branch and patch stuff and only changes involving game settings, but in theory these changes can be passed around and pulled between developer's distributed repos like normal mercurial changesets, and any type of record inside the *.esp's can be correctly 3-way-merged between different developers branches. The three way merge handling of conflicts in changes to records can always be simply handled via the interactive decision approach, but for things like leveled lists or scripts more advanced merges can be done using wrye bash or text merges. A text merge would also prompt you to recompile the script in the CS after the merge.

no interest from me because I didn't yet see it.
I think it sounds useful for large projects such as provincial building ones.
a nice trick for recompiling those scripts is
-> merge
->load in cs
->open script window
->hit recompile all (yes somewhat bad form but fixed...)
-> save
-> open Wrye Bash
-> Remove Compile All (all removes the recompile to the unchanged vanilla scripts)
-- awesomely quick compared to hand recompiling each script if you have more than a few.
Pacific Morrwoind

» Wed Aug 26, 2009 9:59 am

Rocket, do you mind if I copy this into the TESNexus article database? If you have an ID there, I can re-assign it to your ID so you have full control over it...plus, it won't get auto-purged like topics on this forum.

Thanks,
LHammonds

» Tue Aug 25, 2009 10:02 pm

So it looks like no interest... either because no one really wants it, or no-one quite understands what this means :-)

What I'm proposing allow a cygwin session like this;

$ # get a textual diff of two mods$ moddiff old.esp new.esp

This example is pretty simple, in that it only shows local branch and patch stuff and only changes involving game settings, but in theory these changes can be passed around and pulled between developer's distributed repos like normal mercurial changesets, and any type of record inside the *.esp's can be correctly 3-way-merged between different developers branches. The three way merge handling of conflicts in changes to records can always be simply handled via the interactive decision approach, but for things like leveled lists or scripts more advanced merges can be done using wrye bash or text merges. A text merge would also prompt you to recompile the script in the CS after the merge.

This sounds very interesting to me at least. It seems like it could be very useful for large team projects.

I'd say please do take a shot at it.

» Wed Aug 26, 2009 3:45 am

Has anyone else been able to successfully build up worldspace landscapes using the Construction Set's version control? I ask because while messing around with the version control of the GECK I had some decent success with it. I had been able to compile a blank 36 quad worldspace (using a heightmap derived from an area of Mars), into an ESM before. And unlike my manual efforts to compile such a big worldspace together (by stitching separated worldspace quads together in another tool), it actually compiled the file at the right file size as well (instead of bloating 10x times like it was doing when stitched manually for me).

» Wed Aug 26, 2009 1:38 am

Rocket, I received your PM and have created a mirror article at TESNexus which will be safe from "archival"

http://www.tesnexus.com/articles/article.php?id=348

Let me know what your TESNexus account is called and I will transfer ownership of this article over to you.

Thanks,
LHammonds

» Wed Aug 26, 2009 7:49 am

Hi, Rocket. Thanks for tutorial.

I haven't anything to add to this topic though. But I must say that we've tried to setup CS for modding by the network. Unfortunately we've encountered with some troubles...