----------------------------------
http://www.gamesas.com/bgsforums/index.php?showtopic=814180
----------------------------------
TES4Gecko - Version 15
(Current stable release: Version 15.2)

Created by TeamGecko
- New development and Original TES4 Plugin Utility code: http://www.elderscrolls.com/forums/index.php?showuser=21913 (Ron Hoffman)
- Original Gecko plugin splitter and compare code: http://www.elderscrolls.com/forums/index.php?showuser=320493 (N David Brown)
- Random Guy with Code Access: http://www.elderscrolls.com/forums/index.php?showuser=288454 (Yeah, like that's a pseudonym!)
- Gecko project leader: http://www.elderscrolls.com/forums/index.php?showuser=59412 (Aubrey K McAuley)

Overview

TES4Gecko works with The Elder Scrolls IV: Oblivion plugin and master files (*.esp/*.esm). It is designed to let mod-creators manipulate plugin and master files in ways that Bethesda either did not imagine or decided to leave out of the official Construction Set.

TES4Gecko has a wide variety of useful functions, such as merging two or more plugins into a single plugin or removing data that doesn't need to be in a plugin (accidental changes). It can convert a plugin file to a master file or a master file to a plugin file, move worldspaces, generate silent dialog MP3s, and a ton of other useful stuff. Selected plugin records can be removed from a plugin or copied to another plugin. It will also split a single plugin file into master/plugin components and compare two plugins, highlighting the differences between them.

An incomplete README file for version 15 is currently available. For information in greater depth, check out the http://www.silgrad.com/wbb2/thread.php?threadid=10388 for development history and valuable hints.

New Features and Changes

Version 15.2 adds or fixes the following:

• Added quest reference (QSTI) and landscape texture (LTEX) search.
  
• Added ability to dump all dialogue for a plugin.
  
• Slightly better memory management, which should allow for using the same instance of Gecko more times before having to restart.
  
• Added factions to lip-synch plugin generation; I'll post a HOWTO after I release this one.
  
• Now silence file gen algorithm is 3 words per second, not 4. Also added additional voice files for 16- through 20-second lengths.
  
• Added the ability to set the starting form ID for the merge.
  
• Added the option to "manage" the merge to master. If merge is not managed, default behavior as in v14 and v15.0 takes place.
  
• Fixed a bug that http://www.gamesas.com/bgsforums/index.php?showuser=459728 found that would throw an exception in managed merging if an exterior cell is in more than 5 regions.
  
• Fixed an error in cleaning reported by http://www.gamesas.com/bgsforums/index.php?showuser=119549.
  
• Added more subrecord types for viewing.
  
• Finally (I think) fixed the merging of deleted INFOs.
  
• Added a "remove extraneous quest references" feature for porting dialogue quests to different masters.

Version 15.1 adds or revamps the following:

• Now the music type of all exterior cells can be changed to one common type (Default, Public or Dungeon). Apparently StreamMusic operates more consistently if the music type for the cell is Public.
  
• Thanks to the observations of http://www.gamesas.com/bgsforums/index.php?showuser=116004, the generation of silence files and the production of the LIP ESPs have been revamped. Now all plugins in the master list are loaded to determine if there are races or NPCs that have dialogue added or modified in the current plugin. For the LIP ESP generation, NPCs used in the dialogue have been added to the ESP and also conditions relating to NPC or race are now kept, which results in far less editing of conditions in the LIP ESP.
  
• Also at bg2408's suggestion, generic LIP files have been added to the silence file generation so that lips are moving as the dialogue captions are seen.

Version 15 came about mostly because Steve wanted TES4Gecko to do stuff that it didn't do and that nobody else wanted. Since Steve on occasion gets paid to write Java code, he asked if he could try his hand at it. ScripterRon and dev_akm graciously gave both their consent and the then-current code base. Since then Steve has been beavering away in the decrepit and ill-lit back rooms of http://oblivion.silgrad.com/, working with the advice, requests and encouragement of the rest of the Silgrad Core, in particular sandor, our Master of the Master File. Steve started out with his list of wants, which were mostly related to quest cloning and working with dialogue. As time went on, more requests were received relating to the maintenance of a very large landmass mod as the "stone-souping" of this version continued. Now we feel it's ready to unleash upon the world. Here is an incomplete list of features added in this version:

• The ability to generate reports on objects modified within a certain plugin that are defined in master files to detect contamination.
  
• The ability to dump quest or NPC dialogue into a text file, or to modify that text file and put the result back into the plugin.
  
• The cloning of dialogue quests by selecting a quest and associated topics, copying them to the clipboard, renumbering the form IDs and renaming the quest.
  
• The ability to copy quests and dialogues as shown above, and then "boiling down" the clipboard plugin to the minimal size needed for loading in CS1.0 for LIP generation.
  
• Now the clipboard can be saved to a new file and then cleared to allow for further copying and saving. Also saving is now disabled unless something has actually changed.
  
• Generate reports for a plugin of objects introduced by that plugin.
  
• Searching has been greatly extended. Searches can now be done of form ID, editor ID, XY location for cells, base reference ID, script text and response text among others. String searches can use the asterisk as it is used in Windows, e.g. *bat* will match "bat", "abattoir" and "Black Sabbath's Greatest Singer: Ozzy or Dio?"
  
• Generation of silence files has been revamped. Rather than one 10-sec silence files, there are now 15 more, from 1- to 15-sec with a basic word count done for each response to roughly guess the correct length; no more waiting for ten seconds after Hello.
  
• References with a common base ID can now be removed or changed en masse at the cell, region, world space or plugin level.
  
• Now when merging, there is the option to include or exclude sets of regions; very useful with landscaping claims.
  
• Displaying subrecords as their actual type.
  
• Fixed the VWD merging bug (missing interiors).
  
• Fixed the cell deletion bug.
  
• Fixed the deleted topic merging bug.

Incorporated into this release is an extensive right-click system on the tree nodes; this is where most of the new functionality resides. Although thoroughly tested, all users should still back up their plugins while working with this version. Public release has a way of uncovering what even the best beta testers miss.

Hot on the heels of the last release, Version 14 mostly adds several important worldspace fixes. It now adds the exterior cell X,Y coordinates to the tree node description, assigns interior cell block/subblock based on the merged form ID for the cell, and changes the Split Plugins function to move visible-when-distant groups to the plugin.

TES4Gecko Version 13 added a ton of new features, drastically revamped the code base, and fixed a bunch of bugs. Here's a few of the biggest new features.

The Compare Plugins function has been redesigned to improve performance. It now allows comparisons of very large files, such as Oblivion.esm, which in turn made it possible to add a new "Clean Plugin" function to remove any entries from a plugin that do not actually differ from Oblivion.esm (accidental changes). The 'Toggle Ignore' and 'Copy to Clipboard' functions are now available on the Compare Plugins screen. The "Split Plugin" function now has an option to create an independent ESM/ESP pair (where the ESM has no dependency on the ESP). The Display/Copy function now displays the plugin master list with the associated modindex for records tied to that master, supports both FormID and EditorID search functions, and provides automatic highlighting of master records. The master list order can now be changed, Worldspaces can be moved into the Oblivion.esm modindex (00) to avoid vanishing landscape problems, and BOOK subrecords are merged.

See the version history near the end of the README for a complete list of changes and fixes prior to version 15.

Installation

To install this utility, place the TES4Gecko.jar and all of the TES4Gecko silence MP3 & LIP files into a directory of your choice. To run the utility, create a program shortcut and specify:
as the program to run, where should be replaced with the directory where you extracted the jar file. If the path for java isn't set on your system, you will need to specify the full path to javaw. If install-directory includes any spaces, the quotes are required; if you add a path containing spaces to javaw, quotes are required around the entire path name (including javaw). That entire line is case-sensitive, even for Windows. Installation troubles have been reported with Vista, probably due to the extra security required to access anything in the Program Files directory.

A sample program shortcut is included that specifies \Tmp as the install directory. The -Xmx1024m argument specifies the maximum heap size in megabytes (the example specifies a heap of 1024Mb, or 1Gb). You can increase the size if you run out of space merging very large plugins. Note that Windows will start swapping if the Java heap size exceeds the amount of available storage and this will significantly impact performance.

The http://java.com/download/index.jsp is required. If you are unsure what version of Java is installed on your system, open a command prompt window and enter "java -version".

Credits

For Version 15, Steve would like to thank:

• ScripterRon and dev_akm for the use of their code and the answering of my stupid questions,
  
• The guys at http://www.silgrad.com/wbb2/portal.php, especially sandor. ST is a pretty demanding mod to work with and the guys there were invaluable with their suggestions, encouragement and patience while testing.

TES4Gecko Version 13 represents ScripterRon's return to working on TES4Gecko project, and the results are dramatic. TES4Gecko Version 12 was created using the source code from ScripterRon's wonderful TES4 Plugin Utility (Version 11.1), which he graciously donated to the project. The vast majority of the functionality here was created by ScripterRon -- TeamGecko merely seeks to expand and enhance his amazing work.

Huge thanks to Lightwave for helping us figure out how to deal with vanishing landscape problems in new worldspaces.

Download

http://www.tesnexus.com/downloads/file.php?id=8665
----------------------------------

Fantastic, work, man! Thanks so much for updating this great tool! :tops:

Thanks Steve

I can find these option useful for npcwithjobs worldspace, I had a problem of missing landscape whenever I used moveworldspace's with gecko 14, will have to try out the new version to see if this fixes the problem or not.

A mild warning: our VWD bug was with merging interiors with VWD objects e.g. trees. I did not really look much at moving WSs at all. Having said that, once a reproducible issue is discovered, I'll be all over it.

Steve

Thank you! Most welcome this update. Most kind of you to extend the utility of this utility.

Wow, very very thank you.

[edit]I just saw I can now view the content of the records with Display Subrecord. Previously it was just some strange giberish. :goodjob:[/edit]

very cool stuff :goodjob: thank you for the excellent work .

Thanks for the update, much appreciated!

While trying to get this to work tonight, I kept getting "Could not create Virtual machine" and this was even with a complete fresh install of Java and of Gecko. After a few minutes of searching on google, I ran across a post that said to remove "-Xmx1024m" from the command line. This was for a completely different app. However, at wits end, I decided to try it with my Gecko shortcut. Worked like a charm. You may want to consider adding this tip to your readme.

I use Windows XP Pro SP3

Great, a new version of Gecko! Thanks!

Perhaps there was some other problem with your shortcut?

Unless you have very limited or fragmented RAM, the -Xmx switch should not cause any problems. It lets you process larger plugin files.

yay! :foodndrink:

thanks for all the hard work guys, this utility gives my quality of life +2

No, shortcut was exactly as specified in the documentation, pointing correctly to where my custom install locations are. I also run 1.5Ghz ram and it was not fragmented. After doing some further research I found my Xmx limit is 930.

Something you may want to add to the instructions is this:

If you receive the error that you cannot create the virtual machine in Windows XP, go to start, run, then enter cmd and click ok.

Type in "java -Xmx1024m -version"

If successful you will see something like this:

java version "1.6.0_07"
Java™ SE Runtime Environment (build 1.6.0_07-b06)
Java HotSpot™ Client VM (build 10.0-b23, mixed mode)

If unsuccessful, you will see something like this:

Error occurred during initialization of VM
Could not reserve enough space for object heap
Could not create the Java virtual machine.

Lower the Xmx value by 10 each time until you successfully get the version number and adjust your shortcut accordingly.

I changed mine to 896 purely for the reason that I prefer to work like ram does. 768 + 128 = 896 whereas 930 is not divisible by a standard ram count of 64 or more.

I've got a "small"* problem with the silent voice generate option. See, I'd like to create them for a mod which has three masters:
00 - Oblivion.esm
01 - bgBalancingEVCore.esp (Race Balancing Projekt)
02 - bgMagicEV.esp (LAME)
Now whenever I select the mod, the function results in an error message telling me http://www.tikanderoga.net/bg2408/pics/tes/tes4geckosilentvoiceerror.jpg. The mentioned FormID doesn't exist in none of the master files, neither it is referenced in the mod for which I'm trying to create the silent voices.

For testing I've tried another mod, with no new dialogue, but also depending on both RBP and LAME. Same error message.

... and then I've tried FCOM_Convergence.esp, which also depends on more than two master files. Same error, this time though 01000FC8 was missing. Then I tested the next plugin, and... let's just say whenever I tried a plugin with three masters, it resulted in this error, only with a different FormID each time, but it was always a FormID starting with 01.

Do I have any chance to get that function working for me? If not, is there another program to create silent voices?

Note: Creating voices only for one of the master files each (exclusing Oblivion.esm of course) works, even if doesn't do anything, it doesn't prompt this error message .

(As a small suggestion: How about a "standard" .lip file copied for each silent wav?)

* "small" = if I can't find any way to create silent voices, be it with gecko or another program, I'll stop developing any mods having new dialogue immediately. Really, dialogue in Oblivion is like hell on harddrive. Sorry for the rant, but what the *!/%&Q?!* was Bethesda thinking? Nothing against you guys, without gecko I would have lost my mind long ago, but dialogue is driving me crazy, CRAZY :wacko:!

There is the option of Elys' http://www.gamesas.com/bgsforums/index.php?showtopic=832514.

Of course, it may not be suitable for your particular mod. If it doesn't otherwise use OBSE, you may not want to add that dependency. Even if it does use OBSE, you might not want to add another dependency

I must say I'm confused; I thought that the concept of ESPs depending on other ESPs was tricky and not recommended. I'll be happy to look at your ESP if you send it along, but one issue is that TES4Gecko treats (and always has treated, I believe) a plugin as a unit. Any dependencies are expected to be in the master list, and there is no provision for unenforceable constraints like, "Always have AAA.ESP loaded before BBB.ESP." If there is a more or less agreed-upon method in handling this, I can try to implement it in Gecko. BTW the error that you received is probably an unrecognized race from one of the ESPs. I would appreciate it if you can send me the ESP; even if I cannot make it work, I need to make it so that it doesn't blow up.

Steve

ESP mastering isn't recommended for the beginning modder simply because it requires using programs other than the CS to implement.

For modders used to using additional tools, it is pretty easy to implement. I've never had any problems creating my own esp mastered plugins, and they've always behaved in-game as expected.

For such an ESP-mastered ESP, is there any data within the ESP itself that indicates this or is that info maintained in an exterior plugin-management program?

Thanks,
Steve

I may be one of the first outside of the FCOM crowd that have used mod deisolation (back in the beginning of '07), but it's "in the use" and working flawlessly now for one and a half years .


It doesn't only happen with my .esp for which I need it, that's the odd thing. I've tested it with just about every file of me having more than two masters and any race entry, the same error occured always, and always the FormID in question did not exist - and if it did, it was applied wrong (e.g. for my plugin 01000907 does not exist, while 00000907 are Oblivion.esm Imperials, for the FCOM example 01000FC8 is a Welkynd Stone reference in OOO.esm, but a Dremora without voice in Oblivion.esm [00000FC8]).

My guess is there's some hiccup going on with FormIds :blink:.

The FormID does not exist in neither plugin. Also only one of them even has new races. As long as a plugin is only depening on it (which means two masters, Oblivion.esm + plugin-master + dependencee) it works. If another plugin is added, even if that one doesn't include any race record, it goes *poof*


Gladly, uploading them as soon as I'm home again! Thanks for helping !


It's simply the master list.

The problem with .esp masters is that the Construction Set doesn't allow .esp-mastered .esps, therefore for .esp mastered .esps the "master esps" are turned into esm files ("Esmify" function in bash) when used as masters in the construction set. Oblivion itself doesn't have problems with them, only the CS - if not esmified before

Yep :nod:
Wrye or ElminsterEU would be better qualified to answer any questions about any differences in processing, but as far as I know, you can treat the esp in the master list exactly the same as you would treat an esm. The CS simply refuses to put esps into the master list, but the game engine doesn't care.

Yes and no.

There's a secret "master" bit in the mod file that's not well documented. You can set this bit without changing the file extension and TESCS will recognize the file as being a master. This is how the Wrye Bash Esmify function works. See http://www.uesp.net/wiki/Tes4Mod:Tool_Features#Modding

I'm sure Wrye can give you more details if you need it. You can also probably find it by looking at the Convert to Master code.

However, this trick is only necessary for TESCS. The game engine itself doesn't care at all. Any mod (plugin or master) listed as a master by another mod will be considered a master (for override purposes). This means any plugin can safely change any other plugin. No additional tools or tricks are necessary. The capability has been extensively tested for more than a year now and proven to be completely stable.

The only exception is Delete (you can't override a record from an ESP with a Delete in another plugin), but Deletes should be avoided anyway since they can cause in-game problems with de-allocating memory when reloading a savegame.

EDIT: for clarity.

Any news regarding my silent voice creation problem ? Did the PM with the test files arrive?

I think I found da issue, but I need another test case with dialogue.

bg2408: I sent a PM, but I'm not sure how well PM notification works around here.

Steve

Version 15.1 released

Version 15.1 adds or revamps the following:

• Now the music type of all exterior cells can be changed to one common type (Default, Public or Dungeon). Apparently StreamMusic operates more consistently if the music type for the cell is Public.
  
• Thanks to the observations of bg2408, the generation of silence files and the production of the LIP ESPs have been revamped. Now all plugins in the master list are loaded to determine if there are races or NPCs that have dialogue added or modified in the current plugin. For the LIP ESP generation, NPCs used in the dialogue have been added to the ESP and also conditions relating to NPC or race are now kept, which results in far less editing of conditions in the LIP ESP.
  
• Also at bg2408's suggestion, generic LIP files have been added to the silence file generation so that lips are moving as the dialogue captions are seen.

I will shortly post brief HOWTOs on dialogue extraction and LIP ESP generation to demonstrate why if you deal with a lot of voice acting in your mod, you'll really want this version.

Steve

Generating a LIP ESP with version 15.1

What this functionality allows for is essentially a more elaborate form of Kivan's method of LIP generation. With this you can take a plugin, whether SI-based or not, and "distill" it into a minimal ESP that doesn't depend on any other plugins and can be loaded into the version of the CS that allows for LIP file generation. The steps are:

• Copy the entire groups of NPCs, races and dialogue to the Gecko clipboard.
  
• Then go into the quest group and copy only the quests for which the dialogue is required. For smaller plugins, it may be possible to copy all the available quests; if you are dealing with a plugin on the scale of ST or TR, you're better off selecting only a few quests. Quests are the filtering mechanism in this process; which lines are kept and which NPCs are used are determined by the quests.
  
• Right-click on the clipboard side at the plugin level and choose Prepare for Lip Synch. You'll see a large popup with dire warnings. They're all true, but are largely irrelevant since this plugin cannot be saved over an existing one (except for Gecko Clipboard.esp).
  
• Then hit Save Clipboard to New and pick a name. I use the old plugin name with LIP appended.

What did the preceding actually do?

• It went through all the plugins in the master list and gathered all the races and NPCs defined in those plugins into the clipboard.
  
• The clipboard is then purged of all object groups except for DIAL, NPC_, RACE & QUST.
  
• The DIAL objects are cleared of all QUST references except for those quests remaining in the QUST group; all INFOs that do not refer to those quests are purged.
  
• All QUST & INFO conditions that are not GetIsId ,GetIssix or GetIsRace are culled.
  
• All NPCs not mentioned in the GetIsId conditions in the previous step are also culled.
  
• Finally all remaining records have all subrecords removed except for the minimum required for voice file processing.

To reiterate, this ESP is unusable for either playing or modding.

Then how is this used?

Fire up the 1.0 version of the CS and choose this file; it should come up very quickly. I have occasionally gotten a warning where a race ID has collided with an INFO ID, but hitting Yes and continuing harms nothing. At this point, you can choose either a quest or the Filtered Dialogue option to bring up the lines. The race and ID conditions have been saved and almost always those are the conditions determining which LIP files need to be generated; however it may be occasionally necessary to add additional conditions to filter the list of race/gender conditions for a given line.
At this point, make sure your voice files are in the proper directories, bearing in mind that the plugin itself has a different name, and generate the LIP files normally. Once generated, simply transfer those files to the directory tree under Date\sound\voice\ and you're in business.

This ESP is also useful for extracting dialogue for the voice actors in your project. I'll cover how to do that later.

Hope this helps,
Steve