[WIP] BAPI - BOSS API

Post » Fri Sep 09, 2011 1:29 pm

BAPI: BOSS API

BOSS API

This is a discussion thread for all interested parties to hack out what they need and want. The idea behind BAPI is that the BOSS team (or rather, I) supply the code needed to parse the BOSS files or perform some BOSS-related functionality in a DLL which other utility makers than then load and call functions from as they require, rather than having to write out their own parsers, etc.

This will make keeping up-to-date with BOSS functionality easier, and will ensure that whatever the range of utilities available that reference BOSS information or functionality, they all have access to the same level and form of information.

I have created an issue for this on the Google Code page too: http://code.google.com/p/better-oblivion-sorting-software/issues/detail?id=111. The Google Code comment system is a bit rubbish though, so it's probably best to keep discussion here for now.

I can provide some information up-front already regarding the output encoding: BAPI output will be encoded in UTF-8. If you want to use it in another encoding, you're going to have to convert from that.

So far, Lojack has given me the following request:
// BOSS API// NOTES:// - This is just what I could tell that Wrye Bash needs to use for it's information.//   I'm sure more things may be useful to expose (the actual ordering stuff, etc).// - Need to think about unicode.  Best solution might be the 'Windows API way', with//   for example a 'ParseMasterlistA' and 'ParseMasterlistW' function both exported.//   I'm thinking encoding for the unicode versions would be UTF-8?  Or would UTF-16//   be better?  Encoding on the 'tags' variables can always be ASCII, the variables//   I see that would probably need to be unicode are://    ParseMasterlist   - masterlistPath//    ParseUserlist     - userlistPath//    GetBashTags       - modName//    GetBashRemoveTags - modName//    GetDirtyMessage   - message/* Initializing functions.  Parse the userlist and masterlist for data */bool ParseMasterlist(const char *masterlistPath=NULL);// masterlistPath: path to the masterlist.  If NULL, try the cwd.// return true if parsing successful, false otherwise.  Possibly use// more return codes to pass error information?// A second call to this would reparse the masterlist, updating with// new information.  An error on a reparse would not clear out old// masterlist information.bool ParseUserlist(const char *userlistPath=NULL);// userlistPath: path to userlist.  If NULL, try the cwd.// return true if parsing successful, false otherwise.  Possibly use// more return codes to pass error information?// A second call to this would reparse the userlist, updating with// new information.  An error on a reparse will clear out old// userlist information./* Get information for a specific mod */int GetBashTags(const char *modName, char **tags, unsigned int maxTags, unsigned int bufferSize);// modName: name of the modfile to get information for (not case sensitive)// tags: list of strings, allocated by the caller to hold resulting tags.// maxTags: max number of strings that 'tags' can hold.// bufferSize: max length (including NULL) of each string in 'tags'.// return: <0: indicates an error, otherwise, indicates the actual number of tags filled into 'tags'//// There might be a better way to pass this information around?int GetBashRemoveTags(const char *modName, char **tags, unsigned int maxTags, unsigned int bufferSize);// modName: name of the modfile to get information for (not case sensitive)// tags: list of strings, allocated by the caller to hold resulting tags.// maxTags: max number of strings that 'tags' can hold.// bufferSize: max length (including NULL) of each string in 'tags'.// return: <0: indicates an error, otherwise, indicates the actual number of tags filled into 'tags'bool GetDirtyMessage(long crc, char *message, unsigned int bufferSize);// crc: CRC of the mod to get the dirty message for// message: buffer to hold the message, caller allocated// bufferSize: max length of 'message'// return: true if there was a message for that crc, false if not.

User avatar
Lucky Boy
 
Posts: 3378
Joined: Wed Jun 06, 2007 6:26 pm

Post » Fri Sep 09, 2011 7:08 am

Thanks wrinklyninja for starting this up :celebration:

I know I've never really done API programming, so my idea of what an API should look like is probably way off. I know what information Wrye Bash needs, I just don't know the best way to pass that information back and forth.

The first two proposed functions up there are just initialization stuff, which I think would be common to all applications of the API. So there might be a better way to expose that *shrug*

Keep in mind, you could use this for loads of stuff. BUM could tie into it for masterlist/userlist parsing for example. The more of a full API we have fleshed out now, the less changes would have to be made and less functions exported in the future.
User avatar
SamanthaLove
 
Posts: 3565
Joined: Mon Dec 11, 2006 3:54 am

Post » Fri Sep 09, 2011 6:23 pm

Double post, but new thoughts:

I'm not sure how you want to do versioning. Would you be exporting a 'GetBossMajorVersion', 'GetBossMinorVersion', or would you just be using the file version info for versioning?

Also, what would be the general rule of thumb for when a version increment happens? What makes sense to me would be something like:
- Increment in "MajorVersion" means it's not backwards compatible with previous versions. For example, functions were renamed or moved, or paramenters or return types were changed
- Increment in "MinorVersion" means it's still backwards compatible with, but maybe there are new functions available, or maybe just internal improvements or something.

Also, at least for Wrye Bash, I think the plan would be:
- look for the dll in the BOSS directory. Use that as long as the version numbers tell us it's compatible.
- If the dll in the BOSS directory is incompatible, or there is no dll there, use a copy of the dll distributed with Wrye Bash. Assuming this is ok with whichever license BOSS uses.


myk002 also had some ideas on the actual API, seemed better than what I had come up with.
User avatar
x a million...
 
Posts: 3464
Joined: Tue Jun 13, 2006 2:59 pm

Post » Fri Sep 09, 2011 1:25 pm

I figured BOSS already has all the memory for the data allocated internally, so in general this is a "getter" API instead of a "filler" API. Also, to reduce the overhead of data transfer, we can transmit the bash tags once and thereafter just refer to them by id. This is a C API, as required by Python, though I'd expect it would be implemented with C++ on the backend.

#include #ifdef __cplusplusextern "C"{#endif//////////////////////////////// types// name is encoded in UTF-8 (though in practice it won't be more than ASCII)typedef struct{    uint32_t id;    char * name;} BashTag;// These are returned from the functions that return an // int.  We define error codes instead of error messages so// Wrye Bash can deal with the localizationconst uint32_t ERROR_SUCCESS = 0;const uint32_t ERROR_BAD_ARGUMENT = 1;const uint32_t ERROR_FILE_NOT_FOUND = 2;const uint32_t ERROR_FILE_ACCESS_DENIED = 3;const uint32_t ERROR_FILE_INVALID_SYNTAX = 4;const uint32_t ERROR_NO_MEM = 5;const uint32_t ERROR_INTERNAL = 6;//////////////////////////////// data definition functions// returns an array of BashTags and the number of tags in// the returned arrayvoid GetBashTagMap (BashTag ** map, uint32_t * numTags);////////////////////////////////////// lifecycle management functions// explicitly manage the lifetime of the database.  this// way we can free up the memory when we want/need to, like// when the user starts Oblivion and the system is low on// memory.uint32_t  CreateMasterlistDb  (void ** db);void DestroyMasterlistDb (void * db);/////////////////////////////////////// db loading functions.// can be called multiple times.  if masterlist is loaded,// then userlist is loaded, then masterlist is loaded// again, the previously-loaded userlist should still be// applied over the new masterlist.  path strings are in// UTF-8.  on error, database is expected to be unchanged.// path is case sensitive if the underlying file system is// case sensitiveuint32_t LoadMasterlist (void * db, const char * path);uint32_t LoadUserlist   (void * db, const char * path);/////////////////////////////////////// db access functions// returns an array of tagIds and the number of tags.  if// there are no tags, *tagIds can be NULL// modName is encoded in utf-8 and is not case sensitiveuint32_t GetModBashTags (void * db, const char * modName, uint32_t ** tagIds, uint32_t * numTags);// returns the message associated with a dirty mod and// whether the mod needs cleaning.  if a mod has no// dirty message, *message should be NULL.// modName is encoded in utf-8 and is not case sensitive// message, if not NULL, should be encoded in utf-8// needsCleaning://   0 == no//   1 == yes//   2 == unknownuint32_t GetDirtyMessage (void * db, const char * modName, uint32_t crc, char ** message, uint32_t * needsCleaning);#ifdef __cplusplus}#endif


Depending on how you (wrinklyninja) want to do your data structures, the GetModBashTags version above might be better written as a request for an iterator, and then have iterator access functions. This would prevent you from having to construct integer arrays if you don't have them already, but then we'd have to allocate iterators on the heap and do lifecycle management for them. Classic space vs. time tradeoff, I guess.

Also, from a logistics point of view, I think this is what is needed:
  • a 32-bit dll with a name that ends with "32", such as "boss32.dll" (so we can easily load the correct bit size dll once we offer both 32- and 64-bit)
  • if it is convenient for you to build, a 64-bit dll with the same name, but with "64" instead of "32"
  • dll should be deployed in the Oblivion/BOSS directory along with the BOSS executable (we might want to deploy a copy with Wrye Bash as well, in case the user doesn't have BOSS installed, so we can read our taglist.txt file, but at runtime we'd use the one deployed with BOSS if its there)
  • The DLLs should have their version header populated (so we can figure out which of the DLLs is the more recent version and so we can change which functions we call based on which version it is, if necessary)
  • Alternately, we could use BOSS-specific version retrieval functions (e.g. int GetBossMajorVersion(), int GetBossMinorVersion) if he has to provide them anyway for his own purposes


@Lojack: you bring up a good point with API stability. It could be done with two version specifiers, a BOSS version and an API version. The API version could equal the last BOSS version where the API was changed in a way that would make previous clients break (additions to the API are ok without having to change the API version number as long as the old API is still functional). So if Wrye Bash version X uses a certain set of function calls that are offered with BOSS version Y and API version Z, then it would ensure at runtime that the loaded version of BOSS >= version Y and that its API version == Z. Does that make sense? It might be more flexible just to do it this way:
bool IsCompatible (uint32_t bossVersionMajor, uint32_t bossVersionMinor);uint32_t GetVersionString (char ** bossVersion); // for display

and BOSS could just centralize compatibility information.

edit: specified the bit size of all integers so things stay the same between 32- and 64-bit and made a number of clarifications
User avatar
Dean
 
Posts: 3438
Joined: Fri Jul 27, 2007 4:58 pm

Post » Fri Sep 09, 2011 9:21 am

I've started to tidy up the code in preparation for this. Thanks for all the stuff posted so far, it's no doubt going to be a great help.

Although part of the point of this is to distance third parties from the inner workings of BOSS somewhat, I thought I'd mention that I've started to write up the masterlist syntax documentation in HTML. It needs a bit of formatting work but is content-complete (and more correct than the wiki page for the moment). It can be found http://better-oblivion-sorting-software.googlecode.com/svn/data/boss-common/BOSS%20Masterlist%20Syntax.html (right-click, save as).

I've also got to update the Notepad++ language definition for the newer syntax...

File version: I'll be using the DLL headers, and I suppose that I can provide some functions to provide the version info. I'll be using the major-breaking minor-revision numbering. I'm not sure what sort of breaking changes there would be though, I've always written things to be backwards-compatible (apart from a re-purpose of MF2 incompatibility and requirement messages to be conditional, but they were never used in their original form).

License/legal: I'm the author of the code involved, though copyright is spread through the BOSS team. Derivative works aren't allowed by the license, but there's a waiver clause that I can use to allow the Bash team to distribute their own DLLs.

32/64 bit: I can do both no problem.

Once I've got the parser code tidied up, I'll create a new code tree for the API.
User avatar
ILy- Forver
 
Posts: 3459
Joined: Sun Feb 04, 2007 3:18 am

Post » Fri Sep 09, 2011 3:21 am

I thought of another tidbit that would be useful:
#include // writes a minimal masterlist that only contains mods// that have tags (plus the tags themselves) in order to create the// Wrye Bash taglist.txt file.  outputFile is a UTF-8 string specifying the// file to use for output.  outputFile can be overwritten if it exists iff// overwrite is trueuint32_t DumpTags (void * db, const char * outputFile, bool overwrite);

User avatar
sam smith
 
Posts: 3386
Joined: Sun Aug 05, 2007 3:55 am

Post » Fri Sep 09, 2011 3:40 am

I thought of another tidbit that would be useful:
#include // writes a minimal masterlist that only contains mods// that have tags (plus the tags themselves) in order to create the// Wrye Bash taglist.txt file.  path strings are in UTF-8.uint32_t DumpTags (void * db, const char * path, bool overwrite);


How does the overwrite flag behave? Also, what form do you expect the db to be in, if any? BOSS uses the following internally:

	enum keyType {		NONE,		//Userlist keywords.		ADD,		OVERRIDE,		FOR,		BEFORE,		AFTER,		TOP,		BOTTOM,		APPEND,		REPLACE,		//Masterlist keywords.		SAY,		TAG,		REQ,		INC,		DIRTY,		WARN,		ERR,		//Legacy masterlist keywords.		OOOSAY,		BCSAY	};	enum itemType {		MOD,		BEGINGROUP,		ENDGROUP,		REGEX	};	struct message {		keyType key;		string data;	};	struct item {		itemType type;		fs::path name;		vector messages;	};	vector masterlist;


fs::path is the BOOST filesystem path type, but it can be translated to a string or C-string or whatever.

I've gotten the common functionality library all separated out now, and the CLI BOSS executable done too, so now I've got to adapt the GUI code and add sorting functionality to that, and also start work on the API itself. Using CBash as an example on what DLLs need code-wise. :D

I'm going on holiday for a week starting Saturday though. Although I'm taking my laptop, I doubt I'll get any large amount of work done while I'm away. I'm aiming to get the GUI sorted tomorrow.
User avatar
Chad Holloway
 
Posts: 3388
Joined: Wed Nov 21, 2007 5:21 am

Post » Fri Sep 09, 2011 8:45 am

How does the overwrite flag behave? Also, what form do you expect the db to be in, if any?

fs::path is the BOOST filesystem path type, but it can be translated to a string or C-string or whatever.

I'm going on holiday for a week starting Saturday though. Although I'm taking my laptop, I doubt I'll get any large amount of work done while I'm away. I'm aiming to get the GUI sorted tomorrow.

There's no rush. Have fun on your vacation. We'll have plenty of time during the 296 dev cycle to hash things out.

The overwrite flag is to control whether an existing file will be overwritten (sorry, forgot to document it -- will edit above post). The db format is entirely internal to BOSS -- to users of the API, it's just an inaccessible void * pointer. This is all a C API, so we can't use BOOST classes (or any classes, for that matter) across the calls. Of course, the db pointer might be a class from BOSS's internal point of view, but users of the API don't have to know that.
User avatar
John Moore
 
Posts: 3294
Joined: Sun Jun 10, 2007 8:18 am

Post » Fri Sep 09, 2011 6:13 pm

here's an example usage of the proposed API:
#include "BOSSAPI.h"static void _displayError (uint32_t err) { ... }void makeTagList (const char * masterlistPath, const char * taglistPath){    void * bossDb = NULL;    uint32_t err = CreateMasterlistDb(&db);    if (ERROR_SUCCESS != err)    {	_displayError(err);	return;    }    err = LoadMasterlist(db, masterlistPath);    if (ERROR_SUCCESS != err)    {	_displayError(err);	DestroyMasterlistDb(db);	return;    }    err = DumpTags(db, taglistPath, true);    if (ERROR_SUCCESS != err)    {	_displayError(err);    }    DestroyMasterlistDb(db);}

User avatar
Tanya
 
Posts: 3358
Joined: Fri Feb 16, 2007 6:01 am


Return to IV - Oblivion