just wanted to pop in and say ESsearch has been invaluble for me recently. Thank you for this wonderful tool

You may have to put up a poll for this one Yacoby.

I'm for TESsearch.com. I think that if it sounds similar to tessource.net that its actually a plus, since those who remember that site will have an easier time remembering tessearch.

TESsearch is close to what we've been calling it already (ESsearch), especially if everyone is probably going to continue calling it that anyway. ;P

Is anyone able to give this a quick test:
http://10022011.yacoby.net/

Anything obvious that I have missed?

(The subdomain will be active for a day or so)

I can only see two thing here and first one is that you have forgot the information about the account of Morrowind Modding History, but if you try to download a mod without a account or you're not a logged in user you will see http://img696.imageshack.us/i/scr20746342089873.jpg/ without a login or registration. :user:

It seems that you have missed one thing and that's there two "advlt" links with "nudity" between them after that you have re-run ES Search one time they are stuck even if you refresh it.

But I must say that ES Search is much faster now than previous version and it looks more smooth now, so little more tweaking here and there ES Search would look great. :thumbsup:

And http://code.google.com/p/es-search/updates/list the updates continue! :ahhh: Thanks, Yacoby!

The 'Recent Searches' line is acting kinda wonky:
* single field searches I perform during the session don't seem to propagate the 'Recent Searches' line unless I also navigate to results pages beyond the first results page.
* some single field searches with results never get listed in 'Recent Searches' after navigating back (and refreshing) to the main page; the search param does get listed under 'Recent Searches' on the results page (even when there are no results found).
* sometimes I have to close the page and revisit for it to update; switching between 'Simple' and 'Advanced' sometimes will update after refreshing page.
* there are different recent searches listed for 'Simple' and 'Advanced' modes - shouldn't they both reflect the same, all recent searches?

Would it be possible to add an advanced search option for category (based on the types found in details)?

The issue with the recent searches is that there is a 5 minute cache on some of the pages. That means that if you look at the main page you are probably requesting a cached version where the recent searches are out of date.

:ahhh: I had no idea the confusion it would cause (or that people looked at it. Random fact: it was originally put in so that Google indexed things other than the main page).

Maybe I should reduce the cache time. :shrug:



Ah. Maybe I should talk to Fligg about that.



This is one of the things that I wish I had done before I started indexing.

I can't do this until I get around to rebuilding the index. I could probably write it in now but it would only work with the newer indexed mods.

I don't know when I will rebuild the index as it isn't simple. Well, it is ish, but it may take a bit of time to write the code to do it and then it takes all night to rebuild it.

I have now moved to the newer version and altered the subdomain to search.yacoby.net

The search2.yacoby.net links still work, but they are just automatically redirected to search.yacoby.net

Categories are now stored in the search index, so hopefully at some point it should be possible to search by category.

I have yet to fix the recent search so that it is consistent across all pages.

I've just found that out. Thanks for the update on ES Search Yacoby. :sweat:

While you're fixing that recent search. I would like to suggest that you also add a one/two pull down menus where the user can select which expansion or all expansions should be included in a search. But also add a linked menu where you can select which category you want to search for. Any of these search menus (just above the recent search) would be great if that's possible.

I can't do this as not enough sites and mods have information about the required expansions. While for example PES has the option to set it most mods don't have it set.


This will probably come in some form or another. It would require quite a bit of work as I would need to introduce a set of category aliases. There are currently 156 categories, most of which have a lot of overlap with other categories so it would need to be simplified.

It's good to see you working on this lately!

I think I've mentioned it to you in the past, but I am working on what I hope is a way for you to index GHF's mods despite our unconventional hosting method. We have a posting template that includes some XML tags. I have a script I've been working on that will pull the data from those tags and build a full XML sitemap containing the title, author, version, and description.

So could you use that XML to create a parser? Or is there a specific format that I ought to use? I'll have a link to it for you as soon as I can.


Edit: Hey, I've got another question I just thought of. Have you given any thought to providing any sort of search statistics? Like the ability to see how many times the term "MWSE" or something was searched for in January?

XML is fine.


Could be good. It shouldn't be too hard to do either. I'll add it to a todo list.

I wanted to throw this out there. I have been doing web/application development (mostly programming) for 7 years. I am really good with PHP/Mysql, and Ajax, as well as quite a few other languages. If you ever need a helping hand Yacoby with some of the tools you are building, I would be more than willing to devote some of my personal time to help you get some of the work load done. If you want to do it yourself I understand, but I wanted to throw that offer out there. This is definitely something interesting I wouldn't mind putting some personal time too.

If you would like to help in any way at all, that would be great!

The http://code.google.com/p/es-search/, from which you can look over the code. It isn't exactly the most intuitively laied out.

The code/html relating to displaying the data is in application/. This follows the MVC pattern. The search model uses data from the Search component in the library directory (library/Search/). The library/Search is where the majority of the code relating to parsing etc is.

Due to my (cheap) host not allowing for small (<3 word) search tearms for full text searches, it uses the Zend_Lucene component of the Zend framework for searching. When adding a mod to the database using Doctrine, a listener is triggered which adds it to the index. Adding or removing data from the database (excluding things like the log) using raw SQL or DQL (Doctrine Query Language) is a bad idea.


Let me know if/when you want commit access to the repository.

Looks good. I took a look at most of the code and it looks nicely structured. You said it wasn't laid out well, but I think you did a great job with it.

I have used a variety of Frameworks but I don't use Zend a lot. I mostly use Codeigniter when I am using a framework. I like the way it's setup better.
Also, I like the dom parsing class that you use, but I don't like that it hasn't been updated in a long time. I generally end up using Curl and Regex when I want to do parsing scripts.

This looks really good. So what areas could I help with to start. Any specific bugs/issues/errors that your having that perhaps I can take a crack at, or did you want me to write up a few
parsers that you can hook into it?

I can do a clone of your system on my server at Hostgator and set up a test domain and I can build a parser or two for it for you and then send you the updated code to load up or something.
I need to clear out my schedule some for the first few days this week then I wouldn't mind putting some hours into helping you make this better. Also, the layout itself is kind of basic. I have about
5 graphics designers. You interested in giving this thing a custom design? I could have one of my graphic designers work up some brand new custom designs from scratch and you could do approval on one of
them. Luckily a few of my graphic designers owe me favors, so I can have them do it as a favor. We can get something that "looks" a lot better, a lot more user friendly (as far as design). And either you could
do the xhtml/css for it, or I could do that for you. Let me know what you think.

Either way, point me in the direction where my help will be most useful and I will see what I do. PHP/Mysql are my primary server side language/database platform. I have done them consistently for 7 years, going on
8. I have done a variety of other languages well, but I know PHP the best. So whatever areas you want me to focus on, just point me in the direction and I will start making progress on it for you this week.

I look forward to helping you out with this if your interested.

What about hosting..I see you have this setup at Google Host, but are you using a standard web host for your domain name and files? What host is powering http://search.yacoby.net/. How are you covering server costs.
This site seems like it's a high traffic site, so it might be resource intensive. So how is the host your currently using handle the user load, and bandwidth?

I am never happy with anything I design. I always feel there could be something slightly better.


The issue with regex is it is far harder to put together robust parsers. It does have the advantage of being far less memory intensive (regex that is). You can make Zend_HTTP use cURL but given that I didn't need any of the advanced features that cURL offers I just left it so it used the default adapter (which uses fsockopen())



Code wise, top of my mental list would be to implement some so mechanism for parsing a standard form of xml listing the mods on a site. Fliggerty has a example http://www.fliggerty.com/ghf_hosted_mods.xml and I was planning to find a way to parse that every now and again and ensure the mods are in the database. They harder bit is finding a nice way of deleting mods that no longer appear on the list but appear in the database. I was also planning to allow the above to work on local xml files in the same format to allow manual adding of mods on sites that are too difficult to parse.

The two main sites that need parsers for them are:
http://theelderscrolls.info/
http://tesalliance.org/forums/index.php?/files/

There were some vague other thoughts about things I could/should do here:
http://code.google.com/p/es-search/issues/list


A custom design would be amazing. I am not that great at designing sites (although html+css I can do).

The one thing I would point out is that the reason the mod details page has the design it does is because it is sometimes offers more than one download location (http://search.yacoby.net/mods/4905/Swimming-Realism), so that may have to be factored into a design.


The host is a standard fairly cheap one. They don't seem to be complaining about the load it is putting on the server at the moment, however this doesn't have traffic that previous versions had yet.

Is http://www.moddb.com/mods/ a site you consider to add a parser for ES Search?


Yes please do. :thumbsup:

I haven't been working on this for a while for RL issues, but hopefully this time next week I should be able to work on it again for a few weeks. I have a essay to write atm


Actually, probably what I should do is to write it so that there is a generic "updater" interface that can be updated and returns the deleted and new mods. That way it would be easier to write several different updaters with totally different update methods without having to try and fit everything into the system I have at present. So I could have an xml updater and the current parser based on.

One thought I did have was to parse the forum and pull out the RELz threads. It would be far more inaccurate (very unlikely to catch everything) and the mods would have an expiry time to deal with forum pruning (6 months or so). :shrug: idk.

Yes. Although it is slightly lower down the list of priorities than the other two listed due to the high proportion of dead projects.

Yep, that's RL in a nutshell. It's good that you haven't lost your commitment for this project and keep up the good work. :tops:


I agree and that sounds good to me. :yes:


Thanks for ES Search Yacoby. :foodndrink:

Any chance you could add Wolflore to the search list? Our advlt mods require registration to see, so we'll have to set you up a search account for that (or a bot, but we can discuss that later). A link to the site is in my sig. Let me know if you're interested and what we need to do to make the site indexable. Thanks.

I've problems to establish a connection to ES Search. Is this something to do with the search engine it self or is it something else? :unsure2:

Wow, great work!

Shouldn't be too much of a problem. I am currently reworking things so that it should make it easier. (Hopefully moving from the current database to the new one shouldn't be a problem...)


Is it fine now? My host may just have been down for a few minutes or something.

Yes it's fine now. I was a little confused at first why it was down and wondered if you suddenly change server. :tes:

Yacoby, can't seem to send you a PM here.

i was trying to send this: