I have been putting together ES Search 2. Better code, better searching, more error handling, caching. It hopefully is less CPU heavy as well. It is also now far easier to put together a parser for a site.

I would like to know any requests anyone has, or anything that they think could make it better. While I don't promise to implement it, I will at least try and make sure it can be implemented easily so I can do it at a future point.

A alpha beta new version of the search engine can be found http://search.yacoby.net/. Works better with JavaScript.

:goodjob:
ES Search is an awsome utillity, and the fact that your improving it makes it even better!!

i was wondering, does it also search Morrowind modding history? If not could you include it?

The latest version does (did), but I am waiting to see if Fligg moves it to another domain name, as it is shifting around a bit atm.

ah cool


also is there a list of sites that it searches, so its easier to pinpoint which ones havnt been included?

elderscrolls.filefront.com - Removed until they stop talking about remaking the site
www.elricm.com
modhistory.fliggerty.com - Removed until the domain stabilizes
planetelderscrolls.gamespy.com
www.shsforums.net
www.tesnexus.com
yacoby.silgrad.com - For some reason, the one small site I search

Awsome, all the main ones are there then.

Thanks again for this, if i think of anything else i will let you know

would it be possible for the results to either convert or filter out old tessource links?

And like everyone else, thank you for making this great tool for us. I just reinstalled Oblivion, and I've been using it pretty exclusively for filling the old hard drive up. Also I don't know if you saw http://www.gamesas.com/bgsforums/index.php?showtopic=1020928 over in the Ob forums. Its not something I would use, but it seems like alot of people are interested in it. :shrug:

MMH is going to have one more "move" in the near future. I am going to be registering a domain for it; I'm not entirely sure about the name though. I believe I am going to be expanding it (or rather just doubling up the function) to include Oblivion mods as well. So I will probably get a domain like TESModHistory.com or so; and then have a subdomain for each game. As soon as I get my finances stabilized a bit from buying a house, I'll get this registered and get together with Yacoby to ensure everything is indexed properly. :goodjob:

Thanks for the update Yacoby!

Yacoby, I really liked the mod descriptions field in the results in ver1, any chance of that being implemented in this version?

yes. on the home page or with a normal search, press advanced.

Probably due to my selfish behaviour I typed a name of my mod into the search engine, which rendered a dead end page (looks like it didn't work for me)? It took a long time to load, then produced http://search2.yacoby.net/search?general=Makunde&game=MW&act=Search, which was blank.

I am fairly sure that is the first thing everyone types in

Regarding the long loading times, that only seems to happen on the first search, as if the database isn't cached, or the hardrive is in the wrong place.

The blank page is strange, and I don't know why it would serve that.

Well hey, it's working now. Thanks for the engine!

hmm its not working for me anymore.

i get one of three pages:

404 - this page cannnot be found

The server was inturpeted please try again

of i get a white page with one line saying: unable to recgonse table - followed by numbers and letters

Perhaps the http://morromods.wiwiland.net/? There's stuff on there that isn't anywhere else.

I'll give it ago. Looks a nice site to parse

(I know other people have suggested it before. I finally decided I may as well do it)

Current site list:
yacoby.silgrad.com
www.elricm.com
planetelderscrolls.gamespy.com
www.tesnexus.com
www.shsforums.net
morromods.wiwiland.net
oblimods.wiwiland.net
elderscrolls.filefront.com - Removed until they stop talking about remaking the site
modhistory.fliggerty.com - Removed until the domain stabilizes
www.filefront.com - Maybe when they stop messing around with their site. Huge amount of Ajax which is a PITA to parse
Buddha's list of links - Maybe when I set up a nice way of doing things for a static list. Planned.

As I am getting fed up of this, and I know of only one (very minor) bug, I am going to start indexing for the 2.0 db soon

:goodjob: nice work.

I tried ESsearch again and its working agian, so dont know what happened there lol

Ah, I got an idea, how about an "image search" of some kind? It would be useful to search for screenshots with. Not sure how it would be implemented though, it could index links from the screenshot threads here on the forums, and collect thumbnails etc from the mod pages.

In addition to this, when searching mods, there could be an additional "find screenshots" option beside entries, would be useful.

True Story!

Also, perhaps Canadian Ice's forum, if that's feasible?

So Yacoby, as it seems that indexing the downloads section of my forum isn't possible, what is necessary to make it work. And what will optimize it?

(I intend to use my new-found PHP skills to create an integrated download section at GHF at some point in the maybe-not-so-near-future, and I want to ensure that it can be easily and efficiently indexed.)

Unfortunaly this is a "great idea and I would love to do it" thing but.... It would take quite a time to implement and use up quite a bit of CPU resources. I will keep it in mind.



I have had another look and I may be able to parse it. Maybe. I don't think I can parse descriptions, but I should be able to grab the title and author.


Regarding parsing, the vauge good ideas are:

A page with a mod on should be identifiable via the url. For example, a TESNexus mod page is:
http://www.tesnexus.com/downloads/file.php?id=number
Which makes it easy for me to give it a higher priority, and not try and parse junk pages, filling up my error log. It also ensures I don't have to parse thousands of pages to get to a single mod.

Every mod page should have an exact HTML structure, and the section with the mod details shouldn't differ at all (Which is part of the reason I struggle (and have given up trying) to parse some smaller sites).

The mod elements should be identifiable in some way. Wiwiland has every element (Name, author, version etc) in a unique css tag which is ideal, even though they have the author details prefixed with author. I can strip that off.
Table based structures also aren't a problem.

Ok, I can do all of that. I'll keep all of those guidelines in mind when working out the code for it. :goodjob:

So, any more refining done to this? Its such a helpful resource.

I just have a request/idea. Is it possible to make a firefox search plugin from it like your PES and other site ones? It would make it a lot easier.

I'd love this! There's a PES one though? Where at?