Fligg: I am back messing around with essearch and am looking into this. There is an error in the xml document: http://www.fliggerty.com/ghf_hosted_mods.xml on line 1263 in that from what I can see the title element is not closed and description is not opened leading to this problem.

The one issue I can see is that possibly my scraqer may run out of memory when parsing the larger XML document. We will see. (This may just become an issue if the number of mods that mwmoddinghistory has increases by a large amount)



It isn't a problem, the code for pulling the data out of the xml is easy. Setting up the backend is slightly more difficult (only slightly)

How have I not known about this? Insta-bookmarked!

I second some other's calls to place ratings, where possible, into the search results, and possibly make the search results sortable (by date submitted, by rating, etc).

This has already simplified my search for clothing replacers, so thanks! :goodjob:

Good to see you around Yac. I'll check that XML error. It's probably just a bad character from one of the newly posted mods. I definitely am working on something better than just posting mods to a forum.

Fligg: Any news on getting the error sorted?

Basically I need the mod information in a standard format. I could probably parse it now, but I wouldn't really be able to get author information.

Let me know how you want the information sorted and we'll get on it. Some of our content is only visible to registered users; how do you want to handle that? It's our advlt and AlienSlof's (which is mostly advlt, but some isn't) content that's only viewable to registered users.

I would create an account for the bot. I am not sure how I would handle it on my sites side. Shouldn't be too much of a problem though as IIRC your site prompts for sign up with a nice-ish error message and the same thing happens for TES-Nexus with advlt mods.


I have had another quick look and I am fairly sure that I could get the majority of the mods, the one issue being that I can't reliably get the author so I would in most cases just have to leave it as "Unknown". (I would be able to get the author if an entire sub forum was made up of one author, Solf for instance).

If you see the lack of being able to parse author as being an issue (Not being able to would occur when the title isn't in the format "NAME'S mod" or "mod by NAME", basically one of a finite set of patterns. (I couldn't find anything indicating the author in the first post)) then you may want to look into something like the XML that Fligg did. Or maybe something in the first post. In other words, some way to find the author that can be broken down into a logical set of steps.

Yacoby => sorry to wrote this here but your mailbox is full.
Can you take a look at my post please :sweat: => http://www.gamesas.com/index.php?/topic/1180562-yacobys-wrye-mash-fork/page__view__findpost__p__18229106

We'll reorganize our mods with the author's name if you'll add our site to your engine.

I am getting errors for ESsearch - saying its timeout and now saying theres a problem loading page. Anyone else having this problem?

I'm getting an "unable to connect" too.

May be down for maintenance? :shrug:




KF

I hope so

It is down, I know about it and am complaing to my host about it (They move boxes and don't have something installed that they should have)

If you really really need to use it, there is a very development version here that won't be up for long. Has most of the data though.
http://tessearch.jacobessix.com/

Thanks and keep up your hard work on this. :foodndrink:

Forgot to add, if you see any stupid errors with the above (like not filtering by game, which was happening just now) feel free to point them out. I know about it not showing the number of mods for each game correctly.

That version there is the remote development build for the new version, which mainly has fixed (and more) scraqers and far faster searching.

FYI your old http://search.yacoby.net/ site is now back online and here some statistics. :user:


As you can see there is 380 mods on the new site that aren't included in the statistics of your new site.

Why not ask Fligg if you can host ES Search on GHF and if ES Search is down you can look on http://status.fliggerty.com/index.php of what status ES Search has.

It still had the old Morrowind Modding History mods on. I removed them and it reduced the number of mods by about 550 or so. It may drop again as I may remove all the wiwiland mods so they can be reindexed to make use of UTF-8.


Given how the the newer versions work, I quite like having root access to my server. It will all be shifting over to the new server soonish anyway. (the tessearch.jacobessix.com one has a load of improvements, like full UTF8 support (stil needs tweaking), better parsers and that sort of thing).

(Of course the url will change).

Yacoby, I sent you an email at the address you have on file at GHF FYI.

Are you planning on setting up any sort of wildcard mod_rewrite redirect or anything at the old URL? That would be most useful for those of us that have incorporated ES Search into various development projects....

Yacoby, would it be possible to get some sort of data dump of all mods hosted at one particular site you have indexed? ie can you get a list of all mods at ElricM?

Uh, that address is old. Any chance you could change that address to jacob at jacobessix dot com? That way I can recover my GHF password...


Yes. The old url will work fine when I port it all over.

IDK when that will be. I have loads of other projects to try and sort.


Sure. Anyone who wants is welcome to the entire db dump if they really want it. It may not be totally up to date though.

Done. I'll resend my email from yesterday.



Excellent to hear. :celebration:



Yes please...that would be fantastic. What would your preferred method of transferring it be?

If you just want the ElricM mods:
https://docs.google.com/spreadsheet/ccc?key=0Akfb_9mpiGHNdDdVa1lLdXZ6TVVSQThfMVcwdjFZVlE&hl=en_GB

The majority of the db is here (or will be in about 3 mins when it is finished uploading):
http://dl.dropbox.com/u/26677509/Morrowind/plain_partial_backup_11_09_29.sql.backup

Can anyone see any horrendous errors with this?:
http://www.tessearch.com/

If not I will redirect the old site to the new (all old links will work fine)

In other news, hopefully soon I will have MMH parser setup fully.
I think in total I have about 14 sites being parsed (just, a few are broken) and after that you start running into issues with the number of mods.

See here for parsers that haven't been done: https://github.com/Yacoby/es-search/issues?labels=PaserReq&sort=created&direction=desc&state=open&page=1

What I have in various states of working (FilePlanet, ElricM, Fliggerty.com aren't working):
"http://www.elricm.com/"
"http://elderscrolls.filefront.com/"
"http://www.fileplanet.com/"
"http://modhistory.fliggerty.com/"
"http://planetelderscrolls.gamespy.com/"
"http://www.tesnexus.com/downloads/"
"http://morromods.wiwiland.net/spip.php?article"
"http://oblimods.wiwiland.net/spip.php?article"
"http://yacoby.silgrad.com/MW/Mods/"
"http://tesalliance.org/forums/index.php?/files/file/"
"http://www.shsforums.net/View.php?view="
"http://www.wolflore.net/viewtopic.php?f="
"http://www.fliggerty.com/"
"http://mw.modhistory.com"

Any other sites that need doing, let me know. They need 250+ mods for me to consider writing a full blown scraqer for the site soon as I don't have much time for projects atm, if they have less they are likely to be put on the backburner. If less than that, if you can setup an XML/JSON sheet with all the mods on, then I can just scraqe that one a week or so (as it is a standard format it takes me about 10 minutes to setup a scraqer for it, including testing)

You can add this ite well known and update translated mods (and also mod created by frenchs) into French =>
- for Morrowind mods => "http://www.confrerie-des-traducteurs.fr/morrowind/index.php"
- for Oblivion mods => "http://www.confrerie-des-traducteurs.fr/oblivion/index.php"

Ok, cheers. Added to the todo list.