I'm following the directions from Hoddminir's blog as closely as I can:
http://hoddminir.blogspot.com/2012/02/from-heightmap-to-worldspace-in-skyrim.html
So I generate a height map in GeoControl2 and it comes out looking like this:
https://dl.dropboxusercontent.com/u/26225/images/MasserMap05.jpg
I save it as a .TIFF and then import it into Photoshop CS2 and save it as a .RAW file, IBM PC format, Header 0. Here it is in RAW format if you're all that interested.
https://dl.dropboxusercontent.com/u/26225/images/MasserMap05.raw
Then I use TESAnnwyn to convert it to an .ESP. I use Hoddminir's advice on the executable arguments:
TESAnnwyn.exe -i Skyrim -p 1 -b 16 -d 4096x4096 -x -64 -y -64 -h -22192
Everything seems to be going fine by this point. If I load that ESP up in the CK, I can load up the world space and nothing seems out of place. No "corrupted land" for example, which often happens if the heightmap changes elevation too quickly (if I understand it correctly).
But of course I can't load that up in Skyrim without first generating LOD. Well, I can, but then I just end up standing on a flat ocean that extends to the horizon in every direction.
Running the CK's LOD crashes, of course. So I tried running Oscape2 instead. This means I have to convert the ESP to an ESM, which I do via Wrye Bash (right-click on the ESP, copy to ESM).
Loading the ESM up in Oscape2 goes as normal. I select the ESM, along with the rest of my load order, and hit "Fill" to find and select my Worldspace. I select Feature-Map and Surface-Map and hit Extract. So far, so good.
Oscape2 chews on things and then puts me into the Preview tab. Here's where things first appear to be going sideways. Have a look at what I'm presented with in Oscape2:
https://dl.dropboxusercontent.com/u/26225/images/Oscape2oopsie01.png
Clearly that's not the way things are supposed to look. The Dimensions and Config sections are also totally wrong.
Interestingly I can actually continue to run Oscape2 and the program chews on the ESM without a problem, really. If I load up Skyrim after Oscape2's done, I can see terrain, but it's… it's just wrong. It looks like that preview, actually. Everything's just islands of broken terrain a million miles too high.
Where's the break happening? Has anybody else generated terrain properly with these tools?