4 Gigs, but the fact that I'm running x32 bit might be a limiting factor there. Also, I was able to use the tool (albeit slowly) in the Oblivion CS when I was modding with that, so, maybe a post for the GECK bugs thread?
I've tried in Vista 64 with 8gb of RAM; it crashes immediately. As far as I know, it's just flat-out broken.
Though I think I managed to import a heightmap a while back. The data it produced was random garbage, but at least it didn't crash. Trying to paint anything, though - kaboom.
I've run into the same issues with the heightmap editor, just flat out doesn't work regardless of what I'm trying to edit. I was able to get a heightmap I created elsewhere in game and working just fine with TESAnnwyn though, unfortunately I haven't seen any workaround for it besides simply not using it.