I am currently toying around in GMax with moderate success. I have used some vanilla textures and I downloaded a thatch texture.
https://onedrive.live.com/?cid=B60C11D037429D8E&id=b60c11d037429d8e%2130784&v=3
It looks ok in general but how do I achieve the effect of the uneven straw hanging off the edges? Right now it looks like a very neat hair cut.
https://onedrive.live.com/?cid=B60C11D037429D8E&id=b60c11d037429d8e%2130783&v=3
I thought that colouring black parts in the texture would achieve this but it shows as black. I searched around and read some info. about making the texture alpha and/or compressing to a different format. I use BC3/DXT5 for diffuse and normals at the moment.
Help and advice appreciated, thanks.

Look at existing roof textures such as for Whiterun buildings and see how they did it there. Typically there is an alpha layer that is mostly all white but at the bottom of the texture it's painted black to mark transparent areas. The UV map for the roof texture is then aligned in such a way that the transparent bottom part of the texture is only used on the edges.

The effect is primarily achieved by having one or more layers of "skirt" polygons sticking out from the edge that have a transparent texture on them, UV mapped so that the transparent parts "hang off" the main model.

I looked at farmhouse01 in Nifskope and it has a "skirt" which is where the thatch hangs off. It's a different element in GMax and can be UVmapped separately. I am going to play around with this after modding my model.
Thanks.

I think I am making it worse. I used the thatch texture that the game uses for farmhouse01 but... well I dunno what else to do.
https://onedrive.live.com/?cid=B60C11D037429D8E&id=b60c11d037429d8e%2130786&v=3
https://onedrive.live.com/?cid=B60C11D037429D8E&id=b60c11d037429d8e%2130785&v=3

Does your texture have an alpha layer as suggested above?

Hi, Antstubell.

The transparencies are managed via alpha channel on the diffuse map then add a NiAlphaProperty node on the trishape to get the trasparency working.

Hi Ant, Looking at some of your textures your UV mapping looks a little off. I don't know if you've seen this series of tutorials before from Millenia, I know they're for 3DS Max and Fallout but he goes into UV mapping in a big way https://www.youtube.com/watch?v=2Tgy0lBdJK0&list=PL925958D31D8D30C7&index=1 You won't have anywhere near the tools he has at his disposal for UV mapping. You can also skip the parts about Normal map baking since you can do a pretty good job in GIMP.

When doing transparent meshes like the edges of your thatched roof and things like leaves etc. I will make sure I've got one side of the mesh UV mapped properly and the textures all working, I will then duplicate the mesh and with all the polygons selected flip the normals and then reattach the duplicated mesh with the original (with no vertex welding), that way you will end up with the textures matching up correctly when you compare sides.

Ok, thanks guys. I am checking the tutorials and just learned how to make an alpha channel. Gonna be a long day.

Manny - (Anybody)Do I just copy a NiAlphaProperty node from an existing mesh and paste it into the BSLightingShaderProperty of the NiTriShape where I have the alpha texture? Is there anything to be changed in the block details?

EDIT: Scratch that, yes I just add that node, tried and tested.