I tried saving as Dxt5 with a completely black alpha channel and a scale of 20 on the height generation to try to bring out the cloth texture.
I use Photoshop CS3 with the Nvidia normal map filter, btw.
This made a major difference to the glossiness.
http://i237.photobucket.com/albums/ff292/VonGrantoven/TacBetterGloss-1.jpg
Unfortunately, its not quite there yet. There are still some very clear highlights (you can just notice the brighter areas in the screenshot), which inspite of the texture, seem to act like a light reflection on a smooth polished surfaced.
Its much better but still a bit annoying.
Is it possible that these are somehow built into the actual model rather than coming from the normal map?
I wouldn't know, sorry.
Mm, I am very new to Fallout's ways, and MadCat likely knows much better, but I probably would recommend trying 100% black, which should suit a matted surface. You could perhaps maybe also try adding a noise layer (correct term?) over the normal map, which I am presuming is the same used by the original 'smooth' helmet? Mm, where the original more defines the overall shape, the 'noise' would be a normal map generated from your diffuse texture, to add... well, more texture. ^_^;
Overlay the red and green channels, and multiply the blue channels to merge normal maps, I think.
Edit: Sorry if that was redundant. I am not sure what "and a scale of 20 on the height generation" meant, and assumed it to mean you were converting the normal map to a height map (though, the scale doesn't effect that, as far as I know), and then reducing the brightness of that and using it as the alpha channel. Mm, and I use Photoshop CS2 with the same plug-in... just to avoid confusion. ^_^