Hi
For many that are going to Nilheim to do the quest "Kill the Bandit leader at Nilheim" will only see peaceful guards. It looks like a bug but is actually a trap. The idea behind the quest is probably that you travel to Nilheim by road and encounter a guy sitting beside a crossroad. He claims he has been attacked by bandits and ask you to escort him to his camp. When you reach the camp he will attack you and the guards will turn to bandits. Killing the male npc with a Twohanded weapon in the tower solves the quest.

Most people will however go to Nilheim straigth from Ivarstead crossing the lake and will only find guards.
They can contiune and follow the road from Nilheim to the crossroads and speak with the NPC that sits there or just kill the npc with the Twohanded weapon in the tower. This solves the quest aswell.

Irratical behavoir can also occur with the guards if there is a dragon nearby.

I hope this help

I found a note on a dead courier about Nilheim and proceeded to go investigate the "bandits" there. I found only guards who I killed, figuring it was just a glitch, and got the "cleared" status on my map. Quite a while later I asked the Steward in Riften for work and she gave me the bounty to kill the Bandit Leader in Nilheim. Problem is, the bandits, or guards, won't respawn for me no matter how long I wait. Also when selecting "show on map" in my journal I just get the denial sound. I wouldn't mind failing the mission, I just don't like having a quest at the top of my miscellaneous list that I can't complete. Not to mention I can't get more work for the rift area. Any help would be greatly appreciated.