For some reason after I attacked a guard, rather than try to fight me, or accost me he ends up talking to me over and over. I applied some clothing modifications to him but I don't know
why his overall behavior has changed...any idea what might be causing this?

Are you using any mods that affect Guards - RGO, SM Bounty, undercover etc.?

No actually I think I found what it is, I changed the guard's race from Imperial to an Imperial race with a beard from WAC, and it made him lose his guard dialogue...(just tested it as a regular "Imperial" and it worked.) But I want to use this bearded race, is there any way to work around this? The beard is key to the NPC's design.

That will break Guards. Their dialog is dependent upon them being classed as Guard and also being of the Imperial race. Otherwise things break.

How does VOILA do it? That mod adds imperial legion guards of different races to the game...

They modified the guard dialogue so that it works with other races. The Crime quest, which is what controls all that, is specific to Imperials. VOILA adds the other races. Not all of them, but Nords and Reguards in particular.

So might there be any way I can port this in the same sort of method? How the guards look is pretty important to me lol

Certainly, but it's a good bit of work.

This is a common issue when user put xeosp after ooo in my country. xeosp changes NPCs' race, face, size and many things.
Wrye Bash can generally resolve most of these through the proper use of tags. Try tag the mod with NPC.Race and NpcFaces, then rebuild your patch.