i created 3 new enemy npcs. however they all just fight each other. One of them was a boss i made i set him to ally so i could talk to him before fighting. but he just attacks on sight and they all fight each other even though i have set them all to the bandit faction and are set to ally.

Nevermind.

If their Aggression is set to Frenzied they will attack everybody including each other. Also if the Boss is in the Bandit faction, it is set as an enemy faction of the player faction so it might overrule your settings. You could try the boss set to Unaggressive, that way he will still be an enemy but won't fight and then as a TIF in the post dialogue script him back to Aggressive.

Nothing happened even the boss still attacked.