I don't know if there is an 'exact' formula since there are a large number of factors at work.
The http://www.uesp.net/wiki/Oblivion:Aggression is...
...if the NPC's disposition towards you or any other NPC drops more than five points below the NPC's aggression, the NPC will attack.
If you check the AI for most of the friendly NPCs (like merchants, etc.) you'll notice that they're all set to Aggression 5, which effectively prevents them from initiating an attack against the player. (An Aggression of 0 prevents them from defending themselves as well.)
So, to know whether or not an NPC will attack a player, you have to know what their disposition is, which http://www.uesp.net/wiki/Oblivion:Disposition.
Faction modifiers can raise or lower disposition, but only the lowest faction modifier applies. So, if the player is a member of a faction that gives him a +10 disposition modifier toward an NPC, and is also a member of a faction that gives him a -20, only the -20 applies. If the player has his weapon drawn and talks to an NPC that doesn't like his faction, it's not hard to push his disposition below 10. Committing crimes, having a high infamy, etc., can also be factors.
Personality determines the starting disposition toward the player and works in conjunction with the personality of the NPC (I've never worked out the base value, though). A player with a high Personality is http://www.uesp.net/wiki/Oblivion:Personality. Apparently, this was a problem even with vanilla OB, which is why enemy NPCs usually have an Aggression of 100.
If you want to ensure that NPCs attack players even with high Personality, you're best to assign them to factions with a high negative disposition modifier toward the PlayerFaction and ensure that their Aggression is > 5. For NPCs you
don't want attacking the player, ensure that their Aggression is < 5. The tricky NPCs are the ones you want to be aggressive enough to attack the player, but not so aggressive that they attack other NPCs. If you put all of these NPCs into a faction with a high disposition modifier toward each other, you should be able to keep them in-line. In any case, you're going to have to play around with the numbers a bit to get a system that works for you.
hth