I want to make sure that this is dependent on the NPC's Aggression stat and their disposition toward the source of the friendly fire. Anyone with 100 Aggression should be completely unforgiving. I depend on this for at least one mod effect.
If you can find a way to work Confidence in there, that'd be pretty neat too... lower Confidence might make the NPC more likely to stick with their friends even if they're taking hits from same. A bit lower still, of course, and they'll just run.
I would love to see a breakthrough in tweaking AI in Oblivion, but I'm concerned that it'll be harder than some of the comments imply. Remember that Bethesda started out with a very complex Radiant system in place, then dumbed it down because it produced too many unpredictable and nonsensical behaviors. For instance, go into the CS and notice that almost every creature and NPC has Aggression 5 or Aggression 100. Ditto for Confidence. In order to get satisfactory flee/seek out allies/surrender behavior (I've played with this a bit in Diplomacy and Mercy), you either have to change a huge number of CS entries, or you have to run scripts for every person in combat, all the time, taking into account their factions and so on.
That being said, I really only scratched the surface with Diplomacy and Mercy, and a dedicated effort to change the way NPCs make decisions in combat might make some real progress. If it does, I'd love to integrate some of D&M's surrender/truce features into the AI overhaul, since they tie into the sorts of behavioral models you're talking about. Good luck!