This is an area of the game which is crying out for improvement; alas there are precious few (if any) mods that improve the AI globally. Rather, piecemeal (no offense intended) solutions are provided by way of mods that address the AI within a certain area of activity, e.g. combat, or guard behaviour.
BRAINS (or INSTNCTS) by Syclonix was an attempt to address AI in combat, however he had to abandon it due to RL pressures. There are a few others which I can’t think of off the top of my head, but will post them if/when I do….
Here is a summary of SOME of the mods that address, more or less, the AI:
http://www.oblivionmodwiki.com/index.php/Category:AI
Edit:
http://www.gamesas.com/?showtopic=1005791 is Syclonix's WIPz thread for BRAINS/INSTNCTS
http://www.gamesas.com/index.php?/topic/1060697-wipz-npc-ace-npc-advanced-control-engine/page__hl__instincts is Qquix's NPC ACE - an advanced NPC control engine - more a modders' resource, I think(?).
Edit 2:
http://www.gamesas.com/index.php?/topic/1084200-any-mods-with-interesting-ai/page__p__15798063__hl__advanced%20control%20engine__fromsearch__1entry15798063 is a previous thread on the same subject which garnered a grand total of two replies; one from yours truly & one from phoenix1213 who I see is currently reading this thread. Unfortunately, I think most modders put AI into the 'too hard basket' as I believe it is highly scripting intensive and difficult/time consuming to debug and balance.