Here is the complete guide to ArchiveInvalidation.txt:
If an entry in archiveinvalidation.txt has a \ character in it, then everything up to the last \ (i.e. the path part of the entry) is hashed and used to invalidate folders. For example, if the entry in archiveinvalidation.txt is menus\prefabs\test\test.txt or menus\prefabs\test\ then it will invalidate (by hash) the folder menus\prefabs\test (and all the files in it) if that folder exists in any .bsa file
If the entry does not contain a \ character then the filename/extension are hashed and used to invalidate individual files. For example, the entries test.txt and menus/prefabs/test/test.txt will both cause it to invalidate test.txt by hash if that file exists in any .bsa file, regardless of the path it resides in. If a file is invalidated, the file load code will act as though the file is not even in the .bsa file and will say "sorry, no file by that name in this file" (thus causing the code to look elsewhere).

That should completly explain the inner workings of archiveinvalidation.txt based on my reverse engineering of the related code, both the part that reads and parses ArchiveInvalidation.txt and the part that loads the .bsa files and invalidates things based on archiveinvalidation.txt. (if you dont understand what file/directory hashes are, open a .bsa file in bsa commander and use "save list". Then look at the resulting txt file). I have seen the claim "Listing a path in your AI.txt has the same effect as listing individual textures, in the sense that it causes all the items in that path to be invalidated. However, once you've listed one such folder, Oblivion will automatically invalidate all further instances of any common filenames it found in that first folder. *" but specific anolysis in the code shows this to be false.
If I put textures\menus80\icons\clothes\middleclass\01\f\ in my archiveinvalidation.txt, it will invalidate textures\menus80\icons\clothes\middleclass\01\f\pants.dds, textures\menus80\icons\clothes\middleclass\01\f\shirt.dds and textures\menus80\icons\clothes\middleclass\01\f\shoes.dds and will NOT specifically invalidate any other copies of pants.dds, shirt.dds or shoes.dds (the only explanation for the above claim would be if there is code which is somehow invalidating files with the same hash as one that's already been invalidated but I see no evidence that such code exists)

As for bsa-redirection (that is, creating a dummy bsa file and editing SArchiveList) the way the game works is that the game retains a special reference to the first .bsa file it sees that has the "I contain textures" flag set in its header. This seems to be an optimization (the same references exist for other file types like meshes and audio) to allow it to find textures from the main texture file (which is likely to be the one containing most of the textures) more easily without having to search every bsa file.
Why this optimization causes certain textures to fail to properly load from the data folder I dont understand.

I intend to dig deeper into the code to find out exactly whats going on but I need the help of some of the gurus here who know this stuff.
I need 2 things:
1.A reproducible case (i.e. specific set of files in the data folder, specific configs etc) for the claim "Listing a path in your AI.txt has the same effect as listing individual textures, in the sense that it causes all the items in that path to be invalidated. However, once you've listed one such folder, Oblivion will automatically invalidate all further instances of any common filenames it found in that first folder. " if someone can produce one. I also need steps I can use to go in game and see that a texture file with a matching name has been invalidated when it should not have been.
and 2.A reproducible case (i.e. specific set of files in the data folder, specific configs etc) for the situation where having no ArchiveInvalidation.txt file, no edited SArchiveList and no edited .bsa files causes a texture file loose in the data folder that is overriding a texture file in Oblivion - Textures - Compressed.bsa to not be correctly loaded (i.e. its reading the file from Oblivion - Textures - Compressed.bsa and not loose from the data folder). I also need steps I can use to go in game and see that the texture file has been incorrectly loaded.

If I can get test cases for both of these issues that I can play with myself (including save-game/steps to see in-game that the texture isn't loading or that its using the texture from the archive instead of the loose data folder texture) I should hopefully be able to find out exactly why its loading the wrong texture/not loading the texture at all and hopefully come up with a genuine no-more-issues-ever fix for all this that will eliminate the need for archiveinvaildation.txt, edited SArchiveList, edited .bsa files or any other such things.