Hello,
I am aware that the limit for mods for Oblivion is 255 because of the way the modindex is stored, but theoretically (maybe with some black magic), could this be changed?

I'm pretty sure 255 is a hard limit imposed by the game engine. What people have done to work around it is merge mod files to reduce the total number of them below 255 (that number includes the game and DLC master files, BTW, so the total number of plug-ins you can use will be less than 255). I've never used that many mods in a game, so I don't know how the merging process works.

If you could reprogram how the Mod Index is stored, yes. But that would be ill-advised, unless you're a professional programmer.

The 255 limit is because the index codifies each mod as a unit of an 8-bit binary code, also known as a byte. If you expand the index to use two bytes for mod identification instead, you'll have more than enough space for any modding needs. And if you expand it for three bytes... well, let's just say that I'd like to see how stable Oblivion is if you manage to hit that maximum value.