I have OOO 1.33 with GOTY installed and my leveling attribute modifiers seem messed up.

My understanding is that in stock Oblivion, when you gain 10 skill ranks you get the chance to level up, and when you level up, you can increase your attributes. The formula for increasing attributes is: for every 2 skill ranks, you get +1 attribute increase for the attribute that is governed by that skill. That is my understanding, please correct me if I'm wrong. And I had thought that Oblivion (and all its associated mods that are installed with it) don't change that. But that's not what seems to be happening for me.

I have a level 2 character that I've been waiting to level to 3 until I can get +5 Agility, +5 Speed and +5 Endurance increase. If I level him right now, he gets a +3 Agi, +3 Speed and +4 Endurance. So I have been waiting a while and I got impatient, so I went and just leveled him to level 3 and took those attribute increases. Then I noticed that I was "eligible" for another level-up. So I immediately slept again, went to level 4. To my surprise, I could add +4 Agi, +2 Speed and +2 Endurance this time. Now this confused me. Why would I be able to increase Agi AGAIN, if it wasn't maxed out on my 2 - 3 levelup? So again, I took them, and was able to level-up again to level 5. This time with +4 Agi, +3 Speed, and +2 Endurance. Ok, so now I'm total confused about this system. It doesn't seem to fit the description of how stock Oblivion lets you increase attributes. So what is going on?

I actually think the original was 1 skill rank for 1 attribute increase.
Also you must gain 10 major skill ranks(the one you chose at character creation) to gain levels(ie if you want to optimize your level ups you need to level minor skills).
OOO doesn't change any of that.

There are leveling mods available that can fit your every need and get rid of that ridiculous modifier thingie. nGCD+Progress is a popular combo, there is also realistic leveling, oblivion XP(XP system D&D style), and more . If you really want to stick to the original there is even a mod that gives you automatic +5s...

Ok, so does the way my attributes were increasing make sense? (+3 Agi, then +4 Agi, then +4 Agi for levelups 2->3->4->5) What is the formula for normal OOO? How are the modifiers calculated?

It's not OOO. OOO doesn't change the way leveling works at all, other than optionally making it a lot slower to gain skill increases. The problem is your understanding of how vanilla leveling works.

First thing, you only get to level up when you have 10 skill increases for your major skills, you can get as many skill increases as you want in your minor skills, but without 10 major skill increases you are going nowhere.

Now, what I don't think you've grasped is that once you have those 10 major skill increases your attribute modifiers are frozen for that level, so once you've got 10 majors you cannot influence how big your attribute modifiers are by leveling up minor skills. Instead those skill increases will start counting towards your attribute modifiers for the next but one level up.

When you sleep you will have the opportunity to level up for as many bunches of 10 major skill increases as you have achieved, with appropriate attribute modifiers attached to each bunch of 10. Note that minor skill increases do count towards the attribute modifiers.

Essentially the game is storing potential level ups as you qualify for them, you then claim as many of these level ups as you wish when you sleep. Most people of course sleep and level up as soon as they are able to, but there is a technique called Under leveling where level ups are deliberately not taken.

To get a +2 you only need one skillup of that attribute
To get a +3 you need five skillups
To get a +4 you need 8 skillups
To get a +5 you need 10 skillups

OOO does not change this formula

So, the FIRST 10 major skill increases you achieve in a level and the minor skill increases you achieve during those 10 majors are the ones that contribute to the modifiers?

If so, that is very different than what I had been thinking...

Yes. That's what I was trying to say. It comes over a bit incoherent when I re read it.

As an aside, I wouldn't sweat too much about attribute modifiers when playing an overhauled game, eg with OOO, Frans FCOM or whatever. Unlike Vanilla most of the game is either completely deleveled, or the leveling is seriously cut back. Thus you don't have to worry about falling behind the difficulty curve.As you level up you will get stronger while your enemies (generally speaking) won't. Losing out on +5 modifiers will prevent you being as strong as possible at each level, but you will get there in the end.

Many people use a leveling mod, eg Realistic Leveling or Oblivion XP. I don't, I play with the vanilla system, but I don't work towards getting good attribute modifiers, I just play naturally, and take whatever I get offered at level up time. In two years of playing modded Oblivion I may have had 1 or 2 +5s, usually I get +2 or +3s.