I was looking over some of the magic mods out there ( Midas, AOMS, LAME, Supreme Magicka) but I am not sure if I really want to use them. I am taking it slow with the mods because I don't want to go to crazy and overload my game with stuff that I won't use. Also I am adding them one at a time to make sure that they work right and don't cause any issues with the setup that I already have because what I am using already are the most important mods for my game and anything else is just fancy extras.
I suggest at least looking a bit more closely at, for example, the Less Annoying Magic Experience mod. There's a reason for its name, other than just being catchy.

Supreme Magicka is also good, mind you, and both work together quite well.
Keep in mind, too, that such mods are often fairly modular. LAME certainly is - very much so, even. So if you only want one or two of the options, you can do that.
Essentially, I look at such mods as
fixes for vanilla issues, as much as (no, more than) providing "fancy extras". Just one thing I wouldn't want to do without, is a feature that can be gained via LAME, or via ROM (worth a look too, by the way, if you are already keen on using OOO, and appreciate immersion) - and that is, choosing your starting spells. More to the point, choosing a "flavour" of starting spells, and receiving a couple of spells that fit. Supreme Magicka offers another related option I very much appreciate, and that is starting with a few scrolls each of Flare and Heal Minor Wounds, rather than the spells themselves. Just one way in which LAME and SM are (or can be) complementary.
Basically I am just going to live the life of my character, find a job for him to do go for walks in the wilderness explore the countryside and just enjoy the world. I won't be randomly raiding forts, ruins, caves, or anything else. There will be a reason for exploration of things like that such as rumors of rare magics or books.
This is very much how I play my curretn character as well, by the way.

edit: And yes, there's really no downside to adding OBSE to your setup. Tons and tons of potential upsides, however...