I really like fcom and I would keep it, but it is just too unstable
No, it is not.
(...) (I)t's such a hassle to reinstall every time a new mod breaks it.
You do not have to, even in that
extremely unlikely event. But I can imagine that working with that assumption might be "a hassle", yes!
So I'd like to find a replacement for it if there is one. I guess I'm mainly looking for one that will make the game hard at a low level but become easier as you level up.
You might like TIE (Tamriel Immersion Experience), in that case. Or - for a far more elaborate set of modifications - something like OOO (Oscuro's Oblivion Overhaul).
Et cetera...
I'd also like random loot, new quests/items, random monsters... so yeah I guess just the closest thing to fcom there is.
You could just add your own selected mods that provide "loot", quests, items, monsters, and so on.
I'm fine combining multiple overhauls too, as long as they're fairly stable. Seems like fcom likes to crash my game and react badly to mods for some reason.
You will need to become accustomed to reading documentation, and using the correct tools properly, in order to avoid this, even if you "simply" want to use a fair few mods (with no overhauls,
per se).
A useful place to start is http://tescosi.com/wiki/Main_Page. In addition, http://www.tesnexus.com/downloads/file.php?id=35230 is highly recommended, if you are new to Wrye Bash (which I am guessing you might just be).
Some free hints:
Use BAIN (and the rest of
Wrye Bash), and
use BOSS. And, most of all, take the time to read, heed what you have read, and act upon it accordingly. Readme files (etc.) are there for a very good reason.