There are two ways for this: what I call "soft" barriers and diminishing returns.
"Soft" barriers are when the game gradually increases the difficulty of some task, action, or thing to try to immersively warn you that you can't do something.
Diminishing Returns is when doing something over and over gets you less and less.
Together, soft barriers and diminishing returns encourage exploration of the landscape and your abilities, and reduce the feasibility of level grinding.
Allow me to use the magic system as an example. (Remember: "easy" and "hard" or "simple" or "difficult" should ALWAYS be relative to your level. That restore-1-hp spell is only hard if you've never done magic before!)
Put in soft barriers around what spells you can cast. Combine what you did in OB and MW: have a system that lets you cast more reliably, but lets you try your hand at really hard spells. Simple spells should always work, no matter how tired you are. You should be able to cast most spells "at your level" extremely easily, unless you are very tired. Relatively hard spells should have a chance of not working. The harder the spell, the more it should be 'sensitive' to how wounded/fatigued you are. And spells that are, relatively, super difficult, have a high chance of failure, but also have a small chance of backfiring.
The effect: You try to cast spells mainly "at your level," though simple spells are nifty if you're tired and still useful if you only want a small effect. Hard spells are rewarding to try: sometimes they work! But they're risky in that if that spell really needs to work right now, you might fail it and be eaten by that scamp. Very very hard spells have a higher payoff (they're very effective) but carry some danger. This creates a "comfort zone" where people don't have to worry about spell failure, while allowing people to move above or below that zone with different rewards.
Put diminishing returns in to prevent level grinding and encourage people to try, occasionally, things above their "comfort zone" of where stuff reliably works. On the one end of the scale, small, simple cheap spells should do very little for gaining experience. If you cast a spell that takes 4 magicka, and your "comfort zone" of spells is around 20 magicka, that 4 point spell should do almost nothing. In the middle of the scale, with that same comfort zone, if you cast a 20 pt spell, you gain some experience-- you've done it before, okay, but you might still learn something. Moving further up the scale, a 30 pt spell should gain you experience even more, because it's new. The harder the spell, the more experience you should gain from having successfully cast it.
The effect: You stop casting that simple spell 800 times to grind up your level. In fact, it's more efficient to cast your normal skills to level up, and even better to try spells where you have some chance of failure, because you learn more from "trying harder." And even though you gain a lot of experience from super-difficult spells, the fact that you're unlikely to cast it right and that you might actually damage yourself with them makes them not so good for grinding.
In total, this system would give each kind of spell a new usefulness.
1) Super simple spells are easy to cast and are 100% reliable, no matter how tired you are. Good part: no chance of failure, ever! Bad part: Not very effective spells, and utterly worthless to leveling.
2) Spells which are "right in your comfort zone" are 99% reliable, and you only need to worry if you're very tired or very wounded. Good part: Solid, comfortable, and reliable. Very little chance of failure, though maybe every now and then. You can actually gain modest experience with these spells. Bad part: While these spells are effective, they're not the best way to gain experience.
3) Hard spells have some rewards now. While you need to worry about if you can actually cast them, they're not so difficult you need to worry about them "going wrong." And you learn a lot from casting them! Good part: Good experience and they're very effective spells when they work. Bad part: You do need to worry if they will work especially if you're tired or wounded. Maybe pick a simpler spell for those time-critical moments in combat.
4) Super-difficult spells have some rewards too. Mainly, these things can do fantastic stuff if you get them to work... if you can get them to work. You need to be very well rested and ready to swig a restore health or dispel potion if it goes wrong. Good part: Can do crazy things and you learn a lot when you succeed. Bad part: you might singe your eyebrows off... or worse if these go wrong. They're super finicky when it comes to how tired you are, too.
So, in all: This system lets all levels of spell difficulty have its own use, encourages light experimentation outside of the comfort zone, but discourages people from doing crazy stuff because they're likely to get hurt.
But you gotta keep in mind: THIS ISN'T LIMITED TO SPELLS!
So, what else can we use this system for? Well: combat moves, exploring the landscape, making potions, running, stealth... this applies to the entire game. Just substitute in the right words.
Basically, easy stuff should be totally reliable, 100%, no matter how tired/dead you are, but should not give you experience. Stuff in your "comfort zone" should be 99% reliable, only "failing" or whatever when you're very tired or nearly dead, say under 1/3 health or fatigue. It should give you a modest experience bonus, enough to keep you leveling up, but slowly. Stuff just above your comfort zone, the "hard" activities, should be the "best" stuff for leveling up, in that it gives you decent experience, but, it has some chance of "not working," especially if you're below, say, 90% hp/fatigue. Very hard stuff should give you lots of experience but not actually be useful for leveling, because of the high likelihood of failure and chance that you hurt yourself somehow. It should be much more likely to fail if you're below 98% of health/fatigue.
Keep in mind, now, that this will require fun fun nonlinear equations to get the right balance of experience vs not so much experience, reliable vs unreliable, "effective" vs "non-effective." I could speculate here on exactly what type of equations you'd use, but I suspect you guys don't have that much interest in multivariate equations. A complex equation is more likely to be smoother and better if it takes more variable into account. It may or may not require array variables, but, guys, shame on you if you don't have array variables at this point.
As an example, though, to determine these things, this is what ought to go into the equation for spells:
Intelligence (higher = gains more xp per spell overall + more likely to succeed, if at 100% health/fatigue)
Willpower (higher = more likely to succeed if not at 100% fatigue)
Endurance (higher = less likely to be damaged by very difficult spells, more likely to succeed if not at 100% health)
Luck (higher = more likely for everything to be peachy)
How high your particular magic skill is (higher = more likely to succeed, less likely to have hard spells go wrong, etc)
How high your other magic skills are (higher = more likely to have harder spells not go wrong, slightly more likely to succeed)
Health and Fatigue (lower = more likely to fail and more likely to have harder spells go wrong)
Anyway, I hope you all read that wall of text. And I hope you can make the leap to extend this idea to other "systems" in-game, like combat and fighting leveled creatures and exploring new land and so on. I can write out examples for all of these but you should be able to fill in in the blanks yourselves.
So, what do you think? I think the math behind this would be a bit snarly to implement, but since it's all reading the current state of affairs and then crunching it all into one output-probability (does it work, if no, does it go wrong?), It should be actually feasible.
:)
EDIT: New thought. I forgot to mention, if you want mages to stay pure mages, then make them have some sort of increasing penalty (costs more magicka or spells less effective) for wearing heavy armor or trying to cast with a weapon in hand or something like that. This is how soft barriers applies to keeping people "in their class." At the same time, maybe you can get more xp from trying to cast with armor/weapon, because it's "harder" to do? So make it the easiest thing to be is a pure mage, but, you can do the battlemage thing, so long as you try harder.
SECOND EDIT: New thought. If you want to see if this system is applicable, then you just need to see if the thing you want to apply it to has "easier" parts and "harder" parts, or "things we do want you to do" and "things we don't want you to do" parts. I actually can't think of a decent way to apply this to combat, that doesn't quite fit in. Oh well, more to think about!