So I had this thought, what if there was actually something we had to do to cast a spell? Or maybe only had to do it a few times, until the spell was "mastered" and we could cast it with a click? I hate mini-games, generally when they interrupt your play, so I'm not thinking anything like that. The only thing that came to mind was tracing out a "rune" shape with the mouse before you hit the cast button. With a controller joystick, it could be a combo pattern where you move the stick around a certain way then hit cast.
I used to play an old N64 snowboarding game that had something like this. In order to do a particular "trick" when you jumped, you had to flail the controller joystick around in this certain pattern, then hit a button or two. It could be done extremely quickly, as you had to in order to make the jump, so I'm not worried about it taking 5 seconds to cast a spell in combat or something. However, it was actually a bit tricky to do, and it took a while to master the jumps so that you could do them with regularity. Even then, there was always a chance of screwing up.
Games these days seem to have lost that sense of skill. There's literally no chance of me screwing anything up. And I suppose... in all honesty that's more the point of this thread. So there ya have it.