at first i was going to dismiss this as 'it just seems quicker' but then i had a closer look at the animations. There are in fact 6 casting animations, a normal and alternate version for onself, ontouch and ontarget.
Here the thing, the onself alternate (which is casting with right arm, like when shield is up) is 0.1 seconds faster then the normal casting. The same with the alternate ontouch, it is 0.1 seconds faster, it has the one handed touch, like when you hold a shield. Ontarget is the same.
Now i also went deeper, it seems the the casting 'moment' of the animation is the same in both normal/alternate, however because the alternate is faster in those two cases, if you are spamming the magic button then the animation will repeat slightly sooner.
How to stop this? Apart from editing the animations, you cant. Also, if you just time scaled the animation, or adjusted the speed of it via the frequncy, then you would also unsync the casting moment in the animation, so now the first cast would be happen laster on the alternate animations, though then it would only be able to repeat at the same rate as the normal casting.