Quite recently, I was playing on my Mage High Elf and killing some other mages living in some cave. Then I noticed something, my magicka is going down, which means it's unlimited!
Not sure what the heck happened but It's the best glttch I ever stumbled upon.

Any ideas?

is it still happening? because if it did .....pretty much broke the game lol u will just own everyone and everything.

If you do http://elderscrolls.wikia.com/wiki/Out_of_Balance the reward can be buggy and result in essentially unlimited magicka for an extended period.

Could you have enchanted gear perhaps that reduces cost of casting? For example if you enchant something with fortify destruction it doesn't actually raise your skill it lowers casting cost.

If it really is a bug then I suggest having fun with it, if you get sick perhaps make a new char that might not have the bug, up to you though.