A lot of them require you to charge it. Try holding down the button.
That's what I've been doing, but it doesn't seem to work. I seem to recall holding the button down for at least 10 seconds, but the spell still doesn't go off.
Hmm, just thought of something. Typically my character has a weapon in the right hand, and a spell in the left. Thus I'm used to using the right mouse button for casting spells. Will casting the scroll work with the right mouse button. IIRC, the scroll effect seems to fill both hands.