You can argue that every time you cast a "fortify speed" spell (or drink a potion), time either slows up for you or speeds up for everybody else. Slow works in the reverse direction. So there lies the connection between Magicka and Time.
I agree, your idea is pluasible. However, I feel inclined to believe that fortify speed simply makes your pc or npc move faster rather than altering time. On the other side I would imagine Slow would just cause your body to move at a slower pace. IMO, its physical alterations, rather than the manipulation of time.