I'm changing my play style from a pure mage to a sorta nightblade (mage assassin). For this play style I usually cast calm on the enemy before they see me and then sneak up behind them and slit their throught. This time around, though, I was wondering if I could use a paralyze spell instead of a calm spell (either paralyze or ash shell). So my question is, if i cast paralyze on the enemy, will that alert others around him? Will he still scream? (I have quiet casting)

With quiet casting you should be fine.

stay in sneak ie croutch mode, then again depends on your spell radius. Played skyrim, but never dove into. Common thoughts from other rpg's say stay hidden and out of sight, should work. Then again spells are dispersed from your avatar and npc's can see projectile weapons. Best bet if unsure sneak up to an npc which decreases your spell distance, cast so called spell, and then conclude with melee attack. Efffective vs. Efficient.