You dont get a bounty if you wait until he is moving towards her with the knife drawn, done it 3 times now, lol.
Same here, if you kill the guy, while he's sneaking towards his victim with the knife drawn, you will not get a bounty. I never got a bounty for killing him.
Another thing, you can even prevent the NEW murder that happens when you get the court wizard thrown into jail. After you follow up on the clues and your journal tells you to talk to Jorleif, just go and talk to the Court Wizard!

@ uhappynow: LOLOLOL Re-live his life! I like it! hahahaha