Basically, they trigger when you inflict the killing blow on the last enemy in the imediate area thats a threat (ie: Showing on the rader as a red dot)
Each one depends on certain circumstances, such as position to enemy and power attack used.
Example: A forward power attack with a two handed sword will do the impale animation. The same with a warhammer or axe will trigger the overhand smash.
Some such as the decapitations, can be unlocked via perks.
edit: It's sometimes difficult to trigger them when fighting mages. Especially ones that keep summoning things or raising zombies.
Also, some enemies such as Dragons may also have unique kill animations.