Zombies. Seriously, the sound they make is just creepy and a little sickening. Sure, they move faster than you'd think, hit hard, take a bit much to kill... but it's mostly the damn moan that gets me.
As far as the annoying factor, anything with paralyze. It always takes me 3-4 seconds of getting whacked on to realize why I'm not moving anymore. And harpies. Ugly-ass things, and that "caw!" is grating. I once had a harpy fall into the void--which would be fine, things usually die when that happens, but the witch could fly around out there. So that meant that I was followed around and cawed at for most of the dungeon by this stupid birdwoman who I couldn't kill. Eventually tossed myself into the void just so I could shoot her feathered butt with more arrows than it actually required.