I have a feeling that you and I... are about to become veeeery close.
My thoery is that it's not so much Fargoth that people find annoying, as Bosmer in general, it's just that Fargoth is the first Bosmer your likely to encounter in the game, indeed, typically the first person you see after exiting the office, so he tends to be the one who stands out most, seeing as pretty much the first thing you'll here after passing through the door that leads to your freedom is that horrible squeaky male Bosmer voice. Aside from that, hating Fargoth has also attained a bit of a memetic status, I think, so even people who have no reason to hate him still say he is annoying because it's the "politically correct" thing to do.
Hating Fargoth is the Politically correct thing to do. Love it!
Personally, I don't consider him worse than any other Bosmer, it's just that Bosmer as a whole are rather hard for me to take seriously thanks to some rather unfortunate design decisions Bethesda has made with them.
I'm pretty sure one of the devs has said that Bosmer are almost an 'inside joke'
I've never found him annoying either. I think most of the insecure adolescents who reviled characters like Fargoth and Crassius Curio have either moved on to other games or have grown up.
Crassius
did creep me out, but he was hilarious!