If you're level 1-5 and have a decent build, they're not that bad to fight and you'll get a lot of experience from them. I've found myself desperate for enemies to level my skills on in my recent playtime of the game, and cliff racers are good for that at low levels.
At higher levels, I usually kill them in one or two hits, so I don't see the annoyance?
Maybe it's just hearing that stupid sound they make over and over again?
Their collision/pathing is pretty bad too which often makes them entertaining to encounter. For example, I encountered one near Suran and it got stuck on the wall while trying to follow me into the town.