What's the deal with them? I thought Falmer hated all non-Falmer. Were the servants kidnapped as children? What's going on with the humans living alongside the psychotic, twisted elves we know and hate?

There is an old nordic tale about how the Falmer would appear during blizzards and the night and capture children from their homes, so I'm guessing it was really that specific tribe of falmer that did it.

While the Falmer naturally do hate everything, Gelebor makes mention of a rise in their intelligence.

It's highly possible they have become smart enough to realize the advantages of having slaves, nothing beyond a primitive understanding mind-you, but still, some understanding.