So I'm replaying Dawnguard and talking to Gelebor about the Falmer's pact with the Dwemer and he mentions that some Falmer sought alternate alliances and vanished. I think you see where I'm going with this.

If a large sect of Falmer made a pact with Hermy it would explain quite well why the Nords treat him as a demon associated with elves (tried to get them to turn into elves), and even call him by an elven name (Herma Mora which is also what the Bosmer call him).