Didn't the sword singers come to hammerfell before the rest of the redguards?

I got that impressiong afther reading Redguards, their history and their heros
http://www.imperial-library.info/dfbooks/b004_redguard.shtml

No...the Forebears landed on Hammerfell before the Crowns arrived. Sword-singers can be from either sect.

In my opinion, note that this is base only very loosly on supported material, sword singers figured out how to take their soul (their magica), and to shape in like a sword outside their bodies. This would be a sword made of raw magic, and the ability wouldn't couldn't be destinguished by political lines.

Those are Ansei you are thinking of, the Shehai they possess is literally "Spirit/Soul/Essence/Life-force Sword", and yeah they too would traverse political lines, few as they may be. To add to my "lazy" above answer: the Forebears landed first, followed up by the Crowns (Na-Totambu). The entire event is called Ra'Gada (Warrior-wave), there were singers among both sects.

As a footnote: aw nevermind