They came from Yokuda, as previously mentioned.
Yokuda is a continent west of Tamriel. Redguard is a corruption of the word/political party(?) Ra'Gada.
The Raga left Yokuda either due to a bad political situation, or because their Ansei (magic samurai basically) blew it up with nuclear fission.
Ra'Gada was the original warrior wave who left first and became the forebears, and everyone else who took longer making the trip became known as the crowns. Since the originals were Ra Gada, every Yokudan immigrant became known as Redguard, the corruption of Ra'Gada.