That's what confused me about the Forsworn. To me it seems that it goes Nord > Imperial > Breton in terms chronological appearance.
The Nords settled Skyrim after Arriving from Atmora (supposedly before any other men were in Tamriel.) Then how did Bretons settle it first and why are they so... Tribal. They act like cavemen.
Don't think of the first inhabitants as.....Nords.
Ok here is exactly what happened, roughly ^^.
The continent of Atmora is where men are. The continent of Tamriel is where the elves reside. The men on Atmora are pitched in a civil war, not sure when it started or what it is about. Between the years ME1000 - 800 men start to leave Atmora and settle on Tamriel. They, presumably, land along the northern coasts of Tamriel.
These men live with the elves. They don't try to stake claim on any land they simply assimilate with the elves.
Now I believe that Ysgramor lands on Tamriel around the year ME800, not entirely sure but I heard that number somewhere. (BTW the years of ME are the Merethic era and can be thought of as BC in our time. The years count down so ME1000 was before ME800.)
Ysgramor and his family and presumably others land on Tamriel. They differ from the earlier men who lived peacefully with the elves without staking claim to their own land. Ysgramor claims a bit of land and an all human settlement is constructed, around the location of Winterhold.
This is how the first men arrived on Tamriel. Now to see the separation of culture, ergo the different races of men, you have to look further into history at the split of the cultures. The foresworn would probably be an easy one to figure out. They are a tribal, barbaric group of mostly bretons. They most likely lived in the Reach undisturbed for some time and never advanced in their beliefs. They also became a group that people with nothing to live for would join, seen in the quest where you get to talk to some foresworn members.
The Imperials separated, I believe, when Tiber Spetim began an empire, not the first empire. So those are the 3 races of men; The Nords, the original inhabitants of Tamriel who stayed within the borders of Skyrim keeping to their strong bodies and gruff spirits; The Imperials who separated from their more barbaric descendents to become a more civilized people, living in castles and more modern homes; and The Bretons, who became known as the most powerful human casters of Tamriel, though their ancestors gained this at a price. This is of course excluding the Ra-Gada or Redguards because they are from a different continent all together.