Why did they betray their fellow men, the Bretons and the Redguard, and decided to side with Dark Elves instead?

Who said it's a betrayal?

The Nords and my Dunmer people have had our little wars back and forth over the millennia, and despite our differences our two races highly respect each other as Warriors. It only makes sense that we'd form an alliance, a Pact as it were, to utilize the strength of both great races. The Nords battle ferocity and tenacity mixed with our Dunmeri mastery of Destruction and Conjuration magick mixed with our skill of blade and bow.

It's possibly the perfect joining of two races.

Politics, mostly. You could also ask why the Bretons and the Redguards betrayed the Nords.

Or you could ask why would the invade even though they are all humans?

I think there are two reasons way they have made the factions like they have. One is that they can make all the factions territory hang together so Temriel becomes a pie with three equal peaces. This so you don't have to cross enemy territory to get to your friends territory. The second one I think is that they tried to pair up popular races with less popular races to some what try to make each faction some what similar in appeal.

Why would race determine everything? Identity is about more than genes, as the real world shows. Culture, history, language, politics, geography, religion... none of which maps onto race in a simple way.

The true Sons of Man, joining with mer and raising blades against their fellow men!

What shame, what dishonor!

Ysgrammor would be turning in his grave!

..Though, Dark Elves and Argonians.. with their centuries of slavery... Is also hard to understand...

If that is so, then let me take my Nord self to the covenant!

I shall show those betrayers of mankind how a real Nord fights like!

The way the game plays it off is kind of an agreement of necessity for all the alliances. For the Ebonheart Pact, they had to group up to defend themselves from the Akaviri, if I recall right. So, they stuck together when the other two factions rose up. For the Daggerfall Covenant, I'd guess the orcs are siding with the Redguards and Bretons because they're the nearest force, and the Bretons and Redguards might finally leave them alone. The Aldmeri Dominion is the easiest to understand, as the high elves and wood elves are fairly close allies, and the khajiit are one of the oldest races, so the elves can get along with them.

As for the Nords siding with the Argonians and dark elves rather than their human allies, it's likely because the Pact had already been formed when Tamriel got turned upside-down. Easier to stick with a faction you're already part of then abandon it and hope another takes you.

In another topic asking why High Elves and Dark Elves are not in the fraction, i tried to answer that question.