Why do different races of elf have different colour skin and slight hight differences if they all descended from the Aldmer and simply moved over into different parts of Tamriel. For example the Bosmer moved into Valenwood and now they have darker skin. Is there some sort of Elder Scrolls evolution I don't know about or am I missing something?

Ideological differences becoming reality.

It's mainly gods changing the races. Chimer, as far as I know, looked like Aldmer but just had different ideology. Then Azura changed them into the Dunmer. The Bosmer were changed by the Green Pact, or rather, stabilized by it as before they switched forms constantly. Also aren't Bosmer not from the Aldmer, but just some different Elhfoney?

Also, the Bosmer don't have dark skin. That's the Dunmer, which means "Dark Elf"

A lot of this can be answered by the UESP, check there.

The Orsimer, Dunmer, and Bosmer all had their physical appearances changed due to additional outside forces; respectively, they are Boethiah, Azura, and Y'ffre. It's also implied that shortly after the Creation event, physical appearance was still in a state of relatively high flux, and the Altmer worked especially hard to retain their Aldmeri appearances, more-so than the other Merrish races.

Its worth noting that the various Aldmeri exodus's take place during a period in which various mythopoeic forces could easily mold these creatures prior to the world itself solidifying. The Bosmer are unique in that they formed a pact with Y'ffre way back when having physical shapes wasn't exactly a thing, to the point that concept art in Morrowind had the male Bosmer looking particularly impish, with horns and all. Actually think TES:O features some of that, but don't quote me on it.

The Falmer are a bit of a different case given we don't know a whole lot about them, other then that they're faith is (seemingly) congruant with the Aldmeri pantheon, with temples devoted to Trinimac still being a thing. The rest (Chimer, Alyeids) retain some resemblance to the Altmer with perhaps a different golden hue, or were flat out transformed by several outside sources beyond anyone's control. Maomer are also special, but they don't even live on Tamriel.

Falmer are an example of a Merish race that had an extreme variance in Coloration from the Golden Aldmeri Hue, but a Falmer before becoming a Dwemer Slave was proportioned exactly like the Altmer, and may have been taller than the Altmer of The First Era. The Altmer have thousands of Years of modification through Eugenics to maintain the "Aldmer" ideal.

EDIT for clarity

Although the source of these sudden mutations is most certainly divine in origin, I like to think there might be a somewhat mundane explanation as well. Like with the Dunmer, the volcanic environment accompanied by the corruptive influence of the Heart, has changed the entire fauna and flora of Morrowind and especially Vvardenfell drastically. And with Orcs, they could be seen as a sect of Aldmer that were overthrown by their goblin slaves who then [vehked] the remaining survivors.

All of this is baseless speculation and probably has no support in current #CANON. I just like the fact that TES usually has multiple explanations for each major event.

The Bosmer in ESO can have horns as a hair style. A lot of my RP buddies got super excited by it.

A possible explanation: the Falmer skin color in the same as Altmer gold, and only appears to be white due to a layer of magical perma-frost.

This was stated as fact on the UESP until about a year ago, but there's no basis for it, and ESO has now demonstrated that the Chimer had their own skin hue. But even before that, we had every reason to believe the Chimer got their own skin tone. The Anticipations: "The followers of Boethiah [Chimer] and Trinimac [Orsimer] rubbed the soil of Trinimac upon themselves and changed their skins."

It was assumed by much of the fanbase during the Morrowind years that Almalexia and the lighter side of Vivec represented the Chimeri appearance. The idea that their skin tone was the same as the Aldmer emerged because some people compared the color of these Tribunes to the Altmer of Morrowind, shrugged, and decided they were identical. But a close look should establish that the Tribunes' skin tone was noticeably different than Altmer. "Almost identical" does not equal "identical".

Anyway, ESO has apparently portrayed Chimer as a sort of ... I still don't know how to best describe it. Sort of a http://img2.wikia.nocookie.net/__cb20140330033644/elderscrolls/images/6/60/General_Balreth.png, I guess. Somewhere in between the Orsimer and the Dunmer (which kind of makes sense). I say "apparently" because I'm not 100% sure whether the Chimer in ESO are being seen normally. There may be some sort of tint added to the screen during the parts of the quest(s) where they appear (there typically has been during other timey-whimey quests in previous TES games). Maybe an ESO player can clear that up. Anyway, they're different from the Altmer regardless, and it's presumed the Altmer most closely mimic the Aldmer, so it's safe to ditch the idea that Aldmer skin=Chimer skin.