The Dunmer, who split from the Altmer and were subsequently cursed by Azura are of average height. And because the curse never mentioned getting shorter, I wouldn't think that it was an effect.
Do you understand why the Altmer are taller? It is beacuse the shorter Altmer are exiled/killed/no married and thus only the tall ones pass on their tall traits. Its a common technique in dog breeding.
The Chimer were probebly once tall. However, their numbers were small and thus when a short Chimer was born, he was not exiled/killed but rather married and had children. Thus, as more short Chimer cropped up, the shorter the race as a whole became.
Compare the Nephilim to the story Albides brought up.
The Nephilim weren't actually giants. The original stories on old Hebrew texts state that the Nephilim were not physically any larger, rather that they could fly and generally had superpowers, but were still the same height as a normal human. The giant thing is is misconception stemming from King James's censorship brigade being horrible translators when they were making the king James bible.