Yeah, it's one of those universals of comparative anatomy. The reason it breaks down for real-world mammals is that there's a huge selective advantage to a male who can control a number of females, make of that what you will. So we look for the selective advantage in being small. In a culture whose scripture is "The Thousand Ways of Hiding", there just might be an advantage in being especially inconspicuous while hunting, or raiding, or fleeing from the Khajiit, and being small fits the bill. If females, by and large, stay at home, there is no such advantage in small stature for them.
I'm not favorable to applying real world evolution patterns on Nirn - there will be a selection, but the timeframe don't allows it to have much effect. Especially for the long-lived elves. There's simply not enough generations. Nirn's evolution of sentient races is more a result of mythopoetic forces (an awfully rought summary would be : you become what you believe). The culture and beliefs are what shapes the races.
Whith a bosmeri culture that enphasizes sneaking and hiding, the short stature of the bosmer males is fairly logical. If the bosmer myths ar right, there's also the root cause of Yffre's choice. He's the one who shaped the bosmers. It's doesn't give a clue to what was behind his choices, but it explains the results.