Think about it for a second. First off, it does have metaphorical meaning about the bosmer. I refer you to the Monomyth, where upon it says:
the Bosmer, had soiled Time's line by taking Mannish wives.
The took "mannish" wives, not wives who were of the human race. As such, the women are supposed to be tougher and taller.
Second, think of it as what nature will do to animals. All other races have bigger males, smaller females. With the bosmer, it's reversed. Look beyond the superficial.
I'm also trying to dig up MK's reasoning behind the bosmer height and looks.
EDIT: Found it, question was http://www.gamesas.com/index.php?/topic/1063413-why-are-bosmer-men-so-short/page__p__15459847__hl__bosmer__fromsearch__1entry15459847
Short (snicker) answer: me.
Longer answer (somewhat unsatisfying without a discussion): it was my commentary on Men versus Women in general, as encapsulated in one of the single, isolated cultures that I could use to express it.