There are quite a lot of animal species where the female is bigger/stronger than the male. So....why not?
Yes, it's not as uncommon as many people might think. My personal theory is that they evolved that way because since the females would be the ones giving birth and caring for young, they would need to be larger and stronger so they could protect their children as well as themselves. This wouldn't be as nessassary for Bosmers who are "civilized" but in their more primitive beginnings, I can see why the females would be taller.
With many species, the parent that has the role of care-giver/protector of the young (and sometimes that is the male, other times the female) is usually larger.