In Ancient China, a Prince took a force of around 30 to 40,000 troops into a city. The forces of the city numbered around 5 to 10 thousand, and were able to fend off the prince. Never back down just because they have more forces. Unless they have superior weapons, magic, and tactical ability, then yes, back down.
Yes, my guys are 500 bare-chested retards with steel cutlasses, and 100 guys that are pretty good. Everybody else? 10,000-60,000 troops. They could overwhelm me even if they were peasants with rusty iron daggers! But noooo, I'm either facing the most naturally talented warriors in Tamriel, or Tiger people on steroids (thanks, PFA!). But I probably won't fight Khajiits. Too far away.