There are Khajiit everywhere, but everyone else is everywhere, too. And the Khajiit really are more organized than you're giving them credit for.
Combat wise, we haven't even gotten into the non-humanoid breeds, which are supposed to vary in size and appearance from being able to be mistaken for house cats to massive tigers that humanoid Khajiit can ride, while every variety of Khajiit is fully sapient. The small breeds would be nearly undetectable as scouts and are supposed to be talented in magic. Combined with heavily armored intelligent tigers and more conventional infantry, Khajiit armies would be diverse, suited to nearly any combat scenario, and would have an excellent intelligence network.
On top of that, the Khajiit and Bosmer have been fighting each other off and on for ages, so both sides have some experience battling the other. Khajiit invade Valenwood, Bosmer invade Elsweyr, and neither side ever gains much ground.