This is something of another design philosophy question.
If you tallied up ALL of the NPCs that you encounter in Skyrim, it feels like about half of that entire population are labeled as "Bandits". Economically speaking, anything beyond 20% of the population -- actually closer to 5-10% -- engaged in banditry as a full-time occupation would pretty much guarantee that Commerce would get crippled. The Skyrim Bandit groups number from 3-20, and the merchants that travel between cities would have to guard against the larger number under ALL circumstances. Guards cost money and their wages cut directly into the merchant's profit margin. Worse, it seems that bandits generally would not be content to just collect a "toll" and leave the merchant free to continue on his way. Skyrim bandits seem to generally prefer to _kill_ merchants. Therefore merchants are literally putting their lives on the line just to transport goods from one city to another. And the odds do NOT favor them surviving the trip unmolested.
What margin of Profit would be required to justify taking that kind of risk? Anything less than "fabulously wealthy" wouldn't seem to be worth the risk of making such trips repeatedly. That means that merchants really should be charging more than what the vast majority of Skyrimmers could afford. Insufficient customers at a destination means less motivation for merchants to be merchants. Fewer goods of less variety would inflate prices, leaving locals with hardly anything to survive on. Goods would have to be nearly 100% locally produced, with negligible access to foreign goods. That leads to a stagnant local Economy.
From the bandit side of the equation, just what can they be doing with plundered goods? Wealth must be spent/spendable to actually be worth anything. Sitting on a pile of coins doesn't feed the belly. But if a bandit tries to go into town to sell his ill-gotten gains, he runs the risk of being recognized as a bandit. Incarceration or death would be what he risks. So that leaves the bandits to exchange stolen goods between the members of their own bandit band. Again, a stagnant Economy.
Notice how nearly all communities have guards. If the Economy of a town is threatened by the depredations of nearby bandits, what is it that nearly all communities do? They form posses and go hunt down those bandits. And then usually execute them in such a gory fashion as to "send a message" to any other would-be bandits. That tends to push bandit activities out to the hinterlands, far, far from any community boasting a guard contingent. This posse element is something that is clearly missing in Skyrim.
The balance between risk to merchants from bandits and risk to bandits from guard patrols is what has been overlooked. Both merchants and bandits MUST perceive an adequate reward to warrant the risk that they take by pursuing their chosen professions. Historically, that balance is what tended to hold down the percentage of the population that engage in banditry.