A lone person is much more likely to go unnoticed for longer, thus may be able to perceive the danger. In a carriage you'd create a lot of noise alerting any potential ambushers of your arrival. And you'd be unable to hear anything going on ahead of you. Also, fast travel doesn't necessarily dictate that you must walk along the road, while fast travel using carriages would.
Well, the carriage is making more noise, but also going much, much faster, being almost impossible to stop on-foot. Unless the Bandits are setting up a whole bunch of very effective roadblocks, it would be harder to get the carriage IMO.
And when you map travel, you aren't necessarily on the main roads, your right. but when i'm walking through a forest or something in OB, i feel like i get ambushed by more random enemies than if i stay on the road. The road had more Bandits that tried to rob me though, but the forest had more enemies total IMO. having the road be safer, and the off-road be more dangerous is kind of a common dynamic in RPGs too.