When you walk into a bandit lair they should have a couple of guys sat drinking or whatever (not just standing around), guarding the entrance.
When you approach they should ask your business. You should be able to say your from the Imperial Legion, Fighters Guild, rival bandit faction,etc. At this point they would attack you if you are from an enemy faction they should a) attack you if you are a low level or if they think they can beat you B) ask you to leave if you are too strong c) Invite you in if you say you are a treasure hunter or adventureur etc. They will then ambush you as you are lead into their base.
Similarly any bandits you meet outside a dungeon should also not simply try to kill you. They should demand money or if you are really strong possibly not even try to attack you just beg for your forgiveness. of course if you were really strong and famous and a large group of bandits should attack you to try and gain fame and your wealth.
Of course there could still be mad bandit factions who are too crazy and stupid and will just kill you on site.
I think this would really add to immersion. Discuss