Hi,
One Reason could be that you cannot "connect" with the Player who hosts this session. So if the current Hostplayer leaves you get d/c from the Server.
Before you join a server, select it in the server Browser with one click and press Info on Button of the screen. If you can see (Details of the Server) Players you should can connect and Play. What if you join an empty server and wait for 3 players more? Ever tried?