I do not think the people in that lobby are leaveing just b/c you have enterd,infact they have not left at all.
What is happening is either two things.
1 You have lost conection to the server (host)
2 The lobby migrated and you or they lost conection to the server.....And were separated onto different servers or lobbys.
BUT I may be wrong?......