Happens quite often. Is has nothing to do with your connection. It's caused when people who are hosting the game leaves using the game exit menu rather than dashboarding. This is why most migrating host screens ends up with you all in the lobby again. If you get thrown into a match by yourself, just quit it and search for a new section.
Hopefully, Crytek will have this problem fixed in Crysis 3.