Must Read for Anyone with Connection Problems

Post » Fri Dec 31, 2010 6:20 am

TL;DR (This will be quite a lengthy discussion)
To fix my problems I
- Switched router from a BT Home Hub to a Netgear DG834N
- Upgraded the firmware from 1.01.06 to 1.02.15
- Disabled DoS and Port Scan protection
You might try
- Forwarding Ports
- Disabling Firewall
- Putting computer in DMZ
- Disabling DoS and Port Scan protection
- Upgrading router
- Changing QoS settings
- Changing ISP
- Setting up a VPN or Proxy
- Replacing router/network card (look for router with decent QoS features)
- Updating network card drivers
- Changing send/receive buffers on network card to 512
- Waiting for patch ;/

I have recently been able to connect to my first game of Crysis 2 without being kicked. I'd like to share my experience and knowledge in coming to the solution.

The first thing you must know is how exactly Crysis 2 connects to a server, and what might go wrong. Well the internet is in a basic sense just a load of copper and fibre-optic cables, but in order to communicate with anyone else there need to be protocols in place such that a signal travelling along a cable can be interpreted.

The first "layer" of these protocols are how devices (nodes) physically generate/extract a bitstream and how they segment it. This describes physical behaviour when two devices are joined with a wire (or wirelessly). The segmented pieces are called "packets" and each one contains information relating to each successive layer of protocols. Such protocols include ethernet (likely the way your computer is connected to your router) and ADSL (likely the way your modem is connected to the local exchange).

So what could go wrong in this layer? Well physically there are limits as to the frequency of signals that can be reliably transmitted along a wire. This means a rate of transmission must be negotiated between nodes at either end. Errors will invariably occur from time to time and some packets will never reach their destination. Wires also have a physical length which takes time to traverse.

One would think that these issues would not be a problem if connecting to another online game works fine. However the speed of a connection between two nodes is variable - certain types of packets are sent with different "Quality of Service" (QoS) parameters. This is required because different nodes have different available resources. One node may not have the processing power to deal with a large number of small packets in real time. Another node may be handling connections from many other devices and can only guarantee real-time QoS for a proportion of packets due to limited bandwidth in its connection with other nodes. It is likely the case that Crysis 2 packets are sometimes held up in a node and sent with a non-realtime QoS setting, meaning their delivery is sporadic.

A word on wireless connections - wireless connections are often cited as the reason players lag online. This has spread far and wide and only has basis in older wireless connections which were less speedy and reliable than ones today. So long as you have good signal strength, and your connection doesn't drop, there is no reason a wireless signal should be any different from one sent along a wire. Wireless signals have more than enough bandwidth to exceed most users' connection speed and as such should not be a bottleneck. It is arguable that some latency could be introduced through the hardware however the same could be said for a wired connection.

The second layer is the internet layer. Once packets have been received and turned into digital data that can be processed, some method needs to be used to specify the destination of a packet, and some method needs to be used to direct the packet to its destination. The Internet Protocol (version 4 & 6) performs such a function; the basic location descriptor is the IP Address, but it is far from simple. Each wire connects only two nodes, and a chain of nodes connects your computer to a Crysis 2 game server. At each stage, a node must flip a switch (an automated telephone switchboard) in order to link two nodes and send the packets in the right direction. The speed and bandwidth of all these nodes is a contributing factor to latency - and QoS prioritisation, as well as physical wire length, is the contributing factor to the average latency between nodes.

Difficulty arises because each node may assign for itself a local address space - that is, subordinate nodes are referenced by an IP address known only to this node and not to anyone externally. This means (if you are on a local network) there is no way to send packets directly to you, only to a node with a public IP address (in most cases, your modem/router, but it is possible to have subnetworks). This is a problem that must be resolved with protocols in the next layer, along with the capabilities of the node with a public IP. Unfortunately it is this problem which is probably to blame for a good many connection problems in many games.

The third layer is known as the transport layer. There are two main protocols in use here. The first is TCP, which is generally used for websites and other non-realtime communications, because it requires an acknowledgement for every message sent such that it can auto-correct errors caused by lost packets. This generates a lot of overhead and as such is not very suitable for games, although some games do use TCP packets. TCP introduces the concept of "ports" - basically just a source/destination identifier which allows numerous programs on the same computer to run concurrently without interfering with each other's traffic. They are distinct from UDP ports, but work in much the same way.

The second protocol is UDP. UDP provides relatively little overhead compared to TCP, and as such is suitable for real-time traffic such as online games. UDP is what Crysis 2 uses. As previously mentioned, UDP uses ports and this is one way to overcome the local address space problem discussed earlier. One can open a connection with an external IP via UDP. No problems here. Packets cannot be sent the other way ordinarily, however when a UDP packet is sent from a certain port on your local PC, the router (or other node presiding over your local network) will open up a "session" on that port, such that any UDP packets received on that port will be automatically routed to your local machine. This obviously has problems. For instance, this does not work if BOTH endpoints are in local networks. Furthermore, all UDP ports are effectively shared between all computers in the local network, and there will be conflicts that need to be resolved if two people wish to play at the same time.

A technique called "hole-punching" may be used to overcome the issue where both endpoints are in a local network. This is most commonly seen in peer-to-peer applications. Each router is "tricked" into thinking its local computer initiated the session, and so will route all packets on that port to the respective local machine. However, hole-punching is not supported by all routers. In this case, one needs to manually tell the router to route all UDP traffic through a particular port to a particular machine. So long as the application always uses the same port, the traffic should be routed correctly. However, for most routers this is not at all an issue as opening connections with remote machines via hole-punching is fully-supported. Furthermore so long as SOME traffic comes to you via a particular port that means it is capable of being opened, and as such opening the port manually will do nothing. Port forwarding is often cited as the thing to do in order to get an online game working; however in my experience it is rarely the problem since it corrects a single, rare issue among the multitude of things that could go wrong.

For those for whom port forwarding could be the solution, know that Crysis 2, as well as using its advertised port, 64100, for game data, also uses port 64228 when connecting to a server, which is undocumented. This may vary as well.

The final layer is the layer that Crytek have created bespoke for the game. Players' movements and actions will be encapsulated in their format and transmitted within the UDP packet. This may include cryptography to deter hackers/cheaters, but ultimately ANY game is susceptible to being reverse-engineered as the code is stored on your local machine, and can be anolysed. The same code can then be run to encrypt bogus game data whereby the player is given powers not usually afforded to him. A resource-heavy solution could detect many hacks by comparing the result of the players' key presses to his on-screen movement, and disallowing abnormal behaviour such as moving further than is physically possible in 1 second, however this would be difficult to implement, and wouldn't prevent cheats such as wallhacks and aimbots. A more common method is to use a program like PunkBuster which prevents certain programs from running on your local machine, similar to anti-virus or anti-spyware software. It can be circumvented, but any popularised hack that any script kiddie could run would soon be blocked by PunkBuster, who take pressure off the developer to update their game to cope with cheaters. Crysis 1 used PunkBuster, however it has not been used for the sequel. I think their current system is proprietary or may be provided by Gamespy.

Problems that could cause lag in this layer are fewer than one might realise. It is tempting to blame Crytek for every connection failure but in reality, over a local area network where no packets are lost or delayed, I would expect the game to work every time with very little latency. Even things like the occasional lost or delayed packet should not be a problem, as in the structure of any online game it is necessary to account for possible lost packets or packets that arrive in the wrong order. Of course there may be a few bugs, but I would not expect them in any game to cause regular connection drops.

The main problems there could be are if Crytek make their packets too large, sending too much data that cannot be handled in real-time by the nodes in-between your computer and the server. Or if they split up the data into tiny chunks and the overhead from all the other protocols makes the packets bloated. These may be able to be improved in Crysis 2 but it does not appear to me that Crytek are doing either of these things.


All this sounds very nice in principle, but when people want to abuse the internet it is very easy for them to do so. It costs nothing to spam an address to death with countless packets in a "Denial of Service" (DoS) attack. It costs nothing to scan many computers to see if they are awaiting a connection on a certain port, and then exploit security vulnerabilities in programs known to use those ports to insert spyware or viruses.

For this reason many nodes along a path will perform rudimentary tests to determine if certain traffic is a DoS attack or a port scan attack. Of course there is no logical way to determine if they are - it may be that many people actually WANT to use a website at the same time and that causes an unintentional DoS, it may be that an application is looking for a port to connect to on a remote machine, and as such must look for one which is awaiting a connection - so of course this generates false positives from time to time, but it is necessary in order to weed out basic attacks. Other measures include firewalls which flat-out block connections to certain ports from the outside, and only allow connections initiated from within the firewall. Application-level firewalls such as Windows Firewall can block the online operation of entire programs unless they are exempted. Note that firewalls are a test performed in addition to DoS and Port Scan prevention, and as such disabling the firewall (or even I think putting the computer in DMZ) does NOT guarantee that all packets will be forwarded correctly.

What all this means is that a developer has to try to work around ALL possible devices that could detect it as a threat. This is very hard to do when you are sending lots of packets every second to different ports. I have seen Crysis 2 be misidentified as a UDP Flood as well as a UDP Port Scan attack. Situations like these explain why one can see a server, and superficially connect to it, but not actually join before being kicked. At first the packets are being forwarded correctly, but after a certain volume of packets on different ports (Crysis 2 uses different ports for in-game and when connecting) all subsequent packets are blocked.

The solution to such problems is to disable DoS and Port Scan protection on your router if it supports the option, though it is always possible that some other node is blocking the traffic, or at least delaying it. If this is the case, there is nothing you can do - it's out of your control. ISPs are known to throttle traffic on certain ports and reduce the effective line speed. ISPs also use Deep Packet Inspection which actually anolyses the data Crysis 2 transmits within the packets. There is every chance that a Deep Packet Inspection could throw up a false positive for unwanted/restricted traffic with a Crysis 2 packet. Claiming that DPI does not ever throw up false positives would basically be claiming that it either does nothing, or that it is capable of breaking every encryption scheme ever invented.


By now if you've been paying attention you should have a good idea of what could go wrong when transmitting data between a player and a game server. Ultimately there is likely nothing so wrong with Crysis 2's code that would prevent a player joining a game at all, and so the problem has to lie on one of the nodes leading from your computer to the server.

The candidate problems:
1) Router fails to implement UDP hole-punching correctly
2) Router blocks traffic due to misinterpreting game data as a UDP Port Scan or DoS attack
3) Intermediate node blocks traffic due to misinterpreting game data as a UDP Port Scan or DoS attack
4) QoS prioritisation is set wrong on Router
5) QoS prioritisation is not implemented correctly on intermediate node
6) ISP throttles traffic due to misinterpreting game data as P2P filesharing or other disallowed/illegal activity
7) Node or physical layer is unable to cope with volume of game data
8) Crytek's buggy netcode

Note that this is not a complete list of things that could go wrong, but looking at the entire network topology these are the things that strike me as likely.

Potential solutions:
1) Forward all ports used by Crysis 2 that are not opening correctly. My list:
UDP ports 64100, 64228, 27900, 27901, 29910
TCP ports 6667, 29900, 29901, 28910
Alternatively, disable your firewall completely, or put your computer in a DMZ, but only to test if it works.
2) Disable Port Scan and DoS protection on router. This was one of the things I had to do to get multiplayer working in the end.
3) Out of your control. Changing ISP may fix this issue. A more expensive solution would be to purchase a VPN. This would actually increase general latency for all servers but if you can't connect at all this is better. A proxy might also work.
4) See what QoS options there are on your router, and on your home PC. There are usually none. For me, upgrading the router's firmware added new QoS features which was the final kick that got the multiplayer working. There is also custom firmware available that could greatly enhance performance. http://www.dd-wrt.com/site/index The final option would be to test another router if you have one, or to go out and buy a new one. I switched from my BT Home Hub 2.0 to a Netgear router (which was actually older) and once the firmware was upgraded it did the trick.
5) Out of your control. Changing ISP may fix this issue. A more expensive solution would be to purchase a VPN. This would actually increase general latency for all servers but if you can't connect at all this is better. A proxy might also work.
6) Again, same as 3 and 5, out of your control. This is one instance where changing ISP would definitely have an effect. An ISP such as Plusnet (in the UK) provide a "Plusnet Pro" service which does not throttle traffic. They also detail to what extent certain "types" of traffic are restricted. Of course you have now seen how the entire system works, so you know there is no such thing as a "type" of traffic, but it sounds viable to the average consumer. In reality they will use Deep Packet Inspection and give false positives, and throttle "allowed" traffic.
7) If the problem is with your router or network card, you will have to have them replaced. Upgrading the firmware or drivers MAY work, but I wouldn't get your hopes up. Changing the number of send and receive buffers up to 512 on your network card may work also, but again I wouldn't get your hopes up for these minor changes. If on the other hand the problem is with an intermediate node, it's out of your control again and the only solution this time is to change ISP. Proxies and VPNs probably won't work to solve this one.
8) Now I know for a fact that it IS possible to connect to and play games with little lag in Crysis 2. So this problem is not applicable for those who cannot join a game. But for those who get kicked now and again, or experience glitches, know that there is little you can do other than wait for a patch to iron out the kinks. If the problem is that Crysis 2 is sending out packets too frequently, one can write a program to group these small packets into larger ones to reduce overhead. This method was used by someone to reduce lag in Minecraft and works very well. However, this introduces more latency and I honestly do not think this is a problem in Crysis 2.

To fix my problems I
- Switched router from a BT Home Hub to a Netgear DG834N
- Upgraded the firmware from 1.01.06 to 1.02.15
- Disabled DoS and Port Scan protection

Among the most notable features in the updated firmware were these:

Added support for Wi-Fi Multimedia Quality of Service (WMM QoS) to prioritize wireless voice and video traffic. WMM QoS is a feature that provides prioritization of wireless data packets from different applications based on four access categories: voice, video, best effort, and background. For an application to receive the benefits of WMM QoS, both it and the client running that application must be WMM-enabled. Legacy applications that do not support WMM, and applications that do not require QoS, are assigned to the best-effort category, which receives a lower priority than voice and video. WMM QoS is enabled automatically.

Added support for Link Layer Topology Discovery (LLTD) which is a licensed data link layer protocol for network topology discovery and quality of service diagnostics, developed by Microsoft as part of their Windows Rally set of technologies. The LLTD protocol operates over both wired as well as wireless networks.

This tells me QoS played a big part in allowing me to finally connect to Crysis 2 multiplayer.

Thanks for reading. I hope you too can sort our your connection issues.
User avatar
RAww DInsaww
 
Posts: 3439
Joined: Sun Feb 25, 2007 5:47 pm

Post » Fri Dec 31, 2010 6:45 am

Congratulations, you just wrote a thesis. =P
User avatar
Prohibited
 
Posts: 3293
Joined: Tue Jun 12, 2007 6:13 am

Post » Fri Dec 31, 2010 9:59 am

The quote box at the start sums evrything up nicely but the detail is nice for those tech nerds among us who like to understand why it works, nice post :)
User avatar
Robert Jr
 
Posts: 3447
Joined: Fri Nov 23, 2007 7:49 pm

Post » Fri Dec 31, 2010 3:02 am

Congratulations, you just wrote a thesis. =P


He wants to be employed by Crytek =3
User avatar
Scotties Hottie
 
Posts: 3406
Joined: Thu Jun 08, 2006 1:40 am

Post » Thu Dec 30, 2010 9:52 pm

+1

comprehensive and interesting post. Even tho it might help them, I wouldn't expect many people to read it, therefor its just to indept. Something like this should make its way into a wiki or something like this :)
User avatar
FLYBOYLEAK
 
Posts: 3440
Joined: Tue Oct 30, 2007 6:41 am

Post » Thu Dec 30, 2010 10:08 pm

Even I don't have connection problems in C2, I've read this with high interest.

Good structured text and well articulated.

/+1
User avatar
emily grieve
 
Posts: 3408
Joined: Thu Jun 22, 2006 11:55 pm

Post » Thu Dec 30, 2010 11:10 pm

Buy a new computer.
User avatar
ILy- Forver
 
Posts: 3459
Joined: Sun Feb 04, 2007 3:18 am

Post » Fri Dec 31, 2010 4:35 am

+1

comprehensive and interesting post. Even tho it might help them, I wouldn't expect many people to read it, therefor its just to indept. Something like this should make its way into a wiki or something like this :)

For myself I found it necessary to familiarise myself with all the technologies, so that I'm not stuck wondering what exactly "port forwarding" is doing, and if there are any alternatives. It's best if you know what could go wrong, so you at least know what kind of things to try. I know there must be other people willing to try anything to get this game working, so I hope some read it and it helps them at least. :)
User avatar
Project
 
Posts: 3490
Joined: Fri May 04, 2007 7:58 am

Post » Fri Dec 31, 2010 9:42 am

This thread wins.
User avatar
Emerald Dreams
 
Posts: 3376
Joined: Sun Jan 07, 2007 2:52 pm

Post » Fri Dec 31, 2010 1:31 am

Thank you for taking the time to go thru all of that. I for one really appreciate the detail.

In my case, I have been able to connect to servers eventually, but only after several attempts each time. It is rare that I select a server and connect on the first try.

I also have intermittent disconnects from servers since the game came out. Some nights worse than others. Ranking up in this game has been a chore with all of the "re-ranking" Thousands of points lost to mid game disconnects...

***UPDATE***
I just wanted to check back and let people know that I went ahead and forwarded the aforementioned ports on my router as suggested by the OP... seems to have improved. The in-game Browser is now snappy and normal, selecting a server snaps right into them... on the first attempt! So far, so good.

***NEW UPDATE*** well... all good things... you know how it goes.

I thought things were better, but after a few more days, I am back to where I started. Slow, hanging server menu, random disconnects in game, and general frustration at losing my precious XP everytime it happens. dang, thought I was on to something with the port forwarding.

I have resorted to leaving servers every 10 mintues or so to try and lock and save the XP. I notice many, many players are doing the same. I wonder what all of that connecting and disconnecting to servers is doing to the whole network.

Anyhow, I have more connection challenges in this game than all others combined. What has Crytek done so drastically different than every other game out there? Online population is hangin in there, but frustration levels are high.
User avatar
Jessica Raven
 
Posts: 3409
Joined: Thu Dec 21, 2006 4:33 am


Return to Crysis