Problem Definition
The customers were experiencing video disconnects after 60 minutes whenever the WANScalers were partnered between any two of their three sites.
Environment
The customers had a network design where three offices connected through a Multiprotocol Label Switching (MPLS) cloud.
WANScalers were installed at each of the three locations within their production environment. The headquarters site used a WANScaler 8520. One remote site used a WANScaler 8520 and the other used a WANScaler 8320.
They were running inline, softboost/partial bandwidth, and using firmware version 4.1.5. Each of the 8520 WANScalers were licensed for 10 megabytes per second (Mbps) with Current Throughput set at 4.4 Mbps (approximately 95 percent of each T3 link). The WANScaler 8320 was licensed at 2 Mbps with the Current Throughput set at 1.5 Mbps (approximately 95 percent of the T1 link).
A diagram showing the network layout for the headquarters and remote sites is shown below:
Troubleshooting Methodology
The customers initially attempted some troubleshooting before calling Citrix Technical Support. They reported that while the WANScalers were enabled, their video traffic would be disconnected after 60 minutes. When they disabled a WANScaler between two of the locations, they found that video traffic was not interrupted.
The customers were also in contact with Polycom. We verified that Polycom video traffic is sent over User Datagram Protocol (UDP) as well as TCP ports 1700-1749 and that there is no 60-minute timeout setting.
After we were contacted, we met with the customers through a GoToMeeting and conference call. We found that a coworker had defined a Video service class policy on two of the three WANScalers, but this had no effect on the issue at hand.
Knowing that UDP traffic passes through the WANScaler untouched and that this particular video traffic utilizes a TCP port in the 1700-1749 range for initial call setup and call clearing, we felt a change to the service class policies was in order.
We first looked at the existing Video service class policy defined on two of the three WANScalers. We explained to the customers that the first step was to disable flow control and compression for this policy.
We then moved that policy to the top of the list, ahead of all the other service classes. This ensured that the Video service class policy applied its rules before any other policy, which would avoid conflicts with rules of the other defined service classes. In addition to the steps above, we disabled flow control for the PPTP service class policy because it was defined as using port 1723.
Note: With Flow Control disabled in these service class policies, compression is not available and therefore is also disabled.
These steps were completed on each of the three WANScaler appliances in the customers' environment.
The customers shortly thereafter reported that their video application was no longer being disconnected after 60 minutes as previously experienced.
Resolution
After further review of this issue, other alternatives could have also been employed to correct the video disconnect issue.
It was later identified that the video application sent out a keep-alive packet every two hours by default for the TCP control port. WANScalers have a default 60-minute (3,600 second) connection timeout setting (found in the UI under Configure > Tuning). Given this information, the TCP control port would be disconnected every 60 minutes by the WANScaler, which is the root cause of the original issue.
Understanding this, an option to address the problem would have been to change the WANScaler connection timeout setting to more than two hours to allow the video application's keep-alive packet to be sent so the WANScaler would not reset the TCP control connection prematurely. However, this option adds additional overhead to the WANScaler, so it may not be the best alternative. Still, depending on the load of the WANScaler, this could be used as a potential solution.
Another option would have been to contact the video application vendor to determine if the keep-alive packet send interval could have been changed (to sooner than every 60 minutes) to avoid the TCP reset issued by the WANScaler.
Both of these alternatives would have allowed the video application's TCP control connection to remain accelerated, although the acceleration of this control connection would offer little to no benefit to the application because little data is sent over this connection. The solution originally offered (disabling flow control for TCP ports 1700-1749) was effectively the quickest and simplest approach to resolve the issue.
