Problem Definition
Application launches using a static .ica file were failing when using a NetScaler to load balance requests between two XML brokers. The user would receive the following error message:
“A network error caused your request to time out. Try to connect again. If you continue to receive this message, contact your Citrix Administrator.”
Environment
• Windows 2003
• Citrix Presentation Server 4.0 with hotfixes PSE400W2K3R02.0.1, PSE400R02W2K3003, PSE400R02W2K3049, PSE400R02W2K3065, and PSE400R02W2K3071
• NetScaler with build 6.0-47.29
Users were launching a static .ica file to connect to a load balanced application, for the purposes of this case study the application be referenced as “Customers_Application.” The XML broker address referenced in the “HttpBrowserAddress” field of the .ica file was a Virtual IP Address (VIP) which was routed to the NetScaler device. The NetScaler was configured to load balance requests made to port 80 (the default port of the XML service) on this VIP (10.99.34.11) between two physical Citrix Presentation Server 4.0 servers.
The below is an excerpt of the customer’s .ica file –
[WFClient]
Version=2
HttpBrowserAddress=10.99.34.11
PersistentCachePath=Z:\AppData\ICAClient\Cache
[ApplicationServers]
Customers_Application=
[Customers_Application]
Address= Customers_Application
InitialProgram=#Customers_Application
ClientAudio=Off
Compress=On
ScreenPercent=99
DesiredColor=8
TransportDriver=TCP/IP
WinStationDriver=ICA 3.0
BrowserProtocol=HTTPonTCP
UseLocalUserAndPassword=On
EncryptionLevelSession=EncRC5-128
Troubleshooting Methodology
The first thing we needed to verify was that both of the Citrix Presentation Servers were responding to XML requests when not being routed through the NetScaler. We modified the “HttpBrowserAddress” field in the .ica file to point to the actual IP of each Presentation Server and confirmed that we were able to launch the application.
Now that we confirmed that the XML service on each of the Presentation Servers was able to resolve the application, we needed to isolate what might be different when routing the requests through the customer’s NetScaler device. We captured network traces of both types of application launches – directly to the IP address of the Presentation Server and load balanced through the NetScaler.
We used the Wireshark network protocol analyzer to examine the network traces in both cases. The first item to verify in any network trace of TCP/IP traffic is that the TCP 3-way handshake occurred successfully.
So we verified that the 3-way handshake was occurring in both cases. Moving further down in the trace, there appeared to be some retransmissions from the client when the traffic was routed through the NetScaler – this did not occur when making the XML request directly to the Presentation Server.
To isolate what might be different about the XML requests when routed through the NetScaler, we used the “Follow TCP Stream” function in Wireshark to compare the HTTP application-level data from the requests.
The request when sent directly to the Citrix Presentation Server:
The request when the XML request was routed through the NetScaler:
Because we knew that the server was responding to the client at the TCP level for the 3-way handshake but not at the HTTP application level, we needed to see if anything was different with the client’s request when it was routed through the NetScaler. The most obvious difference was that the client’s request included an extra HTTP request header titled, “clientip:,” when it was routed through the NetScaler, but not when sent directly to the Citrix Presentation Server.
The next step was to verify that the “clientip:” request header was directly related to the problem. In order to avoid configuration changes to the customer’s production NetScaler we used the Microsoft utility, wfetch.exe, which is included in the Internet Information Services (IIS) 6 resource kit. Wfetch.exe is a utility that can be used to construct and send HTTP requests to a remote host and then examine the response to that request. We copied the text from the network trace (we used the “Follow TCP Stream” function in WireShark) and pasted it into wfetch.exe. Note that we had to add the newline character, \n, and carriage return, \r, after each line.
So for the next request we removed the “clientip:” header to match the clients request when bypassing the NetScaler.
Now we had isolated the problem to being related to the “clientip:” field. We subsequently added the field back but renamed the header name to “plientip:” and we received a response. After testing different variations we determined we would not get the expected response if the header began with the letters, “c” or “k” (the problem occurred with both upper and lower case letters).
By bypassing the NetScaler and testing the requests using wfetch.exe we were able to isolate the problem to the XML service on the Presentation Server. The XML service was not responding as expected, but only if the request contained a header beginning with certain characters.
Although engineering assistance would be required to confirm the code change necessary to resolve the issue, because of what we had learned the customer had the choice of a couple workarounds until the problem with the XML service could be resolved:
• Disable the Client IP feature on the NetScaler.
• Change the title of the header used for the Client IP feature to something that did not begin with the letter “c” or “k.”
Resolution
We worked with engineering to develop a fix which is expected be released in Citrix Presentation Server 4.0 Hotfix Rollup Pack 4 and Citrix Presentation Server 4.5 Hotfix Rollup Pack 2.
More Information
CTX109753 - How to Log Clients' Source IP Addresses in a NetScaler-proxied Web Server Environment
IIS 6.0 Resource Kit Tools (wfetch.exe is included)
Microsoft Article 284285 - HOW TO: Use Wfetch.exe to Troubleshoot HTTP Connections
WireShark Network Protocol Analyzer
Microsoft Article 172983 - Explanation of the Three-Way Handshake via TCP/IP
