Connection Flow Through a NetScaler Appliance
Article ID: CTX136705
Updated On:
Description
This article contains an overview of basic NetScaler connection handling, how the maxClients and maxRequests settings come into effect, and how the SurgeQ is used. In addition, basic understanding of connection handling in a NetScaler appliance, which can be used to troubleshoot, diagnose, or isolate the root cause as internal NetScaler appliance or external environmental issues.
Background
There are scenarios where the NetScaler appliance has server performance issues, but the symptoms indicate that the NetScaler appliance is the root cause. However, further analysis of the logs indicates that the appliance is not the root cause and the performance issue is because of some other entity in the environment. There are scenarios where the appliance can cause degradation of the service. These scenarios are caused by resource contention internally in the appliance.
High Level Client Side Connection Flow
When a client connection terminates at a NetScaler virtual server, the connection is placed in a logical entity in the virtual server called the VServer SurgeQ. The SurgeQ is a first in first out queue unless you enable priority queuing, which is not enabled by default. All connections that the appliance terminates are placed into the SurgeQ immediately after connection establishment.
Therefore, the SurgeQ can be considered as the place where connections are placed when you are completing any of the following actions:
- Evaluating the load balancing algorithm and determining the best server to send a new connection to.
- Evaluating whether there is an existing connection in the NetScaler server side re-use pool that you can use for the new incoming connection.
- Determining if all services have reached their maxClient quota or not.
- Determining if there is a spillover virtual server configured as a fallback, if services has reached the maxclient quota.
The SurgeQ usually does not build up in ideal situations as the server connection offloading process is highly optimized within the appliance. However, there are some scenarios where you can expect SurgeQ build up. These scenarios are described in the Surge Q Buildup section.
High Level Server Side Connection Flow
On the Server side of the NetScaler appliance, in most configurations the appliance opens a connection independent from the client connection, especially when using HTTP and TCP type entities.
When the service has a maxClient setting configured, the appliance does not open any connections more than the maxClient number of TCP connections to the backend server. In a TCP type entity, there is no real differentiation between a request and a client. However, in an HTTP type entity, a request represents a single HTTP request.
With the maxRequests setting defined, it controls how many requests will pipeline down the same TCP connection before the appliance terminates it and forces to use a new connection. You can consider it as a method to pipeline X number of requests, and then clean up the connection so that you are turning over connections periodically.
You can consider this as a maintenance measure. It is healthy to ensure memory is de-allocated, wiped, and re-allocated for new connections periodically because these connections can be shared between various frontend clients.
The maxClients setting determines how many TCP connections the appliance must open to the backend server.
When the appliance places a client side connection in the SurgeQ, the appliance completes the following actions:
-
It first looks at the persistence table to verify if there is a server already set for persistence with this client.
-
If there is a persistent entry, then the appliance tries to connect to the existing server entry. If that server has reached the maxClients quota, then the appliance places the client in the SurgeQ until the client can connect to the server.
-
If there is no persistence entry then the appliance selects the current best server based on the load balancing algorithm calculation.
-
After the appliance selects the server, it checks the server re-use pool to verify if there is an existing established connection to use. If available, then the appliance uses that connection and feeds the new payload data through this connection.
-
If there is no re-use connection available, then the appliance looks at the current number of clients and the maxClients setting to verify if the appliance can open a new connection. If the appliance cannot open a new connection to the backend server, then the client request continues to wait in the SurgeQ till the number of established connections is less than the value of maxClient.
SurgeQ Buildup
The following are the typical scenarios when a SurgeQ builds up:
- When the number of connections to the server has reached the maxClient quota and the server connections are long lived.
- When the server response time is high in an HTTP virtual server and the appliance opens more connections to the backend server to offload the individual HTTP request. The servers can get overloaded and fail to establish new connections and therefore causes the SurgeQ to build up.
- When there is internal resource issues on the appliance such as lack of memory, or 100 percent CPU spike, where the appliance is not able to allocate memory blocks for new connections, or CPU load is causing the appliance to stop processing all packets leading to degradation of service.
Additional Resources
Other than the default behavior, there are some configurations for which the preceding content does not apply. For example, when the appliance is in transparent mode and the appliance is L2 or L3 switching the frames. Few additional settings such as spillover are also available on a NetScaler appliance, which allows the appliance to have a backup resource to send network traffic to, when the maxClients number of connections are being processed. Refer to the NetScaler Documentation for more information.
Issue/Introduction
