To resolve the issue, make sure to update the default route on the backend servers to point to the ADC appliance as a default gateway.
With USIP mode on the ADC appliance, the backend servers should always have the NetScaler IP (NSIP) address as the default gateway. If you want to configure the default router as a return path for the response, then enable the Direct Server Return (DSR) feature on the ADC appliance.
The backend servers were responding to the source IP address because the customer had enabled the USIP mode on the appliance. Therefore, the default route did not point to the ADC appliance. The response from the servers was never sent to the client through the ADC. Additionally, the route on the server was changed for the production network traffic. Therefore, the ADC appliance had multiple open connections to the backend servers. As these connections were not getting a response on time, the connections started to collect in the surge queue.
Run the following command to verify the load balancing virtual server statistics:
stat lb vserver <vserver_name>
Run the following command to verify that there are not many requests to the load balancing Virtual IP (VIP):
stat service <service_name>
The output of this command displays the connections in the surge queue for the IP addresses of the servers having issue.
Analyze the network packet traces of the ADC appliance. The analysis of the traces indicates that the monitor probes for the IP address of the servers with the issue are not receiving any acknowledgement from many instances.
Verify if USIP mode is enabled on the appliance. If it is enabled on the appliance, then it indicates an issue with the backend servers.