Go check Load Balance Virtual Server's Statistics data, under the Bound Service Group Member(s) Summary tab, you've observed that the Current client connection count of specific server is significantly higher than the other servers within the same service group.

In NS shell mode, "nsconmsg ConLb=2 tool" command output, large number of counts in SQ(Surge Queue) of the specific server is observed.

In NS CLI mode, when using the 'show connectiontable' command with the service IP as a filter, you might observe a large number of TCP links in the syn-sent state. The source IP (Client IP) and destination IP (Server IP) are on the same network segment

This software application is provided to you as is with no representations, warranties or conditions of any kind. You may use and distribute it at your own risk. CITRIX DISCLAIMS ALL WARRANTIES WHATSOEVER, EXPRESS, IMPLIED, WRITTEN, ORAL OR STATUTORY, INCLUDING WITHOUT LIMITATION WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE, TITLE AND NONINFRINGEMENT. Without limiting the generality of the foregoing, you acknowledge and agree that: (a) the software application may exhibit errors, design flaws or other problems, possibly resulting in loss of data or damage to property; (b) it may not be possible to make the software application fully functional; and (c) Citrix may, without notice or liability to you, cease to make available the current version and/or any future versions of the software application. In no event should the software application be used to support ultra-hazardous activities, including but not limited to life support or blasting activities. NEITHER CITRIX NOR ITS AFFILIATES OR AGENTS WILL BE LIABLE, UNDER BREACH OF CONTRACT OR ANY OTHER THEORY OF LIABILITY, FOR ANY DAMAGES WHATSOEVER ARISING FROM USE OF THE SOFTWARE APPLICATION, INCLUDING WITHOUT LIMITATION DIRECT, SPECIAL, INCIDENTAL, PUNITIVE, CONSEQUENTIAL OR OTHER DAMAGES, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGES. You agree to indemnify and defend Citrix against any and all claims arising from your use, modification or distribution of the software application.

Suggest to disable USIP in service group

In CLI, apply below command:

set servicegroup ServiceGroupName -usip NO

 

---

Problem Cause

As USIP is enabled on the service group, NetScaler will using client's IP instead of SNIP to forward packets to backend server.   

If there is a client sitting in the same network segment with the backend server. The backend server's response is sent to the client directly, which bypassed NetScaler. It results in NetScaler stuck in TCP handshake failures, which explained why it has high current client connections in NetScaler for this server.