Customer is facing issues with intermittent connection drops. This was found out to be due to TCP small window attack protection feature on NetScaler.

Trace analysis

In the non-working trace we see the following:
The ACK is getting dropped after the initial SYN, SYN-ACK. Here we are able to see the retransmitted packet is the ACK packet.

Also the counter is matching the time of the issue:May 18 01:53:49 2016

nsconmsg -K /var/nslog/newnslog -g tcp_cur_SW

reltime:mili second between two records Wed May 18 01:53:07 2016

  Index   rtime totalcount-val      delta rate/sec symbol-name&device-no&time
    135       0        1258237          1        0 tcp_err_SW_init_pktdrop  Wed May 18 01:53:07 2016
    136    7000            136          1        0 tcp_cur_SW_pcbs  Wed May 18 01:53:14 2016
    137    7000        1258239          2        0 tcp_err_SW_init_pktdrop  Wed May 18 01:53:21 2016
    138       0            132         -4        0 tcp_cur_SW_pcbs  Wed May 18 01:53:21 2016
 140       0        1258241          2        0 tcp_err_SW_init_pktdrop  Wed May 18 01:53:49 2016
    141    7000            134          4        0 tcp_cur_SW_pcbs  Wed May 18 01:53:56 2016
    142       0        1258247          6        0 tcp_err_SW_init_pktdrop  Wed May 18 01:53:56 2016
    143    7000        1258248          1        0 tcp_err_SW_init_pktdrop  Wed May 18 01:54:03 2016
    144       0            133         -1        0 tcp_cur_SW_pcbs  Wed May 18 01:54:03 2016
    145   14000            134          1        0 tcp_cur_SW_pcbs  Wed May 18 01:54:17 2016
    146    7000            130         -4        0 tcp_cur_SW_pcbs  Wed May 18 01:54:24 2016

    147    7000            133          3        0 tcp_cur_SW_pcbs  Wed May 18 01:54:31 2016
    148    7000            132         -1        0 tcp_cur_SW_pcbs  Wed May 18 01:54:38 2016
    149    7000            131         -1        0 tcp_cur_SW_pcbs  Wed May 18 01:54:45 2016
    150    7000            129         -2        0 tcp_cur_SW_pcbs  Wed May 18 01:54:52 2016
    151   14000            130          1        0 tcp_cur_SW_pcbs  Wed May 18 01:55:06 2016
    152   14000            129         -1        0 tcp_cur_SW_pcbs  Wed May 18 01:55:20 2016
    153    7000            130          1        0 tcp_cur_SW_pcbs  Wed May 18 01:55:27 2016
    154    7000            129         -1        0 tcp_cur_SW_pcbs  Wed May 18 01:55:34 2016
    155    7000            130          1        0 tcp_cur_SW_pcbs  Wed May 18 01:55:41 2016
    156    7000            128         -2        0 tcp_cur_SW_pcbs  Wed May 18 01:55:48 2016
    157    7000            129          1        0 tcp_cur_SW_pcbs  Wed May 18 01:55:55 2016

NetScaler TCP Small Window Protection

This is a protection feature on the NetScaler which will be evoked when the NetScaler receives a packet with Window size less than the negotiated MSS value.  The NetScaler will drops the final ack packet silently resulting in the session not getting established or packet being sent to the backend server.

More Information

If a Client with an Initial Sequence Number (ISN) advertises a window size in the final ack which is less than the MSS, then such clients ISN will be put into a hash table and marked as malicious. Before marking the packet as malicious it will send keep alives to the client in order to open up the window or send a higher window size packet.

Further if NetScaler receives a connection request with an ISN matching the entry in the hash table from same or different client it will be silently dropped in the final ACK of such transaction.

In NetScaler, we can see following two counters increment when the TCP small window packets are received and dropped silently:
tcp_cur_SW_pcbs 
tcp_err_SW_init_pktdrop

Procedure to disable TCP small window protection on NetScaler

We can disable the small window option through nsapimgr command:
root@ns# nsapimgr_wr.sh -ys small_window_protection=0
Number of PEs running: 1
Changing Connection startup small window protection from 1 to 0 ... Done.

Note: nsapimgr are developers specific and these commands needs to be run only under the advice of Citrix Support. Kindly contact Citrix Support before applying any nsapimgr commands.

This article discusses a case study with TCP small window attack on NetScaler.

https://support.citrix.com/article/CTX231777
https://support.citrix.com/article/CTX113341