• Huge packet drops was observed on NetScaler. The drops are TX drops (software drops).
• Customer attempted to ping NSIP and could see that most of the packet gets dropped. All configured services goes down and comes up frequently. Users could see that application gets disconnected very frequently (Applications have been launched through NetScaler Gateway).
• We can see from newnslog that "nic_err_congestionlimit_pkts_dropped" and "nic_err_tx_dropped" counters" are incrementing:

  /var/nslog]$ nsconmsg101 -K newnslog -g nic_err -s disptime=1 -d current | grep congest |more
     6195       0        6159352      48887     6944 nic_err_congestionlimit_pkts_dropped interface(0/1) Wed Dec  3 17:17:38 2014
     6201       0        6208229      48877     6962 nic_err_congestionlimit_pkts_dropped interface(0/1) Wed Dec  3 17:17:45 2014
     6402       0        6257108      48879     6943 nic_err_congestionlimit_pkts_dropped interface(0/1) Wed Dec  3 17:25:32 2014
     6462       0        6305991      48883     6953 nic_err_congestionlimit_pkts_dropped interface(0/1) Wed Dec  3 17:27:45 2014
     6468       0        6354868      48877     6932 nic_err_congestionlimit_pkts_dropped interface(0/1) Wed Dec  3 17:27:52 2014
     6805       0        6403744      48876     6942 nic_err_congestionlimit_pkts_dropped interface(0/1) Wed Dec  3 17:40:55 2014
  /var/nslog]$ nsconmsg101 -K newnslog -g nic_err_tx_dropped -s time=04Dec2014:17:00 -s disptime=1 -d current | more
   
  reltime:mili second between two records Thu Dec  4 17:06:08 2014
    Index   rtime totalcount-val      delta rate/sec symbol-name&device-no&time
        0 2697427        8897070      48877     6942 nic_err_tx_dropped interface(0/1) Thu Dec  4 17:06:08 2014
        1    7040        8945947      48877     6942 nic_err_tx_dropped interface(0/1) Thu Dec  4 17:06:15 2014
        2    7170        8994878      48931     6824 nic_err_tx_dropped interface(0/1) Thu Dec  4 17:06:22 2014
        3   78151        9092633      97755    13885 nic_err_tx_dropped interface(0/1) Thu Dec  4 17:07:40 2014 

Ensure that you are not using Management port (interface 0/1) for production traffic.

Management interfaces uses software RSS instead of hardware RSS, so this can impact performance.

Enabled the interface 1/1 and then disabled the interface 0/1.

Problem Cause

Management port (interface 0/1) used for production/data traffic.