Huge Packet Drops (Tx drops) Observed on NetScaler

book

Article ID: CTX215843

calendar_today

Updated On:

Description

  • Huge packet drops was observed on NetScaler. The drops are TX drops (software drops).
  • Customer attempted to ping NSIP and could see that most of the packet gets dropped. All configured services goes down and comes up frequently. Users could see that application gets disconnected very frequently (Applications have been launched through NetScaler Gateway).
  • We can see from newnslog that "nic_err_congestionlimit_pkts_dropped" and "nic_err_tx_dropped" counters" are incrementing: 
    /var/nslog]$ nsconmsg101 -K newnslog -g nic_err -s disptime=1 -d current | grep congest |more
   6195       0        6159352      48887     6944 nic_err_congestionlimit_pkts_dropped interface(0/1) Wed Dec  3 17:17:38 2014
   6201       0        6208229      48877     6962 nic_err_congestionlimit_pkts_dropped interface(0/1) Wed Dec  3 17:17:45 2014
   6402       0        6257108      48879     6943 nic_err_congestionlimit_pkts_dropped interface(0/1) Wed Dec  3 17:25:32 2014
   6462       0        6305991      48883     6953 nic_err_congestionlimit_pkts_dropped interface(0/1) Wed Dec  3 17:27:45 2014
   6468       0        6354868      48877     6932 nic_err_congestionlimit_pkts_dropped interface(0/1) Wed Dec  3 17:27:52 2014
   6805       0        6403744      48876     6942 nic_err_congestionlimit_pkts_dropped interface(0/1) Wed Dec  3 17:40:55 2014
/var/nslog]$ nsconmsg101 -K newnslog -g nic_err_tx_dropped -s time=04Dec2014:17:00 -s disptime=1 -d current | more
 
reltime:mili second between two records Thu Dec  4 17:06:08 2014
  Index   rtime totalcount-val      delta rate/sec symbol-name&device-no&time
      0 2697427        8897070      48877     6942 nic_err_tx_dropped interface(0/1) Thu Dec  4 17:06:08 2014
      1    7040        8945947      48877     6942 nic_err_tx_dropped interface(0/1) Thu Dec  4 17:06:15 2014
      2    7170        8994878      48931     6824 nic_err_tx_dropped interface(0/1) Thu Dec  4 17:06:22 2014
      3   78151        9092633      97755    13885 nic_err_tx_dropped interface(0/1) Thu Dec  4 17:07:40 2014

Resolution

Ensure that you are not using Management port (interface 0/1) for production traffic.

The best practice is to use management card only for Management traffic and the Network card for the Production traffic.

Management interfaces uses software RSS instead of hardware RSS, so this can impact performance.

Enabled the interface 1/1 and then disabled the interface 0/1.

Problem Cause

Management port (interface 0/1) used for production/data traffic.

Issue/Introduction

Huge packet drops was observed on NetScaler. The drops are TX drops (software drops). Also, the service group are going UP and DOWN during the production hours, when this issue is happening.
Powered by Wolken Software