MPX 15k-25 Gig interfaces down and dropping lacpdus post upgrade to 13.0-83.x

MPX 15k-25 Gig interfaces down and dropping lacpdus post upgrade to 13.0-83.x

book

Article ID: CTX338513

calendar_today

Updated On:

Description

25G interfaces went down after upgrading the MPX from 13.0-67.39 to 13.0-84.10. The issue goes away as soon as downgrade is done to the original version 
 
We can clearly see that that 25/x interfaces are not receiving any LACPDUs post upgrade. We only see tx packets and no rx lacpdus  :
 
Post upgrade :
 
 nsconmsg -K newnslog -g lacpdus -s disptime=1 -d current | less
Displaying performance information
NetScaler V20 Performance Data
NetScaler NS13.1: Build 9.60.nc, Date: Nov 17 2021, 08:06:36   (64-bit)
 
 
reltime:mili second between two records Sun Dec 19 13:49:32 2021
  Index   rtime totalcount-val      delta rate/sec symbol-name&device-no&time
      0    3579              1          1        0 nic_tot_tx_lacpdus interface(25/1) Sun Dec 19 13:49:32 2021
      1       0              1          1        0 nic_tot_tx_lacpdus interface(25/2) Sun Dec 19 13:49:32 2021
      2    6999              3          2        0 nic_tot_tx_lacpdus interface(25/1) Sun Dec 19 13:49:39 2021
      3       0              3          2        0 nic_tot_tx_lacpdus interface(25/2) Sun Dec 19 13:49:39 2021
      4   28000              4          1        0 nic_tot_tx_lacpdus interface(25/1) Sun Dec 19 13:50:07 2021
      5       0              4          1        0 nic_tot_tx_lacpdus interface(25/2) Sun Dec 19 13:50:07 2021
      6   28001              5          1        0 nic_tot_tx_lacpdus interface(25/1) Sun Dec 19 13:50:35 2021
      7       0              5          1        0 nic_tot_tx_lacpdus interface(25/2) Sun Dec 19 13:50:35 2021
      8   35000              6          1        0 nic_tot_tx_lacpdus interface(25/1) Sun Dec 19 13:51:10 2021
      9       0              6          1        0 nic_tot_tx_lacpdus interface(25/2) Sun Dec 19 13:51:10 2021
     10   28001              7          1        0 nic_tot_tx_lacpdus interface(25/1) Sun Dec 19 13:51:38 2021
     11       0              7          1        0 nic_tot_tx_lacpdus interface(25/2) Sun Dec 19 13:51:38 2021
     12   28001              8          1        0 nic_tot_tx_lacpdus interface(25/1) Sun Dec 19 13:52:06 2021
     13       0              8          1        0 nic_tot_tx_lacpdus interface(25/2) Sun Dec 19 13:52:06 2021
     14   28000              9          1        0 nic_tot_tx_lacpdus interface(25/1) Sun Dec 19 13:52:34 2021
     15       0              9          1        0 nic_tot_tx_lacpdus interface(25/2) Sun Dec 19 13:52:34 2021
     16   35001             10          1        0 nic_tot_tx_lacpdus interface(25/1) Sun Dec 19 13:53:09 2021
     17       0             10          1        0 nic_tot_tx_lacpdus interface(25/2) Sun Dec 19 13:53:09 2021
     18   28000             11          1        0 nic_tot_tx_lacpdus interface(25/1) Sun Dec 19 13:53:37 2021
     19       0             11          1        0 nic_tot_tx_lacpdus interface(25/2) Sun Dec 19 13:53:37 2021
     20   28001             12          1        0 nic_tot_tx_lacpdus interface(25/1) Sun Dec 19 13:54:05 2021
     21       0             12          1        0 nic_tot_tx_lacpdus interface(25/2) Sun Dec 19 13:54:05 2021
 
exec: show channel
1)      Interface LA/1 (802.3ad Link Aggregate) #19
        flags=0x41004000 <ENABLED, DOWN, AGGREGATE, down, HAMON, HEARTBEAT, 802.1q>
        MTU=1500, native vlan=1, MAC=02:e0:ed:9e:a6:4e, downtime 0h10m16s
        Requested: media NONE, speed AUTO, duplex NONE, fctl NONE,
                 throughput 0
        Actual: throughput 0
        LLDP Mode: NONE
        RX: Pkts(0) Bytes(0) Errs(0) Drops(0) Stalls(0)
        TX: Pkts(0) Bytes(0) Errs(0) Drops(0) Stalls(0)
        NIC: InDisc(0) OutDisc(0) Fctls(0) Stalls(0) Hangs(0) Muted(0)
        Bandwidth thresholds are not set.
        LA mode: AUTO
                25/1: unknown                   DOWN
                PortID=(32768,11), Mux=DETACHED, Rx=DEFAULTED, UNSELECTED
                <Active, Long timeout, Agg, Defaulted>
                Partner: SysID=(0,00:00:00:00:00:00), Key=0, PortID=(0, 0)
                        <Passive, Long timeout>
 
                25/2: unknown                   DOWN
                PortID=(32768,12), Mux=DETACHED, Rx=DEFAULTED, UNSELECTED
                <Active, Long timeout, Agg, Defaulted>
                Partner: SysID=(0,00:00:00:00:00:00), Key=0, PortID=(0, 0)
                        <Passive, Long timeout>
 
 
 
 
Post downgrade or on the original version 13.0-67.x :
 
Both rx and tx are seen :
 
[nsconmsg -K newnslog -g lacpdus -s disptime=1 -d current | less
Displaying performance information
NetScaler V20 Performance Data
NetScaler NS13.0: Build 67.39.nc, Date: Oct  7 2020, 08:37:53   (64-bit)
 
 
reltime:mili second between two records Sun Dec 19 14:25:11 2021
  Index   rtime totalcount-val      delta rate/sec symbol-name&device-no&time
      0    4087              2          2        0 nic_tot_tx_lacpdus interface(25/1) Sun Dec 19 14:25:11 2021
      1       0              2          2        0 nic_tot_tx_lacpdus interface(25/2) Sun Dec 19 14:25:11 2021
      2    7096              1          1        0 nic_tot_rx_lacpdus interface(25/1) Sun Dec 19 14:25:18 2021
      3       0              7          5        0 nic_tot_tx_lacpdus interface(25/1) Sun Dec 19 14:25:18 2021
      4       0              1          1        0 nic_tot_rx_lacpdus interface(25/2) Sun Dec 19 14:25:18 2021
      5       0              8          6        0 nic_tot_tx_lacpdus interface(25/2) Sun Dec 19 14:25:18 2021
      6    7000              3          2        0 nic_tot_rx_lacpdus interface(25/1) Sun Dec 19 14:25:25 2021
      7       0              8          1        0 nic_tot_tx_lacpdus interface(25/1) Sun Dec 19 14:25:25 2021
      8       0              3          2        0 nic_tot_rx_lacpdus interface(25/2) Sun Dec 19 14:25:25 2021
      9       0              9          1        0 nic_tot_tx_lacpdus interface(25/2) Sun Dec 19 14:25:25 2021
     10   21000              4          1        0 nic_tot_rx_lacpdus interface(25/1) Sun Dec 19 14:25:46 2021
     11       0              4          1        0 nic_tot_rx_lacpdus interface(25/2) Sun Dec 19 14:25:46 2021
     12    7000              9          1        0 nic_tot_tx_lacpdus interface(25/1) Sun Dec 19 14:25:53 2021
     13       0             10          1        0 nic_tot_tx_lacpdus interface(25/2) Sun Dec 19 14:25:53 2021
     14   21000              5          1        0 nic_tot_rx_lacpdus interface(25/1) Sun Dec 19 14:26:14 2021
     15    7000             10          1        0 nic_tot_tx_lacpdus interface(25/1) Sun Dec 19 14:26:21 2021
     16       0              5          1        0 nic_tot_rx_lacpdus interface(25/2) Sun Dec 19 14:26:21 2021
     17       0             11          1        0 nic_tot_tx_lacpdus interface(25/2) Sun Dec 19 14:26:21 2021
     18   21000              6          1        0 nic_tot_rx_lacpdus interface(25/1) Sun Dec 19 14:26:42 2021
     19    7000             11          1        0 nic_tot_tx_lacpdus interface(25/1) Sun Dec 19 14:26:49 2021
     20       0              6          1        0 nic_tot_rx_lacpdus interface(25/2) Sun Dec 19 14:26:49 2021
     21       0             12          1        0 nic_tot_tx_lacpdus interface(25/2) Sun Dec 19 14:26:49 2021
     22   21000              7          1        0 nic_tot_rx_lacpdus interface(25/2) Sun Dec 19 14:27:10 2021
     23    7000              7          1        0 nic_tot_rx_lacpdus interface(25/1) Sun Dec 19 14:27:17 2021
     24    7000
 
exec: show channel
 
1)      Interface LA/1 (802.3ad Link Aggregate) #19
        flags=0x4100c020 <ENABLED, UP, AGGREGATE, UP, HAMON, HEARTBEAT, 802.1q>
        MTU=1500, native vlan=1, MAC=02:e0:ed:9e:a6:4e, uptime 0h17m02s
        Requested: media NONE, speed AUTO, duplex NONE, fctl NONE,
                 throughput 0
        Actual: throughput 2000
        LLDP Mode: NONE
        RX: Pkts(67274) Bytes(10429481) Errs(0) Drops(27738) Stalls(0)
        TX: Pkts(80302) Bytes(15215185) Errs(0) Drops(0) Stalls(0)
        NIC: InDisc(0) OutDisc(0) Fctls(0) Stalls(0) Hangs(0) Muted(0)
        Bandwidth thresholds are not set.
        LA mode: AUTO
 
 
                25/1: FIBER-1000-FULL-OFF       UP  0h17m08s
                PortID=(32768,11), Mux=DISTRIBUTING, Rx=CURRENT, SELECTED
                <Active, Long timeout, Agg, Sync, Collecting, Distributing>
                Partner: SysID=(32768,00:9a:d2:7d:6c:80), Key=5, PortID=(32768, 563)
                        <Active, Long timeout, Agg, Sync, Collecting, Distributing>
 
                25/2: FIBER-1000-FULL-OFF       UP  0h17m08s
                PortID=(32768,12), Mux=DISTRIBUTING, Rx=CURRENT, SELECTED
                <Active, Long timeout, Agg, Sync, Collecting, Distributing>
                Partner: SysID=(32768,00:9a:d2:7d:6c:80), Key=5, PortID=(32768, 306)
                        <Active, Long timeout, Agg, Sync, Collecting, Distributing>
 
 

Resolution

Problem due to AUTONEG behaviour. Interface stays in DOWN state after upgrade (or sometimes reboot) when AUTONEG is ON.We need to disable AUTONEG on all the 25G interfaces. This needs to be done on both primary and secondary nodes of the HA pair.
 

Problem Cause

Autonegotiation on Fortville NICs is not supported.