25G interfaces went down after upgrading the MPX from 13.0-67.39 to 13.0-84.10. The issue goes away as soon as downgrade is done to the original version
We can clearly see that that 25/x interfaces are not receiving any LACPDUs post upgrade. We only see tx packets and no rx lacpdus :
Post upgrade :
nsconmsg -K newnslog -g lacpdus -s disptime=1 -d current | less
Displaying performance information
NetScaler V20 Performance Data
NetScaler NS13.1: Build 9.60.nc, Date: Nov 17 2021, 08:06:36 (64-bit)
reltime:mili second between two records Sun Dec 19 13:49:32 2021
Index rtime totalcount-val delta rate/sec symbol-name&device-no&time
0 3579 1 1 0 nic_tot_tx_lacpdus interface(25/1) Sun Dec 19 13:49:32 2021
1 0 1 1 0 nic_tot_tx_lacpdus interface(25/2) Sun Dec 19 13:49:32 2021
2 6999 3 2 0 nic_tot_tx_lacpdus interface(25/1) Sun Dec 19 13:49:39 2021
3 0 3 2 0 nic_tot_tx_lacpdus interface(25/2) Sun Dec 19 13:49:39 2021
4 28000 4 1 0 nic_tot_tx_lacpdus interface(25/1) Sun Dec 19 13:50:07 2021
5 0 4 1 0 nic_tot_tx_lacpdus interface(25/2) Sun Dec 19 13:50:07 2021
6 28001 5 1 0 nic_tot_tx_lacpdus interface(25/1) Sun Dec 19 13:50:35 2021
7 0 5 1 0 nic_tot_tx_lacpdus interface(25/2) Sun Dec 19 13:50:35 2021
8 35000 6 1 0 nic_tot_tx_lacpdus interface(25/1) Sun Dec 19 13:51:10 2021
9 0 6 1 0 nic_tot_tx_lacpdus interface(25/2) Sun Dec 19 13:51:10 2021
10 28001 7 1 0 nic_tot_tx_lacpdus interface(25/1) Sun Dec 19 13:51:38 2021
11 0 7 1 0 nic_tot_tx_lacpdus interface(25/2) Sun Dec 19 13:51:38 2021
12 28001 8 1 0 nic_tot_tx_lacpdus interface(25/1) Sun Dec 19 13:52:06 2021
13 0 8 1 0 nic_tot_tx_lacpdus interface(25/2) Sun Dec 19 13:52:06 2021
14 28000 9 1 0 nic_tot_tx_lacpdus interface(25/1) Sun Dec 19 13:52:34 2021
15 0 9 1 0 nic_tot_tx_lacpdus interface(25/2) Sun Dec 19 13:52:34 2021
16 35001 10 1 0 nic_tot_tx_lacpdus interface(25/1) Sun Dec 19 13:53:09 2021
17 0 10 1 0 nic_tot_tx_lacpdus interface(25/2) Sun Dec 19 13:53:09 2021
18 28000 11 1 0 nic_tot_tx_lacpdus interface(25/1) Sun Dec 19 13:53:37 2021
19 0 11 1 0 nic_tot_tx_lacpdus interface(25/2) Sun Dec 19 13:53:37 2021
20 28001 12 1 0 nic_tot_tx_lacpdus interface(25/1) Sun Dec 19 13:54:05 2021
21 0 12 1 0 nic_tot_tx_lacpdus interface(25/2) Sun Dec 19 13:54:05 2021
exec: show channel
1) Interface LA/1 (802.3ad Link Aggregate) #19
flags=0x41004000 <ENABLED, DOWN, AGGREGATE, down, HAMON, HEARTBEAT, 802.1q>
MTU=1500, native vlan=1, MAC=02:e0:ed:9e:a6:4e, downtime 0h10m16s
Requested: media NONE, speed AUTO, duplex NONE, fctl NONE,
throughput 0
Actual: throughput 0
LLDP Mode: NONE
RX: Pkts(0) Bytes(0) Errs(0) Drops(0) Stalls(0)
TX: Pkts(0) Bytes(0) Errs(0) Drops(0) Stalls(0)
NIC: InDisc(0) OutDisc(0) Fctls(0) Stalls(0) Hangs(0) Muted(0)
Bandwidth thresholds are not set.
LA mode: AUTO
25/1: unknown DOWN
PortID=(32768,11), Mux=DETACHED, Rx=DEFAULTED, UNSELECTED
<Active, Long timeout, Agg, Defaulted>
Partner: SysID=(0,00:00:00:00:00:00), Key=0, PortID=(0, 0)
<Passive, Long timeout>
25/2: unknown DOWN
PortID=(32768,12), Mux=DETACHED, Rx=DEFAULTED, UNSELECTED
<Active, Long timeout, Agg, Defaulted>
Partner: SysID=(0,00:00:00:00:00:00), Key=0, PortID=(0, 0)
<Passive, Long timeout>
Post downgrade or on the original version 13.0-67.x :
Both rx and tx are seen :
[nsconmsg -K newnslog -g lacpdus -s disptime=1 -d current | less
Displaying performance information
NetScaler V20 Performance Data
NetScaler NS13.0: Build 67.39.nc, Date: Oct 7 2020, 08:37:53 (64-bit)
reltime:mili second between two records Sun Dec 19 14:25:11 2021
Index rtime totalcount-val delta rate/sec symbol-name&device-no&time
0 4087 2 2 0 nic_tot_tx_lacpdus interface(25/1) Sun Dec 19 14:25:11 2021
1 0 2 2 0 nic_tot_tx_lacpdus interface(25/2) Sun Dec 19 14:25:11 2021
2 7096 1 1 0 nic_tot_rx_lacpdus interface(25/1) Sun Dec 19 14:25:18 2021
3 0 7 5 0 nic_tot_tx_lacpdus interface(25/1) Sun Dec 19 14:25:18 2021
4 0 1 1 0 nic_tot_rx_lacpdus interface(25/2) Sun Dec 19 14:25:18 2021
5 0 8 6 0 nic_tot_tx_lacpdus interface(25/2) Sun Dec 19 14:25:18 2021
6 7000 3 2 0 nic_tot_rx_lacpdus interface(25/1) Sun Dec 19 14:25:25 2021
7 0 8 1 0 nic_tot_tx_lacpdus interface(25/1) Sun Dec 19 14:25:25 2021
8 0 3 2 0 nic_tot_rx_lacpdus interface(25/2) Sun Dec 19 14:25:25 2021
9 0 9 1 0 nic_tot_tx_lacpdus interface(25/2) Sun Dec 19 14:25:25 2021
10 21000 4 1 0 nic_tot_rx_lacpdus interface(25/1) Sun Dec 19 14:25:46 2021
11 0 4 1 0 nic_tot_rx_lacpdus interface(25/2) Sun Dec 19 14:25:46 2021
12 7000 9 1 0 nic_tot_tx_lacpdus interface(25/1) Sun Dec 19 14:25:53 2021
13 0 10 1 0 nic_tot_tx_lacpdus interface(25/2) Sun Dec 19 14:25:53 2021
14 21000 5 1 0 nic_tot_rx_lacpdus interface(25/1) Sun Dec 19 14:26:14 2021
15 7000 10 1 0 nic_tot_tx_lacpdus interface(25/1) Sun Dec 19 14:26:21 2021
16 0 5 1 0 nic_tot_rx_lacpdus interface(25/2) Sun Dec 19 14:26:21 2021
17 0 11 1 0 nic_tot_tx_lacpdus interface(25/2) Sun Dec 19 14:26:21 2021
18 21000 6 1 0 nic_tot_rx_lacpdus interface(25/1) Sun Dec 19 14:26:42 2021
19 7000 11 1 0 nic_tot_tx_lacpdus interface(25/1) Sun Dec 19 14:26:49 2021
20 0 6 1 0 nic_tot_rx_lacpdus interface(25/2) Sun Dec 19 14:26:49 2021
21 0 12 1 0 nic_tot_tx_lacpdus interface(25/2) Sun Dec 19 14:26:49 2021
22 21000 7 1 0 nic_tot_rx_lacpdus interface(25/2) Sun Dec 19 14:27:10 2021
23 7000 7 1 0 nic_tot_rx_lacpdus interface(25/1) Sun Dec 19 14:27:17 2021
24 7000
exec: show channel
1) Interface LA/1 (802.3ad Link Aggregate) #19
flags=0x4100c020 <ENABLED, UP, AGGREGATE, UP, HAMON, HEARTBEAT, 802.1q>
MTU=1500, native vlan=1, MAC=02:e0:ed:9e:a6:4e, uptime 0h17m02s
Requested: media NONE, speed AUTO, duplex NONE, fctl NONE,
throughput 0
Actual: throughput 2000
LLDP Mode: NONE
RX: Pkts(67274) Bytes(10429481) Errs(0) Drops(27738) Stalls(0)
TX: Pkts(80302) Bytes(15215185) Errs(0) Drops(0) Stalls(0)
NIC: InDisc(0) OutDisc(0) Fctls(0) Stalls(0) Hangs(0) Muted(0)
Bandwidth thresholds are not set.
LA mode: AUTO
25/1: FIBER-1000-FULL-OFF UP 0h17m08s
PortID=(32768,11), Mux=DISTRIBUTING, Rx=CURRENT, SELECTED
<Active, Long timeout, Agg, Sync, Collecting, Distributing>
Partner: SysID=(32768,00:9a:d2:7d:6c:80), Key=5, PortID=(32768, 563)
<Active, Long timeout, Agg, Sync, Collecting, Distributing>
25/2: FIBER-1000-FULL-OFF UP 0h17m08s
PortID=(32768,12), Mux=DISTRIBUTING, Rx=CURRENT, SELECTED
<Active, Long timeout, Agg, Sync, Collecting, Distributing>
Partner: SysID=(32768,00:9a:d2:7d:6c:80), Key=5, PortID=(32768, 306)
<Active, Long timeout, Agg, Sync, Collecting, Distributing>