Services flaps on VPX/Interface hands observed on XenServer

All the interfaces will showing up in SVM level., but vpx level, it will show down.
If this is the case, then this may match with one of the known bug’s
If you encounter this issue, Please open a support case with Citrix.
This is a known issue and it is fixed in 11.1 build 56.19
 

---

Problem Cause

VIP’s go down and servers will flap.
Interfaces will miss out from the configuration
sh interface <10/1> command will show ‘Stalls’ and ‘Hangs’ in the output.
eg:
Interface 10/1 (10G vNIC Interface, pNIC 10/5) #0
flags=0xe460 <ENABLED, UP, UP, HAMON, HEARTBEAT, 802.1q>
MTU=1500, native vlan=1, MAC=x:x:x:x:x:x, uptime 0h34m32s
Actual: media FIBER, speed 10000, duplex FULL, fctl NONE, throughput 10000
LLDP Mode: NONE,                 LR Priority: 1024
 
RX: Pkts(63484) Bytes(4422612) Errs(0) Drops(15467) Stalls(0)
TX: Pkts(58915) Bytes(2474502) Errs(0) Drops(1) Stalls(67)
NIC: InDisc(0) OutDisc(0) Fctls(0) Stalls(0) Hangs(2) Muted(0)
Bandwidth thresholds are not set.
 
Interface 10/2 (10G vNIC Interface, pNIC 10/6) #1
flags=0x6640 <ENABLED, DOWN, down, HEARTBEAT, 802.1q>
LACP <Active, Long timeout, key 2, priority 32768>
MTU=1500, MAC=x:x:x:x:x:x, downtime 8h19m02s
LLDP Mode: NONE,                 LR Priority: 1024
 
RX: Pkts(11762) Bytes(1144780) Errs(0) Drops(9694) Stalls(0)
TX: Pkts(115) Bytes(14434) Errs(0) Drops(0) Stalls(4)
NIC: InDisc(0) OutDisc(0) Fctls(0) Stalls(0) Hangs(3) Muted(0)
Bandwidth thresholds are not set.
 
corresponding eth interface mapping :
 
eth2", "mtu" : 1500, "port" : "10/7"
eth3", "mtu" : 1500, "port" : "10/8”
eth5", "mtu" : 1500, "port" : "10/6"                      <-----------------     This wil be marked as DOWN on VPX
eth4", "mtu" : 1500, "port" : "10/5"
 
newnslogs shows below info:
ns.log.8:Oct 29 21:55:58 <local0.notice> 10.x.x.x/29/2017:16:25:58 GMT hostname 0-PPE-2 : default EVENT DEVICEDOWN 3168881 0 :  Device "interface(10/2)" - State DOWN
ns.log.8:Oct 29 21:55:58 <local0.notice> 10.x.x.x 10/29/2017:16:25:58 GMT hostname 0-PPE-1 : default EVENT DEVICEDOWN 3212975 0 :  Device "interface(10/2)" - State DOWN
ns.log.8:Oct 29 21:55:58 <local0.notice> 10.x.x.x 10/29/2017:16:25:58 GMT hostname 0-PPE-0 : default EVENT DEVICEDOWN 1351999 0 :  Device "interface(10/2)" - State DOWN
 
Xenserver kernlog shows below info:
cat kern.log | grep "Detected Tx Unit Hang"
Oct 28 21:13:38 netscaler-sdx kernel: [11487774.550966] ixgbe 0000:08:00.1 eth3: Detected Tx Unit Hang
Oct 28 23:26:42 netscaler-sdx kernel: [  368.183337] ixgbe 0000:0b:00.1 eth5: Detected Tx Unit Hang
Oct 29 09:36:20 netscaler-sdx kernel: [36946.735328] ixgbe 0000:0b:00.1 eth5: Detected Tx Unit Hang
 
Fake hang logs:
eg:
Oct 23 05:50:35 netscaler-sdx kernel: [11000391.898410] ixgbe 0000:0b:00.0 eth4: Fake Tx hang detected with timeout of 80 seconds
Oct 23 06:21:20 netscaler-sdx kernel: [11002236.890412] ixgbe 0000:0b:00.0 eth4: Fake Tx hang detected with timeout of 80 seconds
Oct 23 06:50:46 netscaler-sdx kernel: [11004003.034410] ixgbe 0000:0b:00.0 eth4: Fake Tx hang detected with timeout of 80 seconds
Oct 23 07:21:31 netscaler-sdx kernel: [11005848.026412] ixgbe 0000:0b:00.0 eth4: Fake Tx hang detected with timeout of 80 seconds
Oct 23 07:49:37 netscaler-sdx kernel: [11007534.042409] ixgbe 0000:0b:00.0 eth4: Fake Tx hang detected with timeout of 80 seconds
Oct 23 08:17:42 netscaler-sdx kernel: [11009219.034410] ixgbe 0000:0b:00.0 eth4: Fake Tx hang detected with timeout of 80 seconds


 

Services flaps on VPX/Interface hands observed on XenServer

Release notes of 11.1 build 56.19 attached which covers this bug