Citrix ADC experiencing GSLB MEP flapping, every 7-10 seconds goes UP/DOWN

Description

Log Analysis:

ADC Events displaying the flapping:
# nsconmsg -K /var/nslog/newnslog -d event | grep -i mep
7741410     0 PPE-0 Site 'gslbsite_10.200.10.10:(CA_gslb_site)' MEP UP Mon Jul 22 18:11:23 2019
6689704     0 PPE-1 Site 'gslbsite_10.200.10.10:(CA_gslb_site)' MEP UP Mon Jul 22 18:11:23 2019
7741421     0 PPE-0 Site 'gslbsite_10.200.10.10:(CA_gslb_site)' MEP DOWN Mon Jul 22 18:11:29 2019
6689715     0 PPE-1 Site 'gslbsite_10.200.10.10:(CA_gslb_site)' MEP DOWN Mon Jul 22 18:11:29 2019
7741440     7 PPE-0 Site 'gslbsite_10.200.10.10:(CA_gslb_site)' MEP UP Mon Jul 22 18:11:45 2019
6689730     0 PPE-1 Site 'gslbsite_10.200.10.10:(CA_gslb_site)' MEP UP Mon Jul 22 18:11:45 2019
7741459     0 PPE-0 Site 'gslbsite_10.200.10.10:(CA_gslb_site)' MEP DOWN Mon Jul 22 18:11:51 2019
6689749    0 PPE-1 Site 'gslbsite_10.200.10.10:(CA_gslb_site)' MEP DOWN Mon Jul 22 18:11:51 2019

Same on the other ADC:
7075824    14 PPE-0 Site 'gslbsite_10.150.10.10:(NY_gslb_site)' MEP UP Mon Jul 22 18:11:23 2019
6098907     0 PPE-1 Site 'gslbsite_10.150.10.10:(NY_gslb_site)' MEP UP Mon Jul 22 18:11:23 2019
7075830     7 PPE-0 Site 'gslbsite_10.150.10.10:(NY_gslb_site)' MEP DOWN Mon Jul 22 18:11:28 2019
6098913     0 PPE-1 Site 'gslbsite_10.150.10.10:(NY_gslb_site)' MEP DOWN Mon Jul 22 18:11:28 2019
7075846    14 PPE-0 Site 'gslbsite_10.150.10.10:(NY_gslb_site)' MEP UP Mon Jul 22 18:11:45 2019
6098925     0 PPE-1 Site 'gslbsite_10.150.10.10:(NY_gslb_site)' MEP UP Mon Jul 22 18:11:45 2019
7075852     0 PPE-0 Site 'gslbsite_10.150.10.10:(NY_gslb_site)' MEP DOWN Mon Jul 22 18:11:50 2019
6098931     0 PPE-1 Site 'gslbsite_10.150.10.10:(NY_gslb_site)' MEP DOWN Mon Jul 22 18:11:50 2019

ADC nsconmsg counter, displaying MEP eventually timed out:
# nsconmsg -K /var/nslog/newnslog -g gslb_err_sitemetric_mep_timedout -d current | more
    245   21001          17739          1        0 gslb_err_sitemetric_mep_timedout gslbsite_10.200.10.10:(CA_gslb_site) Mon Jul 22 18:11:10 2019
    246   21001          17740          1        0 gslb_err_sitemetric_mep_timedout gslbsite_10.200.10.10:(CA_gslb_site) Mon Jul 22 18:11:31 2019
    247   21001          17741          1        0 gslb_err_sitemetric_mep_timedout gslbsite_10.200.10.10:(CA_gslb_site) Mon Jul 22 18:11:52 2019
    248   21000          17742          1        0 gslb_err_sitemetric_mep_timedout gslbsite_10.200.10.10:(CA_gslb_site) Mon Jul 22 18:12:13 2019

GSLB MSG sent/received:
# nsconmsg -K /var/nslog/newnslog -g gslb_tot_gslb_msg_sent -g gslb_tot_gslb_msg_rcvd -d current | more
   1206    6996       19932630         11        1 gslb_tot_gslb_msg_sent Mon Jul 22 18:11:31 2019
   1207    7000       19932642         12        1 gslb_tot_gslb_msg_sent Mon Jul 22 18:11:38 2019
   1208    7001       19932651          9        1 gslb_tot_gslb_msg_sent Mon Jul 22 18:11:45 2019
   1209    7000        1372400          2        0 gslb_tot_gslb_msgs_rcvd Mon Jul 22 18:11:52 2019
   1210       0       19932664         13        1 gslb_tot_gslb_msg_sent Mon Jul 22 18:11:52 2019
   1211    7000       19932674         10        1 gslb_tot_gslb_msg_sent Mon Jul 22 18:11:59 2019
   1212    7000       19932685         11        1 gslb_tot_gslb_msg_sent Mon Jul 22 18:12:06 2019
   1213       0        1372402          2        0 gslb_tot_gslb_msgs_rcvd Mon Jul 22 18:12:06 2019
   1214    7000        1372404          2        0 gslb_tot_gslb_msgs_rcvd Mon Jul 22 18:12:13 2019
   1215       0       19932697         12        1 gslb_tot_gslb_msg_sent Mon Jul 22 18:12:13 2019
   1216    7001       19932707         10        1 gslb_tot_gslb_msg_sent Mon Jul 22 18:12:20 2019

Trace Analysis:

Trace indicates reset with code 9829

9829	Reset on GSLB other site down or out of reach

Another trace indicating RST code 9849

9849	Reset on GSLB conflict due to mis configuration

Resolution

The analysis indicates GSLB MEP flap is due to a networking problem (networking configuration and/or Firewall issues). In this scenario, there was a Palo Alto Firewall between the two GSLB sites. Even though the Palo Alto Firewall had TCP port 3011 open bi-directional, it will still deny, if the application does not fit.

On the Palo Alto Firewall, changing the settings to “any app” allows the Citrix MEP protocol. It appears that without this "any app" setting, Palo Alto Firewall doesn't recognize this type of network traffic.

Note: The issue does not indicate GSLB MEP port 3011 is not open bi-directional, since MEP actually turned UP (even though it was flapping). If it were a port blocking issue, MEP would never turn UP.

Problem Cause

Problem Cause by Palo Alto mis-configuration

Issue/Introduction

This is a case study for Citrix ADC experiencing GSLB MEP Flapping

Additional Information

Citrix ADC Troubleshooting GSLB MEP and Sync Cheat Sheet

Welcome to "KB Articles"