Citrix ADC experiencing GSLB MEP flapping, every 7-10 seconds goes UP/DOWN
Article ID: CTX258994
Updated On:
Description
Log Analysis:
ADC Events displaying the flapping:
# nsconmsg -K /var/nslog/newnslog -d event | grep -i mep
7741410 0 PPE-0 Site 'gslbsite_10.200.10.10:(CA_gslb_site)' MEP UP Mon Jul 22 18:11:23 2019
6689704 0 PPE-1 Site 'gslbsite_10.200.10.10:(CA_gslb_site)' MEP UP Mon Jul 22 18:11:23 2019
7741421 0 PPE-0 Site 'gslbsite_10.200.10.10:(CA_gslb_site)' MEP DOWN Mon Jul 22 18:11:29 2019
6689715 0 PPE-1 Site 'gslbsite_10.200.10.10:(CA_gslb_site)' MEP DOWN Mon Jul 22 18:11:29 2019
7741440 7 PPE-0 Site 'gslbsite_10.200.10.10:(CA_gslb_site)' MEP UP Mon Jul 22 18:11:45 2019
6689730 0 PPE-1 Site 'gslbsite_10.200.10.10:(CA_gslb_site)' MEP UP Mon Jul 22 18:11:45 2019
7741459 0 PPE-0 Site 'gslbsite_10.200.10.10:(CA_gslb_site)' MEP DOWN Mon Jul 22 18:11:51 2019
6689749 0 PPE-1 Site 'gslbsite_10.200.10.10:(CA_gslb_site)' MEP DOWN Mon Jul 22 18:11:51 2019
Same on the other ADC:
7075824 14 PPE-0 Site 'gslbsite_10.150.10.10:(NY_gslb_site)' MEP UP Mon Jul 22 18:11:23 2019
6098907 0 PPE-1 Site 'gslbsite_10.150.10.10:(NY_gslb_site)' MEP UP Mon Jul 22 18:11:23 2019
7075830 7 PPE-0 Site 'gslbsite_10.150.10.10:(NY_gslb_site)' MEP DOWN Mon Jul 22 18:11:28 2019
6098913 0 PPE-1 Site 'gslbsite_10.150.10.10:(NY_gslb_site)' MEP DOWN Mon Jul 22 18:11:28 2019
7075846 14 PPE-0 Site 'gslbsite_10.150.10.10:(NY_gslb_site)' MEP UP Mon Jul 22 18:11:45 2019
6098925 0 PPE-1 Site 'gslbsite_10.150.10.10:(NY_gslb_site)' MEP UP Mon Jul 22 18:11:45 2019
7075852 0 PPE-0 Site 'gslbsite_10.150.10.10:(NY_gslb_site)' MEP DOWN Mon Jul 22 18:11:50 2019
6098931 0 PPE-1 Site 'gslbsite_10.150.10.10:(NY_gslb_site)' MEP DOWN Mon Jul 22 18:11:50 2019
ADC nsconmsg counter, displaying MEP eventually timed out:
# nsconmsg -K /var/nslog/newnslog -g gslb_err_sitemetric_mep_timedout -d current | more
245 21001 17739 1 0 gslb_err_sitemetric_mep_timedout gslbsite_10.200.10.10:(CA_gslb_site) Mon Jul 22 18:11:10 2019
246 21001 17740 1 0 gslb_err_sitemetric_mep_timedout gslbsite_10.200.10.10:(CA_gslb_site) Mon Jul 22 18:11:31 2019
247 21001 17741 1 0 gslb_err_sitemetric_mep_timedout gslbsite_10.200.10.10:(CA_gslb_site) Mon Jul 22 18:11:52 2019
248 21000 17742 1 0 gslb_err_sitemetric_mep_timedout gslbsite_10.200.10.10:(CA_gslb_site) Mon Jul 22 18:12:13 2019
GSLB MSG sent/received:
# nsconmsg -K /var/nslog/newnslog -g gslb_tot_gslb_msg_sent -g gslb_tot_gslb_msg_rcvd -d current | more
1206 6996 19932630 11 1 gslb_tot_gslb_msg_sent Mon Jul 22 18:11:31 2019
1207 7000 19932642 12 1 gslb_tot_gslb_msg_sent Mon Jul 22 18:11:38 2019
1208 7001 19932651 9 1 gslb_tot_gslb_msg_sent Mon Jul 22 18:11:45 2019
1209 7000 1372400 2 0 gslb_tot_gslb_msgs_rcvd Mon Jul 22 18:11:52 2019
1210 0 19932664 13 1 gslb_tot_gslb_msg_sent Mon Jul 22 18:11:52 2019
1211 7000 19932674 10 1 gslb_tot_gslb_msg_sent Mon Jul 22 18:11:59 2019
1212 7000 19932685 11 1 gslb_tot_gslb_msg_sent Mon Jul 22 18:12:06 2019
1213 0 1372402 2 0 gslb_tot_gslb_msgs_rcvd Mon Jul 22 18:12:06 2019
1214 7000 1372404 2 0 gslb_tot_gslb_msgs_rcvd Mon Jul 22 18:12:13 2019
1215 0 19932697 12 1 gslb_tot_gslb_msg_sent Mon Jul 22 18:12:13 2019
1216 7001 19932707 10 1 gslb_tot_gslb_msg_sent Mon Jul 22 18:12:20 2019
Trace Analysis:
Trace indicates reset with code 9829
Another trace indicating RST code 9849
ADC Events displaying the flapping:
# nsconmsg -K /var/nslog/newnslog -d event | grep -i mep
7741410 0 PPE-0 Site 'gslbsite_10.200.10.10:(CA_gslb_site)' MEP UP Mon Jul 22 18:11:23 2019
6689704 0 PPE-1 Site 'gslbsite_10.200.10.10:(CA_gslb_site)' MEP UP Mon Jul 22 18:11:23 2019
7741421 0 PPE-0 Site 'gslbsite_10.200.10.10:(CA_gslb_site)' MEP DOWN Mon Jul 22 18:11:29 2019
6689715 0 PPE-1 Site 'gslbsite_10.200.10.10:(CA_gslb_site)' MEP DOWN Mon Jul 22 18:11:29 2019
7741440 7 PPE-0 Site 'gslbsite_10.200.10.10:(CA_gslb_site)' MEP UP Mon Jul 22 18:11:45 2019
6689730 0 PPE-1 Site 'gslbsite_10.200.10.10:(CA_gslb_site)' MEP UP Mon Jul 22 18:11:45 2019
7741459 0 PPE-0 Site 'gslbsite_10.200.10.10:(CA_gslb_site)' MEP DOWN Mon Jul 22 18:11:51 2019
6689749 0 PPE-1 Site 'gslbsite_10.200.10.10:(CA_gslb_site)' MEP DOWN Mon Jul 22 18:11:51 2019
Same on the other ADC:
7075824 14 PPE-0 Site 'gslbsite_10.150.10.10:(NY_gslb_site)' MEP UP Mon Jul 22 18:11:23 2019
6098907 0 PPE-1 Site 'gslbsite_10.150.10.10:(NY_gslb_site)' MEP UP Mon Jul 22 18:11:23 2019
7075830 7 PPE-0 Site 'gslbsite_10.150.10.10:(NY_gslb_site)' MEP DOWN Mon Jul 22 18:11:28 2019
6098913 0 PPE-1 Site 'gslbsite_10.150.10.10:(NY_gslb_site)' MEP DOWN Mon Jul 22 18:11:28 2019
7075846 14 PPE-0 Site 'gslbsite_10.150.10.10:(NY_gslb_site)' MEP UP Mon Jul 22 18:11:45 2019
6098925 0 PPE-1 Site 'gslbsite_10.150.10.10:(NY_gslb_site)' MEP UP Mon Jul 22 18:11:45 2019
7075852 0 PPE-0 Site 'gslbsite_10.150.10.10:(NY_gslb_site)' MEP DOWN Mon Jul 22 18:11:50 2019
6098931 0 PPE-1 Site 'gslbsite_10.150.10.10:(NY_gslb_site)' MEP DOWN Mon Jul 22 18:11:50 2019
ADC nsconmsg counter, displaying MEP eventually timed out:
# nsconmsg -K /var/nslog/newnslog -g gslb_err_sitemetric_mep_timedout -d current | more
245 21001 17739 1 0 gslb_err_sitemetric_mep_timedout gslbsite_10.200.10.10:(CA_gslb_site) Mon Jul 22 18:11:10 2019
246 21001 17740 1 0 gslb_err_sitemetric_mep_timedout gslbsite_10.200.10.10:(CA_gslb_site) Mon Jul 22 18:11:31 2019
247 21001 17741 1 0 gslb_err_sitemetric_mep_timedout gslbsite_10.200.10.10:(CA_gslb_site) Mon Jul 22 18:11:52 2019
248 21000 17742 1 0 gslb_err_sitemetric_mep_timedout gslbsite_10.200.10.10:(CA_gslb_site) Mon Jul 22 18:12:13 2019
GSLB MSG sent/received:
# nsconmsg -K /var/nslog/newnslog -g gslb_tot_gslb_msg_sent -g gslb_tot_gslb_msg_rcvd -d current | more
1206 6996 19932630 11 1 gslb_tot_gslb_msg_sent Mon Jul 22 18:11:31 2019
1207 7000 19932642 12 1 gslb_tot_gslb_msg_sent Mon Jul 22 18:11:38 2019
1208 7001 19932651 9 1 gslb_tot_gslb_msg_sent Mon Jul 22 18:11:45 2019
1209 7000 1372400 2 0 gslb_tot_gslb_msgs_rcvd Mon Jul 22 18:11:52 2019
1210 0 19932664 13 1 gslb_tot_gslb_msg_sent Mon Jul 22 18:11:52 2019
1211 7000 19932674 10 1 gslb_tot_gslb_msg_sent Mon Jul 22 18:11:59 2019
1212 7000 19932685 11 1 gslb_tot_gslb_msg_sent Mon Jul 22 18:12:06 2019
1213 0 1372402 2 0 gslb_tot_gslb_msgs_rcvd Mon Jul 22 18:12:06 2019
1214 7000 1372404 2 0 gslb_tot_gslb_msgs_rcvd Mon Jul 22 18:12:13 2019
1215 0 19932697 12 1 gslb_tot_gslb_msg_sent Mon Jul 22 18:12:13 2019
1216 7001 19932707 10 1 gslb_tot_gslb_msg_sent Mon Jul 22 18:12:20 2019
Trace Analysis:
Trace indicates reset with code 9829
| 9829 | Reset on GSLB other site down or out of reach |
Another trace indicating RST code 9849
| 9849 | Reset on GSLB conflict due to mis configuration |
Resolution
The analysis indicates GSLB MEP flap is due to a networking problem (networking configuration and/or Firewall issues). In this scenario, there was a Palo Alto Firewall between the two GSLB sites. Even though the Palo Alto Firewall had TCP port 3011 open bi-directional, it will still deny, if the application does not fit.
On the Palo Alto Firewall, changing the settings to “any app” allows the Citrix MEP protocol. It appears that without this "any app" setting, Palo Alto Firewall doesn't recognize this type of network traffic.
Note: The issue does not indicate GSLB MEP port 3011 is not open bi-directional, since MEP actually turned UP (even though it was flapping). If it were a port blocking issue, MEP would never turn UP.
Problem Cause
Problem Cause by Palo Alto mis-configuration
Issue/Introduction
This is a case study for Citrix ADC experiencing GSLB MEP Flapping
Additional Information
Was this article helpful?
Yes
No
Powered by
