What is a GSLB Service?

A GSLB service is usually a representation of a load balancing or content switching virtual server, although it can represent any type of virtual server or 3rd party load balancer. The GSLB service identifies the service’s IP address, port number, and service type, and defines the site to which the service is local.

GSLB services are bound to GSLB virtual servers on the ADC appliances managing the GSLB sites. A local GSLB service represents a local load balancing or content switching virtual server, or non-ADC server. A remote GSLB service represents a load balancing or content switching virtual server or non-ADC server configured locally at one of the other sites in the GSLB setup. At each site in the GSLB setup, you can create one local GSLB service and any number of remote GSLB services for each GSLB vServer.

GSLB Service Health

GSLB vServer load balancing decisions depend on the health of the bound GSLB Services.  
The health of a GSLB Service is controlled either using the Metric Exchange Protocol (MEP) or binding one or more explicit monitors.
MEP is a Citrix proprietary protocol used by GSLB to provide network, load, and persistence information. This information is used to determine the DNS query responses. MEP is enabled by default and begins exchanging metric information as soon as the GSLB site is created.
Binding explicit monitors to local GSLB services is not required, because the state of the local GSLB service is updated by default using MEP. However, there are certain conditions where monitors are needed:
•    If the local service represents a non-ADC server.
•    If the local service represents a vServer in a non-default Traffic Domain.
If one or more monitors are bound to a local GSLB service, the health of this service is controlled by these monitors instead of MEP.

Trigger Monitors

If one or more monitors are bound to remote GSLB Services, the health of these services will be determined by either MEP or monitors depending on how the ‘Trigger Monitors’ option is set on the respective GSLB Site:
•    ALWAYS: The health of GSLB Services that refer to services hosted on this site are always controlled by the configured monitors.
•    MEPDOWN – The health of GSLB Services that refer to services hosted on this site will be controlled by the configured monitors only if the Site Metric MEP Status is DOWN/DISABLED.
•    MEPDOWN_SVCDOWN - The health of GSLB Services that refer to services hosted on this site will be controlled by the configured monitors under two conditions:

Binding Monitors

When binding explicit monitors to GSLB Services (either remote or local), it is recommended to use application level monitors such as HTTP/S to ascertain the actual state of the vServer/application hosted at that site. By default, if TCP or PING monitors are used then the probe will be successful even if the respective vServer is DOWN – this is expected behaviour. If an LB vServer is part of GSLB, the monitor should probe for the application data so that we can send a redirect to an healthy and optimal GSLB service and prevent timeouts.  
If ICMP is the only usable monitor in your environment, then you can get around this behaviour by configuring the VIP’s ICMP Response to ‘VSVR_CNTRLD’ and set the ICMP Virtual Server Response to ACTIVE in the vServer’s Traffic Settings:

Disabling Health Monitoring

On disabling the health monitor for the GSLB service, the service main state will always remain UP irrespective of the state learnt through MEP. However, this service will not be considered for the GSLB decision if it is disabled on the local site, as we send that update through MEP and that information is maintained internally. So in effect, while the state may be UP, the ADC is aware that the true health, learned through MEP, is DOWN and will therefore not consider this service in its load balancing decision.

Site to Site Communication

GSLB site to site communication, such as synchronization and MEP, takes place between the Remote Procedure Call (RPC) nodes. When you add a GSLB site, the ADC automatically adds the RPC node under System – Network – RPC. During this process, the RPC node is assigned an internally generated user name and password. This is used to authenticate the node during the connection stage.
Furthermore, during the automatic setup of the RPC node, no source IP is defined for sending GSLB requests. Therefore, the NSIP address is used, provided it has a route to the remote GSLB public IP. If no route is available, it will send from a SNIP/MIP that is in the same subnet as the remote GSLB IP.
You can specify what source IP should be used for a particular remote RPC node. Communication between sites can be sourced from the NSIP, SNIP/MIP/GSLB or even a VIP. You can choose to encrypt RPC communication by enabling the secure option on the RPC node. RPC communication uses TCP port 3011 and secure RPC uses port 3009.
You can also change the password for the local RPC node and manually propagate this to the other remote RPC nodes.
All sites taking part in metric exchange should have the same nsroot user ID and password. A mismatch in passwords will cause MEP failures.

Site Metric Exchange

In site metric exchanges, the connection is always initiated by the GSLB site with the lower NSIP address. The node that initiates the connection refers to its list of RPC nodes to determine which source IP it should use to communicate with the remote site. It sends its MEP packets from an ephemeral port to the public GSLB IP address of the remote site on port 3011 (or 3009 for secure MEP). Site metrics are sent every second.

Network Metric Exchange

With network metric exchanges, the connection can be initiated by any of the sites involved in the exchange. MEP communication still initiates from a specified source IP address and is sent to the remote site’s public GSLB IP address on either port 3011 or 3009. Network metric exchanges take place every 5 seconds.

GSLB Service State v Effective State

A GSLB service has a ‘state’ and an ‘effective state’. The ‘effective state’ is determined by the metric exchange protocol. State can be determined either by MEP or by an explicit monitor.
If the effective state is Down but the State is UP, then this could mean that MEP communication has failed but an explicit monitor that is bound to the GSLB service has succeeded.
Reasons for why the effective state may show as DOWN can include;
•    MEP is disabled.
•    Communication between the necessary IP addresses and ports for MEP has failed.
•    The GSLB service represents a third party load balancer.
•    The virtual server that the GSLB service represents is in a non-default traffic domain.
•    Site metrics between Parent & Child is DOWN but Parent to Backup Parent is UP.

MEP Flaps

MEP flaps may occur for the following reasons:
•    Network issues between sites
•    HA failovers
•    Parent site failover to Backup.
In such circumstances, the actual state of the GSLB Services may be healthy yet traffic will no longer be sent to these until the Site Metric MEP status returns to UP. To reduce the impact to GSLB services caused by MEP flaps, you can configure GSLB Service State Delay Time as a GSLB parameter on each node.

MEP & Monitors in Parent-Child Deployments

In a Parent-Child deployment, MEP traffic flows between Parents and their own Children, and between Parent sites.
When Child sites are added to Parents, the Site Metric MEP Status for Children belonging to other Parents will be reported as DOWN. This is because the site metrics are controlled by that Child’s Parent, and this is expected behaviour:

When creating a GSLB Service on the Parent, you specify which Child site this service belongs to.
Services hosted on Child sites are added as remote GSLB Services on all Parent sites.

This method allows Parents to know whether the health of the GSLB Service will come from one of it’s Children or from another Parent, via MEP.
If explicit monitors are bound to remote GSLB Services on Parent sites, these are triggered depending on how the ‘Trigger Monitors’ option is set for each site.
For instance, in the below diagram, the ‘Trigger Monitors’ option is set to MEPDOWN on each site. In this case, even if monitors are bound to GSLB services, the health of the remote services is dictated by MEP, as MEP is UP (green). As Parent1 has a local GSLB Service with an explicit monitor bound, this monitor will always be used to control its health yet other Parent sites will receive this health information via MEP:

The monitor ‘Last Response’ at the other Parent sites will show the following:

Trigger Monitor: ALWAYS

If the ‘Trigger Monitor’ option is set to ALWAYS on each site, it will be the explicit monitor that controls the health of the GSLB Services (if bound), at all Parent sites:

Trigger Monitor: MEPDOWN

In the event that MEP communication between Parent and Child fails, and the Trigger Monitor option is set to MEPDOWN for each site, only the Child’s Parent will trigger a bound monitor for any GSLB Services hosted on that child. Other Parents will receive that health information via MEP. The Effective State for those GSLB Services will be DOWN and the State will be UP.

In the event that a Parent site goes DOWN completely, the Backup Parent will detect this through MEP and initiate a connection with their Children to begin the adoption process. During this time, if the Trigger Monitor option is set to MEPDOWN for each site, all Parents (including the backup) will probe the GSLB Services hosted on these Children with the bound monitors. When the Backup Parent adoption process is complete, the health of these GSLB Services will be sent via MEP from the Child to Backup Parent, and monitors will stop triggering. The health of any GSLB Services that were local to the failed Parent will be determined by any bound monitors on all other Parent sites.

Trigger Monitor: MEPDOWN_SVCDOWN

In the below diagram, the lb vServer hosted on P1Child1 is DOWN yet the Site Metric MEP Status for P1Child1 on Parent1 is UP. If the Trigger Monitors option is set to MEPDOWN_SVCDOWN for each site, Parent1 would use any bound monitors to check on the health of the GSLB Service for P1Child1LB service, but the other Parents will continue to receive this information via MEP.

For more information on Parent-child topology deployment using the MEP protocol, click here.

Troubleshooting GSLB Cheat Sheet

Check out the link below for more info on troubleshooting GSLB MEP:
https://support.citrix.com/article/CTX244517