Note: This article applies to Citrix SD-WAN WANOP.

The following are high level troubleshooting steps for WCCP cluster deployment issues:

1. Verify the following:

   • Has deployment planning been performed? Refer to Citrix Documentation - Preparing for Your Deployment for more information.

   • Has the number and models of the CloudBridge appliance been determined? Refer to Citrix Documentation - Selecting Appliances for more information.

   • Have the two WCCP Service Group (SG) been selected? Refer to Citrix Documentation - Quick Start WCCP Clustering Guide for more information.

   • Has the router been configured? Refer to Citrix Citrix Documentation - Configuring the Router and Quick Start WCCP Clustering Guide for more information.

   • Have the CloudBridge WCCP configuration been determined, especially MaskValue, DC algorithm, and forwarding method?
     Refer to Citrix Documentation - Load-Balancing in the WCCP Cluster, Assigning Buckets to Appliances and Quick Start WCCP Clustering Guide for more information.

   • Have the CloudBridge appliances and routers been configured? Refer to Citrix Documentation - Limitations and Quick Start WCCP Clustering Guide for more information.

   • Is there connectivity between these devices? Refer to Citrix Documentation - Testing and Troubleshooting for more information.

2. If there are multiple routers, focus on a single router and verify the following on that router then repeat for any additional routers one at a time.

   1. Pick a CloudBridge appliance that is a member of the WCCP cluster.  

   2. Go to the WCCP Monitoring page called Cache Status under Monitoring > Appliance Performance > WCCP as shown in the following screen shot.

   3. Examine the specific SG pair.  

   4. If the status shows as Has Assignment then go to Step 5.

      

3. Verify the appliance’s router status under Monitoring > Appliance Performance > WCCP Cache Status. Refer to the status in Table 1 - Router Status Field on WCCP Monitoring Page.
   This status page does not automatically refresh, so a manual refresh is required to determine the latest status.

   1. If it has a Blue text as described in Table 1 - Router Status Field on WCCP Monitoring Page, then it is an SG configuration error. Determine the cause of the error and correct it.
      The following are two examples:

      1. WCCP might be disabled.

         

      2. The interface selected in the SG configuration is incorrect.  

         

   2. If the table entry has a notification, then verify the appliance’s notifications and look up that condition in Table 2 - WCCP Notifications. This table describes the problem and possible corrections. Correct the condition. Refer to CTX200319 and CTX200412 for more information.

   3. If the CloudBridge appliance WCCP monitoring status page has the selected router and SG pair with a "No response from Router" status then the possible reasons and corrections for this failure are listed in this point. The following are the log messages:

      

      1. If there is no connectivity to the router, then perform the following steps:

         • Ping the router from the CloudBridge appliance’s traffic interface used in the configuration, such as apA.  

         • Verify if the interface is enabled and if the correct interface is selected.

         • If the router is on the same subnet as the interface, and ping fails, then examine wiring or any intermediate switch. Correct the condition.

         • If the router is on a different subnet, then the interface on any appliance other than a CloudBridge4000/5000 needs a default gateway on the interface, or on the NetScaler component is a CloudBridge4000/5000 appliance then a static route.

         • In either case ping the gateway and then on that gateway device ping the router. Might be the router needs a static route for the route back to the appliance.

      2. The router has learned SG characteristics from another CloudBridge appliance, and this appliances does not match. Possible items are Password, Priority, Protocol, a reversal of WAN and LAN SGs and hence the direction is a mismatch. An unlikely possibility is that there are already 32 caches in the SG. Correct this appliance’s configuration.

   4. If the status is "SG has socket send error", a reboot of the appliance should resolve this issue. If the issue persists, collect the support bundle and contact Citrix technical support. Refer to Citrix Documentation - Generating a Tar Archive for Technical Support for more information.

   5. If the status is "Needs Assignment" or "Waiting for DC to Assign", this might be temporary, however, if this condition continues (beyond a minute), then the Designated Cache (DC) has decided not to provide an assignment to the router on behalf of this appliances. In this case, go to the WCCP router's monitoring page (Monitoring > Appliance Performance > WCCP (Cache x) > Routers), as shown in the following screen shot. From this page determine the DC. Then examine that DC appliance and view the logs to determine why the DC did not assign any mask elements to the original CloudBridge appliance that has the "Needs Assignment" router status. (It is possible that the original appliance is the DC and it did not give itself an assignment.)

      Generally the reason is that the "Mask Value" has too few bits that are set, effectively it is too small so there are not enough mask elements to distribute to all SG members (see the following screen shots).

      Another reason is that the appliance does not have the same release as the DC. If there is no DC after a few minutes, then:

            - Take a PCAP trace of the cache with the lowest IP address, and also another cache of the SG pair.

            - Create support bundle (or extraction of the logs) of these two appliances. Refer to Citrix Documentation - Generating a Tar Archive for Technical Support for more information.

      It might also be self-correcting if the cache with the lowest IP has WCCP disabled for 40 seconds, then re-enabled. If the status is "Needs Assignment" or "Waiting for DC to Assign", other cluster analysis can be verified, even when not resolving the missing assignment of this appliance. To continue the analysis go to Step 5.

      

      DC Cache Status:

      

      SC1 Cache Status:

      

      SC2 Cache Status:

      

4. Verify the appliance’s Notifications. If there are any WCCP related notifications on either of the SGs in the pair on the selected router, then look up that condition in Table 2 - WCCP Notifications. This table describes the problem and possible corrections. Correct the condition. Refer to CTX200319 and CTX200412 for more information.

5. If the CloudBridge appliance has "Has Assignment" or "Needs Assignment" or "Waiting for DC to Assign" as its status on the WCCP monitoring page then look at all members of the cluster. Go to, Monitoring > Appliance Performance > WCCP (Cache X) > Cluster Summary. If this appliance is a CloudBridge4000/5000, then examining either of the cache pair would be fine (the status on either is 99% identical). An example of this page is shown in the following screen shot. The line with the "Handshake" represents other caches (CloudBridge appliances) in the cluster. Examine this list (for the selected router and SG pair) and determine if all expected caches are present. If there are any missing, then go to that appliance’s management IP and examine the WCCP status. Follow this procedure starting at Step 2 on that appliance to resolve why it is not part of this cluster.

   In the list of original appliances if any of the members of this list has a router status as "Not Seen", then that cache (CloudBridge appliance) is no longer part of the cluster. This might be temporary, such as a reboot. If it is not, then follow this procedure starting at Step 2 on that appliance to resolve why it is not part of this cluster. If the router status is "Seen" but not "Assigned", then you have to determine the reason why it has no assignment. For all of the caches that have no assignment, follow the procedure described in Step 3.e by examining the DC logs.

   

6. At this step, all the SGs cache members are present and they all have assignments. The details of the assignment can be viewed on any of the status pages, however Monitoring > Appliance Performance > WCCP (Cache X) > Service Groups page provides a comprehensive view, as shown in the following screen shot. The example shows two active cache members in the 52/51 SG pair. The MaskValue is 0x1 and 110.0.10.3 has element 0; and 110.0.10.2 has element 1. The evaluation should be the load at each appliance. There is no comprehensive tool to provide this other than verifying each appliance and comparing connection, LAN, and WAN usage graphs. If any appliance is greatly loaded, and one or more are lightly loaded, then the MaskValue choice should be reevaluated. If a network component mask has been selected then a host component mask might be a better solution. If the issue cannot be resolved with a different MaskValue, possible additional appliances needs to be added to support the complete peak load and support a N+1 solution. Refer to Citrix Documentation - Planning Your Deployment and Quick Start WCCP Clustering Guide for more information.

   

7. Iterate on the other SG pairs and the other routers. When all cluster members are active on all routers, and the load is reasonably distributed, then the last step is to verify if there are two or more routers in the same SG pair. In this case the "HSRP Deterministic" algorithm should be selected to support migration of connection between routers. The "Least Disruptive" algorithm should only be used when there is a single router.

Table 1 - Router Status Field on WCCP Monitoring Page

This table shows the Router Status as shown on each router in the following diagram. The status conditions that are shown as configuration errors are colored Blue, conditions representing some internal errors are in Purple and operational determined errors are shown in Orange.

The following diagram indicates a typical flow of router status presented to the user on the WCCP status page for a particular router in a particular SG.
Note: Some of these status messages are brief, and a user might not see all the status messages from "Contacting Router" to "Has Assignment".

Table 2 - WCCP Notifications