Problem Definition
After installing one WANScaler, the customer's Border Gateway Protocol (BGP) failed and could not re-establish their routing information. When the WANScaler was disabled, routing information was exchanged.
Environment
There were two branch offices; each office had a WANScaler installed inline between a router and switch.
A WANScaler 8510 was being used with firmware version 4.1.3. Softboost mode was enabled. However, only one WANScaler was enabled when BGP started to fail.
The customer’s network used BGP. BGP is used to share routing information between the different autonomous systems. BGP uses TCP port 179 for establishing connections. A BGP-speaking device shares network reach ability information with neighboring BGP-speaking devices The network reach ability information contains data on all of the different autonomous systems it has traversed. This information is then used by the BGP-speaking device to create a graph, or tree, of all the autonomous systems in use. The tree then allows BGP to remove routing loops and enforce certain policies for its autonomous system.
Troubleshooting Methodology
Troubleshooting began with gathering traces from the customer. The traces showed that packets were being dropped after passing the WANScaler. Why the packets were being dropped was not clear at that time.
Because BGP was failing, it was the prime suspect. We researched the protocol, specifically focusing on initialization and how routing information is exchanged. We found that when BGP is initialized, it analyzes the network and determines the best path for routing. Because BGP tries to find the nearest router, it sends packets with a Time-to-live life (TTL) of 1 by default.
We verified that the TTL was set to 1 by looking at the PCAP files of the trace file received earlier. Using Wireshark to view the Internet Protocol portion of a packet showed that the TTL was set to 1 when read by the WANScaler. After leaving the WANScaler, the TTL was decremented to 0 and discarded by the next receiving computer.
The WANScaler has a large number of configurable parameters located at <server ip>/parameters.php. Searching for TTL found a setting named IP.EnforceTTL, which can be enabled or disabled. Someone had recommended disabling the setting to the customer. This was found by speculation because no documentation exists for the parameters.php change.
Resolution
This case was resolved by changing the IP.EnforceTTL setting in the parameters page of the WANScaler. Once changed, the customer reported that BGP exchanged routing information correctly.
A second resolution existed that could have also resolved the issue. Raising the TTL value by one would also have solved the problem. Because the WANScaler decrements the TTL value by default, the other solution would be to reconfigure the BGP and set the default TTL to 2. This was not proposed to the customer because the WANScaler itself had the setting to not decrement TTL and changing this parameter was easier than having the customer reconfigure their network.
References
CCNP: Building Scalable Cisco Internetworks Study Guide (Exam 642-801) by Carl Timm and Wade Edwards ISBN:0782142931
