High Packet CPU caused by icmp traffic on loopback IP address 127.0.0.2

High Packet CPU caused by icmp traffic on loopback IP address 127.0.0.2

book

Article ID: CTX236572

calendar_today

Updated On:

Description

On the NetScaler checking the CPU stats you may find similar output as below for CPU stat command on CLI:
>stat system cpu
 CPU statistics
ID         Usage
4            100
5            100
3            100
2            100
1            100
 
In above case this NetScaler has 5 Packet CPU and the output will be shown as per the number of Packet CPU's.

Checking the newnslog counters (under /var/nslog folder) will show that there are a lot of loopback packets on the NetScaler which is higher than the traffic interfaces:
#nsconmsg -K newnslog -d current -s disptime=1 -g nic_tot_rx_packets | more
1       0        9189639189 nic_tot_rx_packets interface(10/1)
    3       0        9129888925 nic_tot_rx_packets interface(10/2)
    5       0       86622975465 nic_tot_rx_packets interface(10/3)
    7       0       86824396495 nic_tot_rx_packets interface(10/4)
    9       0        3525761411 nic_tot_rx_packets interface(0/1)
   11       0      681495258986 nic_tot_rx_packets interface(LO/1)
   13       0       18319518984 nic_tot_rx_packets interface(LA/1)
   15       0      173447330798 nic_tot_rx_packets interface(LA/2)
   17       0      195292661485 allnic_tot_rx_packets

If you do a NetScaler packet capture, you can identify that looping one particular packet is causing high CPU and most of the traffic on NetScaler is the packet as mentioned below:
NSIP -> 127.0.0.2    ICMP 105 Destination unreachable (Port unreachable)
Also you could identify that the actual traffic on the NetScaler is very low compared to mentioned looped packet.
User-added image
 

This is seen when the configured nameServer returns a server failure response and the packet is looped into NetScaler.

Resolution

Workaround:
Execute below command in NetScaler CLI, the command will drop such packets causing a loop within the NetScaler
>set ns rateControl -tcpThreshold 233 -icmpThreshold 100
Fix: 
The fix for the issue has been applied to 11.1 59.X, 12.0 59.X and 12.1 49.X builds, which will be released in Q3 2018. 
 

Problem Cause

Caused by issue #707489, where ICMP error packet destined to 127.0.0.2 is looping in NetScaler and causing high CPU.

Issue/Introduction

NetScaler is reporting high packet CPU continuously and performing a reboot or failover works fine for some time(for a few days) and then packet CPU uses starts increasing again and even when the actual user traffic is very low.