NetScaler 11.1 High CPU due to High Loopback Interface traffic

NetScaler 11.1 High CPU due to High Loopback Interface traffic

book

Article ID: CTX220297

calendar_today

Updated On:

Description

High CPU observed

Totalcount value 1000 = 100% CPU utilization

Index   rtime totalcount-val      delta rate/sec symbol-name&device-no&time

 1411    7006            368        349       49 cc_cpu_use cpu(1) Sun Dec  4 20:29:24 2016

 1412    7021           1000        632       90 cc_cpu_use cpu(1) Sun Dec  4 20:29:31 2016

 

Huge spike in LO/1 traffic

Rate value 1225 = 1225Mbps or 1.2Gps traffic

 Index   rtime totalcount-val   delta rate/sec symbol-name&device-no

 3723       0          33340       8606     1225 nic_tot_rx_mbits interface(LO/1) Sun Dec  4 20:29:31 2016

 3724       0          42500       8607     1225 nic_tot_tx_mbits interface(LO/1) Sun Dec  4 20:29:31 2016

 

Gather trace for five seconds - analysis:

DNS query for <hostname>  is being looped continuously causing this high traffic on LO/1.

 

7266 0.003640912  10.162.5.37 -> 127.0.0.2    DNS 123 Standard query 0x21eb A <hostname>

7267 0.003641224  10.162.5.37 -> 127.0.0.2    DNS 123 Standard query 0x21eb A <hostname>

7268 0.003641840  10.162.5.37 -> 127.0.0.2    DNS 123 Standard query 0x21eb A <hostname>

7269 0.003642144  10.162.5.37 -> 127.0.0.2    DNS 123 Standard query 0x21eb A <hostname>

 

tshark -r nstrace1.cap -T fields  -e udp.port|more

3000,53

3000,53

3000,53

3000,53

3000,53

3000,53

3000,53

3000,53

3000,53

3000,53

 

Relevant Configuration:

add authentication ldapAction ldap_action_name -serverName hostname.domain.com -ldapBase "DC=sub,DC=domain,DC=com" -ldapBindDn "domain\\domainname" -ldapBindDnPassword XXXX -encrypted -encryptmethod ENCMTHD_3 -ldapLoginName samAccountName -groupAttrName memberOf -subAttributeName CN -secType TLS -ssoNameAttribute userPrincipalName -passwdChange ENABLED -nestedGroupExtraction ON -groupNameIdentifier sAMAccountName -groupSearchAttribute memberOf -groupSearchSubAttribute CN

Resolution

WORKAROUND:

Use IP address instead of server name / hostname

PERMANENT FIX

The fix is available in the below version

11.1 53.13 and above

Problem Cause

Huge spike of traffic on Loopback interface, LO/1, which is caused by a looping DNS packet detined to port 53

Issue/Introduction

NetScaler v11.1-50.10, experienced sudden spike in High Packet Engine CPU usage