Multiple SSL Performance Tests on NetScaler Display a Significant Difference in the CPU Usage Level
Article ID: CTX121595
Updated On:
Description
A user reported that after performing multiple Secure Socket Layer (SSL) performance tests on the appliance, a significant difference in the CPU usage level was observed.
Resolution
To troubleshoot the issue, collect the following logs and data from the NetScaler appliance:
-
The ns.conf file.
-
The newnslog file with the time stamp coinciding with the time when the performance tests were performed.
-
The CPU profiling data recorded during the high CPU usage.
To analyze the available data, the engineers performed the following tasks:
-
The engineers plotted a graph with CPU usage counter and identified two performance tests with significant difference in the CPU usage, as shown in the following screen shot:
-
To get an overview of the traffic pattern from the request perspective, the engineers plotted a graph for the values of the si_tot_Requests and si_tot_RequestBytes variables of the SSL VServer on which the performance tests were performed. The engineers noticed that the there were quite less number of requests in the second test, as shown in the following screen shots:
-
To get an overview of the traffic pattern from the request perspective, the engineers also plotted a graph for the values of the si_tot_Responses and si_tot_ResponseBytes variables of the SSL VServer on which the performance tests were performed. The engineers noticed that the average size for each HTTP response was quite larger in the second performance test, as shown in the following screen shots:
-
The engineers plotted a graph with the response and request bytes on the same scale and noticed that the request data was negligible. The response data was the dominant factor in terms of the SSL encoding and decoding, as shown in the following screensshot:
Problem Cause
The NetScaler appliance 9800 model consists of a Broadcom uBsec SSL accelerator. As compared to the Cavium SSL accelerator, the uBsec has the lesser processing capabilities. Therefore, the NetScaler appliance includes designs that are specific for the uBsec accelerator.
The NetScaler appliance uses the following decision logics to distribute the major SSL tasks between the uBsec SSL accelerator and the NetScaler software:
-
The most asymmetric encoding and decoding tasks that are essential for SSL connection establishment are offloaded by the SSL accelerator.
-
After an SSL connection is established, the major SSL tasks are encoding and decoding data by using the common secret negotiated algorithm during the connection establishment phase.
-
The Message Authentication Code (MAC) address related hash computations of the SSL record are done by the NetScaler software, which uses quite some time of the CPU cycles.
-
For the inbound traffic, the NetScaler software takes care of the data decryption jobs.
-
When handling outbound bulk data, the NetScaler appliance verifies if the SSL accelerator has the processing bandwidth. If it does, then the encryption work is offloaded by the SSL accelerator. Otherwise, the NetScaler software performs such tasks. Therefore, during a heavy SSL traffic, relatively high CPU usage is expected due to computation intensive nature of the SSL tasks.
After correlating the significant difference of response data rate of the two performance tests and decision logics of the NetScaler appliance to distribute the major SSL tasks between the uBsec SSL accelerator and the NetScaler software, the engineers concluded that the encoding and decoding activities for the large amount of data by the NetScaler software resulted in a high CPU usage.
The following CPU profiling data confirms the inference of the engineers:
Index HitRatio Hits TotalHit% Length Symbol name ============================================================= 1 33.396% 194988 33.396% 1712 md5_block_asm_host_order 2 24.528% 143211 57.923% 968 _rc4s 3 16.630% 97097 74.553% 124 ns_bcopy 4 1.940% 11329 76.494% 13488 nstcp_input 5 1.436% 8383 77.929% 72 generic_bcopy ============================================================= 5 100.000 % 583873
Issue/Introduction
