This article outlines the steps we can follow to troubleshoot the high memory issues on NetScaler .


 

Instructions

High memory utilization on the NetScaler can lead to the following consequences:
 

• If the memory  usage is increasing indefinitely then the NetScaler could crash because the complete memory is utilized 
• Increasing memory can cause memory allocation failures for certain processes ultimately leading to a certain feature not working at all /not working as expected
• NetScaler interface or accessing certain other resources This also can cause performance issues on NetScaler wherein the users can experience latency while accessing the 

e.g. Website load balanced through NetScaler

In order to find out what cause the high memory , there are several steps to follow :

Output will be as below :

> stat system memory

NetScaler Memory Information:

Maximum Memory Available (MB)                   3684

Memory Currently Available (MB)                  927

Memory Allocated (MB)                            245

Memory Allocated (%)                            6.66

InUse Memory (MB)                                166

InUse Memory (%)                                4.52

Free Memory (MB)                                3438

Maximum Allocation Failed (MB)                     0

NetScaler Shared Memory Information :

Total Shared Memory (MB)                         222

Shared Memory InUse (MB)                          22

Shared Memory InUse (%)                        10.06

Maximum Allocation Failed (MB)                     0

NetScaler Integrated Cache Memory Information :

Maximum Memory (MB)                             4662

InUse Memory (MB)                                  0

InUse Memory (%)                                0.00

This command gives a basic breakdown of the Allocated and InUse memory .

The below steps can be performed to find out the current/historic memory utilization(module wise & total) on the appliance :

* Login to the NetScaler shell >cd /var/nslog .

*Next , if we want to find the current utilization statistics , we can use the command :

nsconmsg111 -K newnslog -d memstats

>>Now in the above command , 111 is used for NetScaler Firmware 11.1, this can be replaced with 105 for NetScaler 10.5 and so on depending on the NetScaler version .

*The above command gives an output like below :

 nsconmsg111 -K newnslog -d memstats | more

Displaying current memory stats information

NetScaler V20 Performance Data

NetScaler NS11.1: Build 51.26.nc, Date: Jan 31 2017, 00:12:01

----------------------------------------------------------------------------------------------------------------------

TotalMEM:  (3806546560/4083154944)     Allocated:  3405153984(83.40%)   ActualInUse: 3137402771(76.84%)    Free:  678000960

MEMPOOL             MaxAllowd           CurAlloc                  ErrLmtFailed   ErrAllocFailed  ErrFreeFailed

                                     Bytes (Own%)(Overall%)

----------------------------------------------------------------------------------------------------------------------

MEM_PE               83886080        151744(0.18%  0.00%)          0                0            0

MEM_LB_SERVER      4294967295       2591104(0.06%  0.06%)          0                0            0

MEM_LB_SESSION      136314880         21504(0.02%  0.00%)          0                0            0

MEM_LB_SERVICE     4294967295         85568(0.00%  0.00%)          0                0            0

MEM_CSWMEM           28311552         37120(0.13%  0.00%)          0                0            0

MEM_IOH               5242880             0(0.00%  0.00%)          0                0            0

MEM_LOGGING        4294967295         65600(0.00%  0.00%)          0                0            0

MEM_CONN           4294967295     302059520(7.03%  7.40%)          0                0            0

MEM_SNMP           4294967295        139648(0.00%  0.00%)          0                0            0

MEM_DEBUG              262144          1088(0.42%  0.00%)          0                0            0

MEM_MISC           4294967295       1161472(0.03%  0.03%)          0                0            0

MEM_SERVMON        4294967295        510848(0.01%  0.01%)          0                0            0

MEM_IPFRAG         4294967295             0(0.00%  0.00%)          0                0            0

MEM_URLMON           15728640             0(0.00%  0.00%)          0                0            0

MEM_TCPBUFFP        209715200             0(0.00%  0.00%)          0                0            0

MEM_DCC              10485760             0(0.00%  0.00%)          0                0            0

Here every pool has a cap on “MaxAllowd”. 

The “CurAlloc” shows the percentage of memory used from the own pool versus overall system.

“ErrLmtFailed” notifies you the number of times you have gone over the particular pool limit

“ErrAllocFailed” shows number of times memory allocation failed for this pool.

“ErrFreeFailed” shows number of times memory free operation failed for this pool.

In the output , we can clearly see which module is utilizing what amount of memory .

If we scroll down further , it also gives the breakgae for certain generic module like MEM_CONN , MEM_SSL etc which stand for the memory utilized by the NetScaler connections and the SSL module respectively .

*In order to find out and compare the memory trends for a particular NetScaler Appliance , we  need the newnslog files from that time and we can use the same command to see the memory stats .

*For finding the memory stats at a certain time stamp , the following command can be used:
 

nsconmsg111 -K newnslog.89 -d memstats -s time=10Apr2017:23:30:00

The above command finds the memory stats from 10 April, 2017 at 11:30 PM GMT .

For more details , please refer to below article :

https://support.citrix.com/article/CTX113341

4. With Memory stats , the memory usage trend can be compared :

• Certain memory module for e.ge. MEM_TBUF increasing continuously and ultimately consuming most of the NetScaler memory .
• We can also find out the sub modules in a partcular module for example NSB , PCB etc consuming the memory .

5. Another article to reference for allocation failures and memory errors counters . This can be done using the NetScaler memory counters as mentioned in the below article :

https://support.citrix.com/article/CTX133887

The common counters values that can be looked for as below :

mem_cur_feature_allocpercent: This counter tracks the percentage of NetScaler appliance memory used by the feature.

mem_cur_feature_allocsize: This counter tracks the total current NetScaler appliance memory available for use by the feature, in kilobytes.

mem_err_feature_alloc_failed: This counter tracks the memory allocation failure for a particular feature.

mem_tot_allocated_pcnt: This counter tracks the currently allocated memory in percent.

A sample usage for the memory error "mem_err_alloc_failed" is given below : 

nsconmsg -K newnslog.92 -d current -g mem_err_alloc_failed | grep MEM_CONN

      0  406011        1829973        129       18 mem_err_alloc_failed MEM_CONN

      6       0  3836098053568      33024     4717 mem_err_alloc_failed_size MEM_CONN

     28  903031        1846248       3306      472 mem_err_alloc_failed MEM_CONN

     41       0  3866148642560 6847211776 978173110 mem_err_alloc_failed_size MEM_CONN

     54   56002        1848872         78       11 mem_err_alloc_failed MEM_CONN

     56       0  3870607315200   39860992  5694427 mem_err_alloc_failed_size MEM_CONN

     58   63003        1867994         61        8 mem_err_alloc_failed MEM_CONN

     60       0  3909729805120      15616     2230 mem_err_alloc_failed_size MEM_CONN

     62   63002        1905734      12459     1779 mem_err_alloc_failed MEM_CONN

     66       0  3988433876544 26128416768 3732630966 mem_err_alloc_failed_size MEM_CONN

     70 1267045        5551517        912      130 mem_err_alloc_failed MEM_CONN

     71       0 11634169822848 1912602624 273189919 mem_err_alloc_failed_size MEM_CONN