Netscaler Inuse memory constantly high

Netscaler Inuse memory constantly high

book

Article ID: CTX230232

calendar_today

Updated On:

Description

Customer over the period of time has been noticing the memory usage gradually increasing, with past few days it is constantly above 70%


Instructions

Netscaler use Core-to-Core framework. When the rate of required Core-to-Core is high, sometimes it is possible that we don't get a slot in Core-to-Core channel. In that case, we queue them in a pending list, and retry in a back ground timer.

Now the issue is that this timer is not firing as expected.
If the delete messages are pending, then RSS Filter based NATPCBs remain on the PEs.

This is the Primary reason why we are hitting the bug, where we see NATPCBs hanging around and taking up lot of memory. This is the behavior in the initial stages of the problem. These RSS NATPCBs are freed only when corresponding Core-to-Core messages are delivered.
We can see that ‘rssf_cur_c2c_msg_pending’ counter keep on incrementing.

Index reltime     counter-value symbol-name&device-no
    1       0          11917809 rssf_cur_c2c_msg_pending

Now if the problem persists for some time, then if a new connection (SYN) comes on the same 4 tuple as that of the one that is pending to be removed, 
we get into situation of 'natpcb_err_pkt_drop_dupnatpcb' and 'mcmx_err_rssf_add_failed' counters getting incremented as seen below, and SYN is not forwarded eventually.

Index reltime     counter-value symbol-name&device-no
    1       0          12669160 natpcb_err_pkt_drop_dupnatpcb

Index reltime     counter-value symbol-name&device-no
    1       0          15086846 mcmx_err_rssf_add_failed

The below command mitigates the above issue,
set rsskeytype -rsstype SYMMETRIC.                                     

More information on the above command can be found in the below link,
https://docs.citrix.com/ko-kr/netscaler/11/reference/netscaler-command-reference/network/rsskeytype.html

Issue/Introduction

The Inuse memory on the primary netscaler is gradually increasing and has been steadily above 70% for two days