NetScaler Load Balancer Drops Traffic When AppFirewall Session Limit is Reached
Article ID: CTX217549
Updated On:
Description
>>Not able to connect to any Load Balancer VIP.
>>In trace file we see RESET being sent by NetScaler with reset code 9845.
>>Issue will be resolved after failover or reboot but might appear again.
>>In newnslog we see below counter increasing at the time of issue:
nsconmsg -g as_err_session_create_failed -d current
Command on support bundle:
nsconmsg -K /var/nslog/newnslog -g -g as_err_session_create_failed -d current
Output:
reltime:mili second between two records Sat Jun 25 05:38:47 2016
Index rtime totalcount-val delta rate/sec symbol-name&device-no&time
0 2464848 207 1 0 as_err_session_create_failed Sat Jun 25 05:38:47 2016
1 13965 208 1 0 as_err_session_create_failed Sat Jun 25 05:39:00 2016
2 48877 209 1 0 as_err_session_create_failed Sat Jun 25 05:39:49 2016
3 139650 210 1 0 as_err_session_create_failed Sat Jun 25 05:42:09 2016
4 6983 211 1 0 as_err_session_create_failed Sat Jun 25 05:42:16 2016
>>We see counter for as_alive_session will be around 1,00,000* number of PE. For example if PE=3 than as_alive_session will be around 3,00,000. Below is an example of issue wherein box had 1 PE:
Command on live box:
nsconmsg -g as_alive_sessions -d current
Command on support bundle:
nsconmsg -K /var/nslog/newnslog -g -g as_alive_sessions -d current
Output:
312 6982 99955 16 2 as_alive_sessions Tue Jun 28 04:00:48 2016
313 6983 99964 9 1 as_alive_sessions Tue Jun 28 04:00:55 2016
314 6982 99975 11 1 as_alive_sessions Tue Jun 28 04:01:02 2016
315 6983 99990 15 2 as_alive_sessions Tue Jun 28 04:01:09 2016
316 6982 100001 11 1 as_alive_sessions Tue Jun 28 04:01:16 2016
Resolution
>>Check traffic rate on NetScaler is not crossing device limitation. If yes than ask customer to reduce traffic or upgrade hardware.
>>As a workaround you can reduce the session timeout on AppFirewall global settings as seen in snapshot below to lower value. (Please note this might have adverse effects in some environment where is necessary to keep session active for long time).
>> If it is a VPX or SDX than number of CPU cores can be increased which will increase PE and eventually increase AppFirewall session capacity. Each PE will increase capacity by 1,00,000.
Problem Cause
By default each packet engine can only handle 1,00,000 AppFW session at a given time. So if the sessions are crossing 1,00,000 per PE than this issue is seen.
Issue/Introduction
