Users complain of authentication not working when trying to authenticate at Netscaler Gateway. When administrator reboots the Netscaler the issue is fixed for some time but after couple of hours the issue comes back again.

From ns.log on Netscaler we consistently see following errors:

Jul 28 22:09:28 <local0.warn> 10.10.10.10 07/28/2016:20:09:28 GMT testvpx 0-PPE-0 : default AAA LOGIN_FAILED 128264 0 :  User test@xyz.com - Client_ip 10.9.9.9 - Failure_reason "External authentication server denied access" - Browser Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/46.0.2486.0 Safari/537.36 Edge/13.10586

In GUI under Netscaler Gateway > Active User sessions we see high number of sessions (8100 in this case) as shown below:




Rebooting the box fix the authentication issue for sometime but the issue comes back after few hours in 1-2 days.
 

Please note: Below troubleshooting step is not the ONLY reason which can cause this issue. But it is worth checking the following settings to see if that helps in resolving the issue based on the troubleshooting done in few of the cases with different customers.
========================================

In this case after we checked the netscaler logs and it seems like the issue is caused due to high high sessiontimeout of 480 minutes=8 hours set in tm sessionaction.

Default -sesstimeout value is 30 minutes. Reducing the timeout to 30 minutes from 480 minutes resolved the issue
===

Below logs were checked from the support file collected:

Before reboot:
===

AAA Statistics (We see high TM sessions and very few AAA sessions, which gave the clue that something may be wrong regards to TM session config)

Rate (/s) Total
Authentication successes 0 35189
Authentication failures 0 648
HTTP authorization successes 62 294017
HTTP authorization failures 0 54
Non HTTP authorization successes 0 26
Non HTTP authorization failures 0 80
Current AAA sessions 0 0
Total AAA sessions 0 8
Timed out AAA sessions 1 26705
Current ICAOnly sessions 0 20
Current ICAOnly Conn 0 10
Current ICA (Smart Access) Conn 0 0
Current TM sessions -1 8245
TM sessions 0 34912

After reboot:

AAA Statistics (We see after the reboot the TM session came down to 590 and users were not complaining of the isue)

Rate (/s) Total
Authentication successes 0 591
Authentication failures 0 6
HTTP authorization successes 1 4208
HTTP authorization failures 0 1
Non HTTP authorization successes 0 0
Non HTTP authorization failures 0 0
Current AAA sessions 0 0
Total AAA sessions 0 0
Timed out AAA sessions 0 0
Current ICAOnly sessions 0 0
Current ICAOnly Conn 0 7
Current ICA (Smart Access) Conn 0 0
Current TM sessions 0 590
TM sessions 0 590

We can check the above statistics on the netscaler box directly by using the below command:

> stat aaa detail

We then further checked the tm session actions in ns.conf file and found high timeout value set on sessionAction as shown below (Below we have listed only one tm sessionAction but there were more than 10 SessionAction profiles with high timeout):

add tm sessionAction owa-session-prof -sessTimeout 480 -defaultAuthorizationAction ALLOW -SSO ON -ssoCredential PRIMARY -httpOnlyCookie NO -persistentCookie ON -persistentCookieValidity 30

Asked the customer to reduce the -sessTimeout to 30 minutes > Save config and reboot the box. After the reboot we never saw users reporting of any issues and also the AAA sessions were not showing high as seen before making the changes.
 

---

Problem Cause

The issue was caused due to high sessiontimeout set on traffic management session actions of 480 minutes.

This article talks about the issue where users complain of authentication not working when trying to authenticate at Netscaler Gateway. When administrator reboots the Netscaler the issue is fixed for some time but after couple of hours the issue comes back again.