NetScaler Kerberos WIKI - Troubleshooting
Article ID: CTX234153
Updated On:
Description
Troubleshooting details for Kerberos Constrained Delegation and Authentication related issues on NetScaler
Instructions
Common issues- DNS not configured correctly
- root@ns200# /netscaler/nskrb kinit myname@DUMMY.COM
- myname@DUMMY.COM's Password:
- kinit: krb5_get_init_creds: unable to reach any KDC in realm DUMMY.COM
- Kerberos related ports are blocked by Firewall
- Clock skew between Netscaler and AD too great
- AD configuration incorrect
- Delegation is not enabled
- Setspn is done with a different account (if KCD account is added with different keytab or with password)
- Certificate mapping is not done (if KCD account is added with delegatedUser’s cert pair)
- CA cert is not imported to AD (if KCD account is added with delegated User’s cert pair)
Ensure the nskrb daemon is running (ps ax | grep nskrb)
Top-level command line options for nskrb
- ktutil: directive to operate on keytab files
- klist: directive to view keytab/cache files
- kinit: directive to do user authentication, ie obtaining TGT
- kgetcred: directive to get service-tickets, including S4U2Self/S4U2Proxy
- kdestroy: directive to purge cache files
Troubleshooting: Where to start?
1. Check under /var/krb location, it contains cached tickets.
2. Do #cat /tmp/nskrb.debug log, see if there is an error code returned other than ‘0’, refer to the error code and the description in troubleshooting section of the document.
3. If it is non-zero but SSO is still working? Probably NTLM fallback. There is a setting in windows 2008 to explicitly turn off ‘NTLM’.4. No logs in cat /tmp/nskrb.debug? Probably Application server is not configured for Kerberos/Negotiate authentication.
5. How to audit Kerberos Success/Failures? Check nsconmsg counters ‘nsconmsg -d stats | grep -i kcd’,
> nsconmsg -d stats | grep -i kcd
10531 0 0 aaa_tot_kcd_fail
10553 0 0 aaa_kcd_invalid_kcdaccount
10554 0 0 aaa_kcd_err_clock_skew
10555 0 0 aaa_kcd_tgt_cache_hits
10556 0 0 aaa_kcd_s4u2self_cache_hits
10557 0 0 aaa_kcd_tgs_cache_hits
10558 0 0 aaa_kcd_impersonation_success
10559 0 0 aaa_kcd_tot_kcd_success
10560 0 0 aaa_kcd_tot_pkinit_kcd_success
10561 0 0 aaa_kcd_tot_pkinit_imper_success
10562 0 0 aaa_kcd_err_kdc_unreachable
10563 0 0 aaa_kcd_err_ticket_verify_fail
10564 0 0 aaa_kcd_err_decrypt_fail
6. How to debug the Kerberos error?
Take a network trace filter for Kerberos
Look at AD event logs ‘windows security log’ event id: 4768/4769/4770/4771
Run the command line utilities to make sure that setup is working
- ktutil: directive to operate on keytab files
- klist: directive to view keytab/cache files
- kinit: directive to do user authentication, ie obtaining TGT
- kgetcred: directive to get service-tickets, including S4U2Self/S4U2Proxy
- kdestroy: directive to purge cache files
nskrb command line help: User Impersonation
- Obtaining a TGT
/netscaler/nskrb kinit --password=secret svc_kcd1@NSI-TEST.COM
If the above command is successful, TGT is obtained and is written into the default cache file /tmp/krb5cc_0
- Obtaining a TGS
/netscaler/nskrb kgetcred -c /tmp/krb5cc_0 http/nsi-dc1-2008.nsi-test.com@NSI-TEST.COM
Constrained Delegation troubleshooting
- Obtaining TGT
- Same as the previous case
- Obtaining S4U2Self
- Below command obtains S4U2Self for user “user1” using the TGT obtaining in the previous slide
- /netscaler/nskrb kgetcred -c /tmp/krb5cc_0 --out-cache=/tmp/imper_cache --impersonate=user1@NSI-TEST.COM svc_kcd1@NSI-TEST.COM
- In the above command, ticket is obtained for user1 using the delegated user svc_kcd1 and cached TGT in /tmp/krb5cc_0. The forwardable ticket is stored in output cache /tmp/imper_cache
- If output cache is not specified, it writes into /tmp/krb5cc_0. These credentials can be viewed with klist command mentioned earlier.
nskrb kinit --password=Secrect svc_kcd1@NSI-TEST.COM
nskrb klist -c /tmp/krb5cc_0
Credentials cache: FILE:/tmp/krb5cc_0
Principal: svc_kcd1@NSI-TEST.COM
Issued Expires Principal
Nov 1 03:04:24 2023 Nov 1 13:10:27 2023 krbtgt/NSI-TEST.COM@NSI-TEST.COM
nskrb kgetcred -c /tmp/krb5cc_0 http/apache.NSI-TEST.COM@NSI-TEST.COM
nskrb klist -c /tmp/krb5cc_0
Credentials cache: FILE:/tmp/krb5cc_0
Principal: svc_kcd1@NSI-TEST.COM
Issued Expires Principal
Nov 1 03:04:24 2023 Nov 1 13:10:27 2023 krbtgt/NSI-TEST.COM@NSI-TEST.COM
Nov 1 03:12:39 2023 Nov 1 13:10:27 2023 http/apache.NSI-TEST.COM@NSI-TEST.COM
nskrb kgetcred -c /tmp/krb5cc_0 --out-cache=/tmp/imper_cache --impersonate=user1@NSI-TEST.COM svc_kcd1@NSI-TEST.COM
nskrb klist -c /tmp/imper_cache
Credentials cache: FILE:/tmp/imper_cache
Principal: user1@NSI-TEST.COM
Issued Expires Principal
Nov 1 03:59:18 2023 Nov 1 13:17:09 2023 svc_kcd1@NSI-TEST.COM
Constrained Delegation troubleshooting Continued...
Obtaining TGS
- The last step in obtaining TGS ticket is S4UProxy, described by below command.
- /netscaler/nskrb kgetcred --delegation-credential-cache=/tmp/imper_cache --out-cache=/tmp/kcd_cache http/nsi-dc1-2008.nsi-test.com
- In the above command, we are using the delegation credentials obtained in previous step (the S4U2Self), and request for TGS for the service “http/nsi-dc1-2008.nsi-test.com”. If this command is successful, TGS is written into outcache /tmp/kcd_cache specified in above command.
- We can view the ticket using the same nskrb klist command.
nskrb kgetcred --delegation-credential-cache=/tmp/imper_cache --out-cache=/tmp/kcd_cache http/apache.NSI-TEST.COM@NSI-TEST.COM
nskrb klist -c /tmp/kcd_cache
Credentials cache: FILE:/tmp/kcd_cache
Principal: svc_kcd1@NSI-TEST.COM
Issued Expires Principal
Nov 1 03:17:23 2023 Nov 1 13:17:09 2023 http/apache.NSI-TEST.COM@NSI-TEST.COM
Kerberos logs and error counters on NS
- Auditing in Kerberos SSO
- Logs in userspace
- cat /tmp/nskrb.debug captures all the logs while doing Kerberos SSO
- Counters in Netscaler
- nsconmsg –d stats | grep kcd
- Needs to be issued on NS shell. Ouputs all Kerberos SSO counters
- Auditlog on Netscaler
- Runtime events/issues in packet engine are logged in /var/log/ns.log*
- Logs in userspace
Generic Kerberos common error codes:
0x6 - KDC_ERR_C_PRINCIPAL_UNKNOWN: Client not found in Kerberos database
1. The actual account does not exist.
2. new account is added but not yet replicated to other KDC.
3. Check if the account is expired or ‘logon restrictions’ enabled.
0x7 - KDC_ERR_S_PRINCIPAL_UNKNOWN: Server not found in Kerberos database
1. The actual name is missing.
2. A new computer account has been created and has not yet replicated to the KDC that the client is using for authentication.
3. Check that correct service class, domain/Realm settings are configured in the SPN.
0x8 - KDC_ERR_PRINCIPAL_NOT_UNIQUE: Multiple principal entries in database
This error occurs if duplicate principal names exist. Unique principal names are crucial for ensuring mutual authentication.
1. Use Setspn tool,
‘setspn –X’ will list the duplicate SPN, ‘setspn –Q’ can be used to query for the existence of SPN.
‘setspn –D’ can be used to delete specified SPN.
0xC - KDC_ERR_POLICY: KDC policy rejects request
1. KDC_ERR_POLICY is usually the result of logon restrictions in place on a user’s account.
Resolution: Check User login hrs. restriction.
2. Constrained delegation is being attempted across multiple domains.
Resolution: windows 2003/2008 does not support constrained delegation across multiple domains.
3. The server receives a ticket in which client’s realm does not match the local realm.
Resolution: ensure Client and Server are in the same realm.
0xE - KDC_ERR_ETYPE_NOTSUPP: KDC has no support for encryption type
In general, this error occurs when the KDC or a client receives a packet that it cannot decrypt.
0x10 - KDC_ERR_PADATA_TYPE_NOSUPP: KDC has no support for padata type
Smart card logon is being attempted and the proper certificate cannot be located. This can happen because the wrong certification authority (CA) is being queried or the proper CA cannot be contacted.
Resolution:
Verify that there is a functioning CA on the domain.
Verify that the client can locate the CA.
0x17 - KDC_ERR_KEY_EXPIRED: Password has expired – change password to reset
The Delegated user’s password has expired.
0x18 - KDC_ERR_PREAUTH_FAILED: Pre-authentication information was invalid.
Kerberos service account name is Case Sensitive, case needs to match.
Wrong password was provided.
Verify the time on the KDC matches the time on the client.
0x19 - KDC_ERR_PREAUTH_REQUIRED: Additional pre-authentication required.
This error often occurs in UNIX interoperability scenarios. MIT-Kerberos clients do not request pre-authentication when they send a KRB_AS_REQ message. If pre-authentication is required (the default), Windows systems will send this error. NetScaler will respond to this error by giving the pre-authentication, in which case the error can be ignored, but some clients might not respond in this way.
Case study: Setting on AD "account is sensitive and cannot be delegated" cause authentication issues.
In case you have problems with KCD (Kerberos Constrained delegation) configured on ADC also please check if service account (configured as KCD account on ADC) have the option below unchecked.
Also please check the same for MS AD built-in account krbtgt.
Note: To change this option for krbtgt account you must use Power Shell command and not MS AD Users and Computers mmc snap-in.
In case you have this option enabled you will have Forwardable flag set as ‘False’ for the Kerberos tickets (TGT or S4U2self, or for both of them) and authorization process cannot be finished, so the KCD account cannot receive S4U2proxy ticket.
You can see the following errors issuing test commands:
At the same time in the network traffic, you will see the error ‘KRB Error: KRB5KDC_ERR_BADOPTION NT Status: STATUS_ACCOUNT_RESTRICTION’.
You can see the flag setting in the network traffic inspecting Kerberos ticket requests:
Was this article helpful?
Yes
No
Powered by
