kinit: krb5_get_init_creds: unable to reach any KDC in realm DUMMY.COM
Kerberos related ports are blocked by Firewall
Clock skew between Netscaler and AD too great
AD configuration incorrect
Delegation is not enabled
Setspn is done with a different account (if KCD account is added with different keytab or with password)
Certificate mapping is not done (if KCD account is added with delegatedUser’s cert pair)
CA cert is not imported to AD (if KCD account is added with delegated User’s cert pair)
Check Kerberos functionality from NetScaler shell Ensure the nskrb daemon is running (ps ax | grep nskrb)
Top-level command line options for nskrb
ktutil: directive to operate on keytab files
klist: directive to view keytab/cache files
kinit: directive to do user authentication, ie obtaining TGT
kgetcred: directive to get service-tickets, including S4U2Self/S4U2Proxy
kdestroy: directive to purge cache files
Troubleshooting: Where to start?
1. Check under /var/krb location, it contains cached tickets.
2. Do #cat /tmp/nskrb.debug log, see if there is an error code returned other than ‘0’, refer to the error code and the description in troubleshooting section of the document.
3. If it is non-zero but SSO is still working? Probably NTLM fallback. There is a setting in windows 2008 to explicitly turn off ‘NTLM’. 4. No logs in cat /tmp/nskrb.debug? Probably Application server is not configured for Kerberos/Negotiate authentication. 5. How to audit Kerberos Success/Failures? Check nsconmsg counters ‘nsconmsg -d stats | grep -i kcd’,
6. How to debug the Kerberos error? Take a network trace filter for Kerberos Look at AD event logs ‘windows security log’ event id: 4768/4769/4770/4771 Run the command line utilities to make sure that setup is working
ktutil: directive to operate on keytab files
klist: directive to view keytab/cache files
kinit: directive to do user authentication, ie obtaining TGT
kgetcred: directive to get service-tickets, including S4U2Self/S4U2Proxy
In the above command, ticket is obtained for user1 using the delegated user svc_kcd1 and cached TGT in /tmp/krb5cc_0. The forwardable ticket is stored in output cache /tmp/imper_cache
If output cache is not specified, it writes into /tmp/krb5cc_0. These credentials can be viewed with klist command mentioned earlier.
nskrb kinit --password=Secrect svc_kcd1@NSI-TEST.COM nskrb klist -c /tmp/krb5cc_0 Credentials cache: FILE:/tmp/krb5cc_0 Principal: svc_kcd1@NSI-TEST.COM Issued Expires Principal Nov 1 03:04:24 2023 Nov 1 13:10:27 2023 krbtgt/NSI-TEST.COM@NSI-TEST.COM
nskrb kgetcred -c /tmp/krb5cc_0 http/apache.NSI-TEST.COM@NSI-TEST.COM nskrb klist -c /tmp/krb5cc_0 Credentials cache: FILE:/tmp/krb5cc_0 Principal: svc_kcd1@NSI-TEST.COM Issued Expires Principal Nov 1 03:04:24 2023 Nov 1 13:10:27 2023 krbtgt/NSI-TEST.COM@NSI-TEST.COM Nov 1 03:12:39 2023 Nov 1 13:10:27 2023 http/apache.NSI-TEST.COM@NSI-TEST.COM
nskrb kgetcred -c /tmp/krb5cc_0 --out-cache=/tmp/imper_cache --impersonate=user1@NSI-TEST.COM svc_kcd1@NSI-TEST.COM nskrb klist -c /tmp/imper_cache Credentials cache: FILE:/tmp/imper_cache Principal: user1@NSI-TEST.COM Issued Expires Principal Nov 1 03:59:18 2023 Nov 1 13:17:09 2023 svc_kcd1@NSI-TEST.COM
In the above command, we are using the delegation credentials obtained in previous step (the S4U2Self), and request for TGS for the service “http/nsi-dc1-2008.nsi-test.com”. If this command is successful, TGS is written into outcache /tmp/kcd_cache specified in above command.
We can view the ticket using the same nskrb klist command.
nskrb kgetcred --delegation-credential-cache=/tmp/imper_cache --out-cache=/tmp/kcd_cache http/apache.NSI-TEST.COM@NSI-TEST.COM nskrb klist -c /tmp/kcd_cache Credentials cache: FILE:/tmp/kcd_cache Principal: svc_kcd1@NSI-TEST.COM Issued Expires Principal Nov 1 03:17:23 2023 Nov 1 13:17:09 2023 http/apache.NSI-TEST.COM@NSI-TEST.COM
Kerberos logs and error counters on NS
Auditing in Kerberos SSO
Logs in userspace
cat /tmp/nskrb.debug captures all the logs while doing Kerberos SSO
Counters in Netscaler
nsconmsg –d stats | grep kcd
Needs to be issued on NS shell. Ouputs all Kerberos SSO counters
Auditlog on Netscaler
Runtime events/issues in packet engine are logged in /var/log/ns.log*
Generic Kerberos common error codes:
0x6 - KDC_ERR_C_PRINCIPAL_UNKNOWN: Client not found in Kerberos database 1. The actual account does not exist. 2. new account is added but not yet replicated to other KDC. 3. Check if the account is expired or ‘logon restrictions’ enabled.
0x7 - KDC_ERR_S_PRINCIPAL_UNKNOWN: Server not found in Kerberos database 1. The actual name is missing. 2. A new computer account has been created and has not yet replicated to the KDC that the client is using for authentication. 3. Check that correct service class, domain/Realm settings are configured in the SPN.
0x8 - KDC_ERR_PRINCIPAL_NOT_UNIQUE: Multiple principal entries in database This error occurs if duplicate principal names exist. Unique principal names are crucial for ensuring mutual authentication. 1. Use Setspn tool, ‘setspn –X’ will list the duplicate SPN, ‘setspn –Q’ can be used to query for the existence of SPN. ‘setspn –D’ can be used to delete specified SPN.
0xC - KDC_ERR_POLICY: KDC policy rejects request 1. KDC_ERR_POLICY is usually the result of logon restrictions in place on a user’s account. Resolution: Check User login hrs. restriction. 2. Constrained delegation is being attempted across multiple domains. Resolution: windows 2003/2008 does not support constrained delegation across multiple domains. 3. The server receives a ticket in which client’s realm does not match the local realm. Resolution: ensure Client and Server are in the same realm.
0xE - KDC_ERR_ETYPE_NOTSUPP: KDC has no support for encryption type In general, this error occurs when the KDC or a client receives a packet that it cannot decrypt.
0x10 - KDC_ERR_PADATA_TYPE_NOSUPP: KDC has no support for padata type Smart card logon is being attempted and the proper certificate cannot be located. This can happen because the wrong certification authority (CA) is being queried or the proper CA cannot be contacted. Resolution: Verify that there is a functioning CA on the domain. Verify that the client can locate the CA.
0x17 - KDC_ERR_KEY_EXPIRED: Password has expired – change password to reset The Delegated user’s password has expired.
0x18 - KDC_ERR_PREAUTH_FAILED: Pre-authentication information was invalid. Kerberos service account name is Case Sensitive, case needs to match. Wrong password was provided. Verify the time on the KDC matches the time on the client.
0x19 - KDC_ERR_PREAUTH_REQUIRED: Additional pre-authentication required. This error often occurs in UNIX interoperability scenarios. MIT-Kerberos clients do not request pre-authentication when they send a KRB_AS_REQ message. If pre-authentication is required (the default), Windows systems will send this error. NetScaler will respond to this error by giving the pre-authentication, in which case the error can be ignored, but some clients might not respond in this way.
Case study: Setting on AD "account is sensitive and cannot be delegated" cause authentication issues. In case you have problems with KCD (Kerberos Constrained delegation) configured on ADC also please check if service account (configured as KCD account on ADC) have the option below unchecked.
Also please check the same for MS AD built-in account krbtgt.
Note: To change this option for krbtgt account you must use Power Shell command and not MS AD Users and Computers mmc snap-in.
In case you have this option enabled you will have Forwardable flag set as ‘False’ for the Kerberos tickets (TGT or S4U2self, or for both of them) and authorization process cannot be finished, so the KCD account cannot receive S4U2proxy ticket.
You can see the following errors issuing test commands: At the same time in the network traffic, you will see the error ‘KRB Error: KRB5KDC_ERR_BADOPTION NT Status: STATUS_ACCOUNT_RESTRICTION’.
You can see the flag setting in the network traffic inspecting Kerberos ticket requests: