Citrix PKI troubleshooting Guide.
Article ID: CTX256666
Updated On:
Description
How to use this Guide:
First go through the checklist to determine the type of problem.
Then review client testing information to narrow down the scope.
If a Certificate installation is needed on the IIS Server or ADC,
The directions are given below for various scenarios.
Checklist:
[ √ ] DNS:
Certificates must resolve to the FQDN listed in the Issue to: field or
it needs to be listed as one of the certificates included as part of a
SAN certificate which contain two or more unique FQDN's.
(If not you will get a certificate mismatch error)
Check DNS resolution with NSLOOKUP or Ping as needed to be sure
that the FQDN on the certificate matches to rule out DNS issues.
[ √ ] Ports:
From the endpoint, check SSL (443) communication by using Telnet to the
fqdn over port 443 to verify from the client that port 443 is open.
If telnet isn't installed use server manager to install the telnet client
on the machine you are on.
Example:
telnet storefront.domain.local 443
(The output will be a blank screen with a blinking "_" if successful)
From the IIS / Storefront server check that port 443 is listening with the Netstat command
Example:
c:\> netstat -ano
Then scroll through the list to see if the StoreFront server is listening on port 443.
[ √ ] Bindings:
Check IIS port bindings by opening IIS Manager and navigate to
default web site > Bindings > Open binding for port 443
While you are there Verify that the hostname field is blank.
If it isn't then remove anything that is in that field.
If you see there is no binding and need to make a binding or change the hostname field,
always remember to run IISRESET from command line with administrator rights
when finished with any changes to IIS.
[ √ ] Types of Certificates:
Server - Certificates that are issued by a Certificate Authority
which are issued to an FQDN that resolves in DNS.
This in turn secures a site via HTTPS.
Intermediate - An intermediate certificate is a subordinate certificate
issued by the trusted root specifically to issue end-entity server certificates.
The result is a certificate chain that begins at the trusted root CA,
through the intermediate and ending with the SSL certificate issued to you.
Root - A root certificate is a public key certificate that identifies a
root certificate authority (CA).
Root certificates are self-signed and form the basis of an
X.509-based public key infrastructure (PKI).
SAN - Contains Multiple "Subject Alternate Names" Which you can use to specify
unique FQDNs on the Same SAN Certificate.
Note: Many certificate authorities limit the number of SAN a
maximum of 100 entries.
(For more information visit the Microsoft FAQ for SAN Certificates)
Wildcard - Notated as *.fqdn they can be used with different subdomains.
Wildcard and SAN certificates would allow a public or private certificate
to installed on multiple Servers or Appliances such as
StoreFront, Delivery Controllers, and an ADC.
[ ≈ ] Additional Notes:
1. Child Domains and Wildcard certificates
If there is a child domain a wildcard won't be able to be used since
it would require a *.*.fqdn. (I.e. access.storefront.domain.com)
In that case a normal certificate with an fqdn or a SAN certificate with
multiple fqdns needs to be created to replace the wildcard cert.
2. Self Signed Certificates vs Internal / Public certificates
issued by a Certificate Authority (CA)-
If the customer is using the term "Self Signed" certificate they may
mean that their internal CA is signing it which is supported.
Or they may literally be clicking "create a self signed certificate"
in IIS where the certificate is issued to and by the StoreFront server.
3. A Self signed Certificate created in StoreFront using
the "create a self signed certificate" option in IIS:
-
Is not supported in production by Citrix.
-
Since it is issued by a StoreFront server it isn't distributed
to domain joined machines. -
Requires that each client have the self signed certificate
installed in the MMC console of every client machine.
Note: This can be a hassle if there are many end point.
4. If needed you can test with a self signed certificate on the endpoint by:
-
Going to MMC on the client machine
-
Adding the snapin for certificates > choose computer account > Ok
-
Right click private and import the self signed certificate.
-
Then right click trusted root certificate and import it there as well.
-
Since the self signed certificate is issued to and by itself it is a Server certificate as well as a
Root CA and must be imported to be trusted root on the client MMC. -
Right click trusted root on the left pane and click import and find the same certificate.
[ √ ] Types of Certificate Hash algorithms:
You can view the algorithm used by inspecting a certificate and scrolling through the details tab.
SHA1 - These are deprecated by the industry because they don't meet the security requirements.
https://docs.microsoft.com/en-us/security-updates/securityadvisories/2017/4010323
If you see a customer using a SHA1 certificate internally advise that
they will need to switch to SHA2 / SHA256 or greater.
SHA256 - The standard Certificate that is issued by a Public CA.
SHA2 - These certificates are widely adopted because of the bandwidth and utility.
SHA-256 is generally faster on 32-bit processors
Typically it is the standard chosen for internal CA's as well. endpoint support -
https://support.citrix.com/article/CTX200114
SHA384 - More secure than SHA2 /SHA256 and is supported but rarely used.
SHA512 - It is generally faster on 64-bit processors and contains collision
resistance which prevents attacks via Quantum computers.
Note: If a customer is attempting to use a SHA512 certificate with Server
2012r2 the following patch must be applied. (Click here for more information)
SHA512 is disabled in Windows when you use TLS 1.2 so the
same patch needs to be installed on the clients as well.
https://support.citrix.com/article/CTX212883
(Speed comparisions mentioned can be measured with the command "openssl speed sha256 sha512".)
Please review the following link to see the support for ADC
https://docs.citrix.com/en-us/netscaler/12/ssl/server-certificate-support-matrix-on-the-adc-appliance.html
[ √ ] Internal Client testing: (Direct to StoreFront)
1. Test with the hosts file pointed to one StoreFront server at
a time bypassing the LB VIP on the ADC.
This will verify if going direct will function:
Open notepad as an administrator on the client machine.
Edit hosts file under C:\Windows\System32\Drivers\ETC
Change file types to all to see the files.
Append the following example with the Customer's information:
192.168.1.11 base.url.local
<StoreFront IP> <BASE URL>
StoreFront supports up to 6 servers in a group
(But there is no benefit to having more than 3 in a group)
Keep testing one at a time by changing the IP and setting it to
resolve to the same baseURL.
Each time just save notepad without closing it.
Open cmd line and ping the base.url.local fqdn to make sure that
it now resolves to the StoreFront IP.
Then do a test with the browser and Workspace App as needed.
2. Click view to inspect the certificate in a browser if the issue persists.
Verify that the certificate name matches the BaseURL exactly or
is a Wildcard *.fqdn. What you should see is the certificate that
is bound to HTTPs (443) in IIS for that server you are pointing to.
Inspect certificate from the client machine:
1. Click the lock at the top in Internet Explorer to
inspect/view the certificate.
2. Verify that there is a Chain under details.
3. Check that the certificate is valid and not expired
4. It should show a Root Certificate Authority as well as
the Server certificate under details.
5. For a public CA (Such as Godaddy) it will show an intermediate
certificate in between the Root and Server certificates.
3. Test over HTTP if you are having an issue with one server in the
group to narrow it down.
1. Test on the StoreFront server directly over Http://localhost/Citrix/StoreWeb from a browser.
2. Then test on the client to the IP of StoreFront Http://x.x.x.x/Citrix/StoreWeb
3. Skip Certificate warnings and click proceed if using Chrome.
4. If the Store doesn't function in a test over HTTP it could indicate an issue with the Store.
5. You can change the BaseURL in StoreFront to be http temporarily if needed by going to
Server group > Change Base URL > Remove the "S" in Https
Then perform an IISRESET from command line with administrative rights.
- This way you can make sure that the Store is functional.
- You may need to create a test store.
- If a test store resolves it you can delete the test store and recreate the original.
- Otherwise you may need to reset StoreFront to factory settings (CTX200239) or reinstall.
Note: In either instance you can rejoin it to another (good) server in the group
to transfer the configuration.
Resetting or reinstalling will give you the option to rejoin a group
or create a deployment. However if it is a single StoreFront server take a
snapshot and manually make note of all the configuration before proceeding.
The recommended uninstallation method for StoreFront if needed is:
1. Create a snapshot
Backup subscriptions if needed so they can be restored after
(https://support.citrix.com/article/CTX216295)
Take screenshots and/or make note of the configuration.
2. Remove StoreFront
3. Remove the IIS Role.
4. Reboot
5. Open Server manager and verify that IIS removal has been completed
6. Reinstall StoreFront which also reinstalls IIS.
7. If it still fails over HTTP there is something wrong with the server
and the customer will need to stand up a new VM.
4. The customer might have attempted to install a certificate downloaded
from a CA but doesn't contain a private key.
In order for a certificate to be valid it must have a private key.
If the following steps do not apply, then a suitable certificate
containing a private key must be used.
This may be because the certificate was provided by the Administrator
that manages the CA. If the file provided ends in a .CER extension it
just needs to be completed using the same IIS server that the CSR
(Certificate Signing Request) Originated from.
1. IIS management console > Hostname on left > Server Certificates
> Complete Certificate Request.
2. Browse for the .CER file > Input a name that is unique so you can
recognize it when binding.
3. Set the Certificate Store to personal from the drop down > next.
4. Bind certificate by going to Default Website > bindings > Edit
or add an HTTPS binding (If it doesn't exist)
5. Find the unique friendly name you chose and click ok.
6. Run IISRESET
Note: For expired certificates or having the wrong type of certificate bound.
Follow these directions:
To obtain a Certificate using web enrollment through IIS -
https://support.citrix.com/article/CTX200292
Run IISRESET via command line with administrative
rights after you bind the new Certificate in IIS.
5. Once you have a certificate installed with the private key in IIS you
can export it if it was marked as exportable.
Follow https://support.citrix.com/article/CTX206492 to get a PFX
certificate to other StoreFront servers in the group.
Notes:
1. If you have tested all servers individually and still cannot reproduce the issue,
then the issue is most likely on the (Netscaler) / ADC LB VIP / Gateway.
2. testing from more than one client machine could rule out a client side problem.
6. If you get the following error it may be due to the certificate
not being marked as exportable.
You may see this issue after installing a certificate in IIS.
In the browser you will see the following:
This page can’t be displayed
Turn on TLS 1.0, TLS 1.1, and TLS 1.2 in Advanced settings and try
connecting to https://<fqdn> again. If this error persists,
it is possible that this site uses an unsupported protocol or cipher
suite such as RC4 (link for the details) <http://go.microsoft.com/fwlink/?LinkId=735074>,
which is not considered secure. Please contact your site administrator.
If you remove the binding and attempt to add it back you may see the error:
Most of the time this can be fixed by checking
“Allow this certificate to be exported”
However when importing a PFX certificate into IIS and left unchecked
it results in a broken keyset which is why the error occurs.
(For more information please visit this link)
[ √ ] Browsers
All browsers share the same shortcut to clear all cache CTRL+SHIFT+DEL
In control panel > within Internet Properties > Content > Clear SSL State
Use InPrivate NewPrivate or Incognito modes to test.
Internet Explorer - Test using In private mode by clicking the top right "gear icon" then click safety > InPrivate browsing.
FireFox - New Private Window CTRL+SHIFT+P or select the 3 "-"'s top right and select New Private Window
Edge - Click the "..." on the top right then choose New inPrivate Window
Chrome - You may get an error with chrome with relation to the Error: "Subject Alternative Name Missing" or
NET::ERR_CERT_COMMON_NAME_INVALID or "Your connection is not private"
If the subject alternate name is missing which can be seen in the details of a certificate by inspecting it
and scrolling through until you see dns=
This might happen even if Internet Explorer and Edge have no issues.
-
If you need to create a certificate request that contains Subject Alternate Name name:
1. Open MMC > Add snappin > Certificates > Computer account.
2. Right click private > all tasks > advanced Operations > Create Custom Request > next.
3. Select Certificate Enrollment Policy page > Active Directory Enrollment Policy > next.
4. On the Custom Request page > select the Web Server template and PKCS#10 format > next.
5. Click details > Properties
6. Subject tab > Subject Name field > select Common Name > add fqdn (I.e. SF baseURL)
> Alternative Name field > DNS > add the fqdn
Under the private Key tab > Select Key options > Key size >2048 >
Check to make exportable if you want to use on other Servers. > Select Hash Algorithm > Select sha256
7. Use certreq and certutil to submit the CSR request file to the CA or submit via web enrollment(CTX200292).
(Reference for commands certreq and certutil)
8. Verify that it exists in the private certificates within MMC.
9. Open it and view the details to confirm the Subject Alternate Name DNS= is now complete
10. Verify that it has a private key.
11. Bind to port 443 in IIS > default web site > bindings > choose file in dropdown > ok > IISRESET
12. Test with Chrome using hosts file to point directly to the StoreFront server's IP in an Incognito window.
Test using Incognito mode CTRL+SHIFT+N or click "..." top right and select New Incognito Window
1. Check that you’ve updated ChromeOS to the latest version.
To update Chrome, press the Customize Google Chrome button.
Select Help > About Google Chrome to open the tab shown directly below.
2. Open the Chrome browser; and click the Customize Google
Chrome button at the top right of its window.
Select Settings to open the tab in the snapshot directly below.
Press Advanced to open further options.
Scroll down to and select the Open proxy settings
option to open the window below.
3. If you need to recreate the certificate because there is
a DNS error only with Chrome, you will need to use the MMC
console to create the certificate and add the DNS properties before
submitting the Certificate request to the CA (Shown above).
[ √ ] Workspace App Clients:
Windows:
Run ipconfig /flushdns from command line to flush DNS.
In control panel > Under internet properties > content > clear SSL state
You can use https://support.citrix.com/article/CTX134341
to bypass the Certificate requirement for testing purposes when testing with HTTP.
Note:
This can only be done with the Windows Platform and
all other platforms require a full certificate chain.
Also, the baseURL needs to be changed to HTTP while testing with CTX134341
if it is Set to HTTPS by going to server group and removing the "S" from HTTPS.
Mac:
May require a root certificate and intermediate imported into the Keychain.
Use Safari to browse to the Internal or External Site.
Click on the lock icon to inspect the certificate.
Drag the "Picture" of the certificate to the desktop or another directory to download it.
1. From the browser, open the directory in which
you downloaded the root cert file.
2. Double-click the certificate.
3. You are prompted to store the certificate in the login
keychain or the system keychain.
4. To make the certificate available to all users of
this system, select system keychain.
5. In Keychain Access, select the System keychain;
then select Cloud Services Root CA certificate.
6. Select File > Get Info and expand the Trust section.
7. Change Secure Sockets Layer (SSL) value to Always Trust.
8. Close the dialogs and enter your password.
ChromeBook:
Check that you have the latest updates to ChromeOS
At the bottom right, select the time. > Select Settings >
Select Menu About Chrome OS > Under "Google Chrome OS,"
you'll see which version of the Chrome operating system
your Chromebook is using. Select Check for Updates.
HTML5:
Must be used with a Netscaler or you will get an error
referencing https://support.citrix.com/article/CTX134123
Otherwise StoreFront must be used with HTTP or it will fail.
Ensure that the latest version is installed on each StoreFront
server https://www.citrix.com/downloads/workspace-app/html5/
Linux:
How to Import the certificates on a fat client.
(Thin clients are supported by the thin client vendors directly
https://support.citrix.com/article/CTX234462)
1. open terminal window
2. sudo firefox (this will allow us to save the files we are
going to export without permission issues)
*Otherwise save it first and copy it to the cacerts folder path below*
3. browse to the fqdn for the ADC Gateway or StoreFront
4. click the lock icon in the browser
5. browse to see more information on the certificate
6. when you see the certificate chain click on each one to export
7. one at a time starting with the root and ending with the server cert
click the export button after highlighting each
8. export each one to /opt/Citrix/ICAClient/keystore/cacerts and append
".pem" to each file name before saving.
9. when you are finished you can break out ctrl+c in the open
terminal window to close firefox.
10. navigate to /opt/Citrix/ICAClient/keystore/cacerts
11. run sudo ../../util/ctx_rehash
12. then try to launch
If issues persist you may need to do a symbolic link to the
mozilla keystore or the Linux keystore by doing the following:
sudo ln -s /usr/share/ca-certificates/mozilla/* /opt/Citrix/ICAClient/keystore/cacerts/
Then run sudo ../../util/ctx_rehash again
If that still doesn't help run the following:
sudo ln -s /etc/ssl/* /opt/Citrix/ICAClient/keystore/cacerts/
sudo ln -s /etc/ssl/certs/* /opt/Citrix/ICAClient/keystore/cacerts/
Then run sudo ../../util/ctx_rehash again
[ √ ] External Client testing: (LB VIP/GW VIP on ADC)
If the issue can't be reproduced going directly to StoreFront the
issue may be on the ADC/Netscaler.
The baseURL will normally resolve to the Load Balancer Virtual server IP (LB VIP).
1. Open Netscaler configuration to Inspect LB VIP
2. Goto Traffic Management > Load Balancer > Virtual Servers.
Ping the base URL from a machine that doesn't have anything in the hosts or
place a "#" at the front of the entry you made in the HOSTS file
earlier to comment it out and save it. To find the IP of the LB VIP just use
command line and resolve the IP address.
Find the Virtual Server and inspect the certificate.
3. Verify that the correct certificate is installed and is not expired and
is chained correctly (Server > Intermediate > Root)
4. Otherwise install the correct certificate and link.
(https://support.citrix.com/article/CTX205404)
Additional configuration and considerations for endpoints:
When troubleshooting:
External Gateway certificates can be verified by inputting the
Gateway FQDN into one of the following sites
to check that the certificate chain is linked and bound correctly.
Https://www.digicert.com/help - Simplified overview with chain.
Https://ssllabs.com/ssltest - More advanced overview with SSL score and tests against all platforms.
(to rerun the test clear cache at the top)
If you see that the Certificate is not trusted because of a broken chain.
1. Navigate to Traffic Management > SSL > Server Certificates >
Right Click correct certificate and click link
2. to link the Server Cert to the Intermediate > Repeat for Intermediate to Root.
3. Then test again with the external sites above to check the certificate chain is correct
and that the FQDN of the Gateway matches the certificate.
Issue/Introduction
