Citrix PKI troubleshooting Guide.

Citrix PKI troubleshooting Guide.

book

Article ID: CTX256666

calendar_today

Updated On:

Description

How to use this Guide:

    First go through the checklist to determine the type of problem. 
    Then review client testing information to narrow down the scope.
    If a Certificate installation is needed on the IIS Server or ADC, 
    The directions are given below for various scenarios.

Checklist:

[ √ ] DNS:


    Certificates must resolve to the FQDN listed in the Issue to: field or
    it needs to be listed as one of the certificates included as part of a         
    SAN certificate which contain two or more unique FQDN's.
    (If not you will get a certificate mismatch error)
    Check DNS resolution with NSLOOKUP or Ping as needed to be sure
    that the FQDN on the certificate matches to rule out DNS issues.

[ √ ] Ports:

    
    From the endpoint, check SSL (443) communication by using Telnet to the
    fqdn over port 443 to verify from the client that port 443 is open.
    If telnet isn't installed use server manager to install the telnet client
    on the machine you are on.

    Example:

 

telnet storefront.domain.local 443

(The output will be a blank screen with a blinking "_" if successful)


    From the IIS / Storefront server check that port 443 is listening with the Netstat command

     Example:

c:\> netstat -ano
     Then scroll through the list to see if the StoreFront server is listening on port 443.

[ √ ] Bindings:

    
    Check IIS port bindings by opening IIS Manager and navigate to
    default web site > Bindings > Open binding for port 443
    While you are there Verify that the hostname field is blank.
     If it isn't then remove anything that is in that field.
    If you see there is no binding and need to make a binding or change the hostname field,
    always remember to run IISRESET from command line with administrator rights
    when finished with any changes to IIS.
    

[ √ ] Types of Certificates:


        Server -             Certificates that are issued by a Certificate Authority
                                  which are issued to an FQDN that resolves in DNS.
                                  This in turn secures a site via HTTPS.

        Intermediate -    An intermediate certificate is a subordinate certificate
                                    issued by the trusted root specifically to issue end-entity server certificates.
                                    The result is a certificate chain that begins at the trusted root CA, 
                                    through the intermediate and ending with the SSL certificate issued to you.

        Root  -                 A root certificate is a public key certificate that identifies a
                                    root certificate authority (CA). 
                                    Root certificates are self-signed and form the basis of an
                                     X.509-based public key infrastructure (PKI).

        SAN -                Contains Multiple "Subject Alternate Names" Which you can use to specify
                                  unique FQDNs on the Same SAN Certificate.
                             
                                  Note:  Many certificate authorities limit the number of SAN a
                                              maximum of 100 entries. 
                                             (For more information visit the Microsoft FAQ for SAN Certificates)

        Wildcard - Notated as *.fqdn they can be used with different subdomains.

            Wildcard and SAN certificates would allow a public or private certificate
            to installed on multiple Servers or Appliances such as
            StoreFront, Delivery Controllers, and an ADC.

  
[ ≈ ]  Additional Notes:


    1. Child Domains and Wildcard certificates


        If there is a child domain a wildcard won't be able to be used since
        it would require a *.*.fqdn. (I.e. access.storefront.domain.com)
        In that case a normal certificate with an fqdn or a SAN certificate with
        multiple fqdns needs to be created to replace the wildcard cert.

 

    2. Self Signed Certificates vs Internal / Public certificates
         issued by a Certificate Authority (CA)-


        If the customer is using the term "Self Signed" certificate they may
        mean that their internal CA is signing it which is supported.       
        Or they may literally be clicking "create a self signed certificate"
        in IIS where the certificate is issued to and by the StoreFront server.

 

     3.  A Self signed Certificate created in StoreFront using
          the "create a self signed certificate" option in IIS:

  •         Is not supported in production by Citrix.

  •         Since it is issued by a StoreFront server it isn't distributed
            to domain joined machines.

  •         Requires that each client have the self signed certificate
             installed in the MMC console of every client machine.

        
           Note: This can be a hassle if there are many end point.

 

     4.  If needed you can test with a self signed certificate on the endpoint by:

  1.    Going to MMC on the client machine 

  2.    Adding the snapin for certificates > choose computer account > Ok

  3.    Right click private and import the self signed certificate.

  4.    Then right click trusted root certificate and import it there as well.

  5.    Since the self signed certificate is issued to and by itself it is a Server certificate as well as a       
        Root CA and must be imported to be trusted root on the client MMC.

  6.    Right click trusted root on the left pane and click import and find the same certificate.

 

[ √ ] Types of Certificate Hash algorithms:


        You can view the algorithm used by inspecting a certificate and scrolling through the details tab.
           
    SHA1 -    These are deprecated by the industry because they don't meet the security requirements.
                    https://docs.microsoft.com/en-us/security-updates/securityadvisories/2017/4010323
                    If you see a customer using a SHA1 certificate internally advise that
                    they will need to switch to SHA2 / SHA256 or greater.

    SHA256 - The standard Certificate that is issued by a Public CA.  
                     
    SHA2    -   These certificates are widely adopted because of the bandwidth and utility.  
                      SHA-256 is generally faster on 32-bit processors
                     Typically it is the standard chosen for internal CA's as well. endpoint support -                                                                           
                      https://support.citrix.com/article/CTX200114

    SHA384 -  More secure than SHA2 /SHA256 and is supported but rarely used.

    SHA512 -  It is generally faster on 64-bit processors and contains collision
                      resistance which prevents attacks via Quantum computers.

             Note: If a customer is attempting to use a SHA512 certificate with Server
                       2012r2 the following patch must be applied. (Click here for more information)
                       
                       SHA512 is disabled in Windows when you use TLS 1.2 so the
                       same patch needs to be installed on the clients as well.
                       https://support.citrix.com/article/CTX212883

(Speed comparisions mentioned can be measured with the command "openssl speed sha256 sha512".)

Please review the following link to see the support for ADC 
https://docs.citrix.com/en-us/netscaler/12/ssl/server-certificate-support-matrix-on-the-adc-appliance.html


[ √ ] Internal Client testing: (Direct to StoreFront)

 


    1. Test with the hosts file pointed to one StoreFront server at
        a time bypassing the LB VIP on the ADC.

        
        This will verify if going direct will function:

                   Open notepad as an administrator on the client machine.
                   Edit hosts file under C:\Windows\System32\Drivers\ETC
                   Change file types to all to see the files.
                   Append the following example with the Customer's information:

            192.168.1.11            base.url.local
            <StoreFront IP>       <BASE URL>


        StoreFront supports up to 6 servers in a group
       (But there is no benefit to having more than 3 in a group)
        Keep testing one at a time by changing the IP and setting it to
        resolve to the same baseURL.

        Each time just save notepad without closing it.
        Open cmd line and ping the base.url.local fqdn to make sure that
        it now resolves to the StoreFront IP.
           
    Then do a test with the browser and Workspace App as needed.

        

    2. Click view to inspect the certificate in a browser if the issue persists.

   
        Verify that the certificate name matches the BaseURL exactly or
        is a Wildcard *.fqdn. 
What you should see is the certificate that
        is bound to HTTPs (443) in IIS for that server you are pointing to.
    
        Inspect certificate from the client machine:

            1. Click the lock at the top in Internet Explorer to
                 inspect/view the certificate.
            2. Verify that there is a Chain under details.
            3. Check that the certificate is valid and not expired
            4. It should show a Root Certificate Authority as well as
                the Server certificate under details.
            5. For a public CA (Such as Godaddy) it will show an intermediate
                certificate in between the Root and Server certificates.
                           

 

    3. Test over HTTP if you are having an issue with one server in the
        group to narrow it down.


        1. Test on the StoreFront server directly over Http://localhost/Citrix/StoreWeb from a browser.
        2. Then test on the client to the IP of StoreFront Http://x.x.x.x/Citrix/StoreWeb
        3. Skip Certificate warnings and click proceed if using Chrome.

        4. If the Store doesn't function in a test over HTTP it could indicate an issue with the Store.
        5. You can change the BaseURL in StoreFront to be http temporarily if needed by going to
            Server group > Change Base URL > Remove the "S" in Https
            Then perform an IISRESET from command line with administrative rights.
            

  •               This way you can make sure that the Store is functional.
  •               You may need to create a test store.
  •               If a test store resolves it you can delete the test store and recreate the original.
  •               Otherwise you may need to reset StoreFront to factory settings (CTX200239or reinstall.
                                                                                                                                                          

      Note:  In either instance you can rejoin it to another (good) server in the group
                 to transfer the configuration.

                 Resetting or reinstalling will give you the option to rejoin a group
                 or create a deployment. However if it is a single StoreFront server take a
                 snapshot and manually make note of all the configuration before proceeding.

                 The recommended uninstallation method for StoreFront if needed is:
            
                    1. Create a snapshot
                       Backup subscriptions if needed so they can be restored after                                                                                                     
                       (https://support.citrix.com/article/CTX216295)
                       Take screenshots and/or make note of the configuration.
                    2. Remove StoreFront 
                    3. Remove the IIS Role.
                    4. Reboot
                    5. Open Server manager and verify that IIS removal has been completed
                    6. Reinstall StoreFront which also reinstalls IIS.
                    7. If it still fails over HTTP there is something wrong with the server
                       and the customer will need to stand up a new VM. 
            

     4. The customer might have attempted to install a certificate downloaded
          from a CA but doesn't contain a private key. 
          In order for a certificate to be valid it must have a private key.
          If the following steps do not apply, then a suitable certificate
          containing a private key must be used.

          This may be because the certificate was provided by the Administrator
          that manages the CA. If the file provided ends in a .CER extension it
          just needs to be completed using the same IIS  server that the CSR
         (Certificate Signing Request) Originated from.

            1. IIS management console > Hostname on left > Server Certificates
                > Complete Certificate Request.

            2. Browse for the .CER file > Input a name that is unique so you can
                recognize it when binding. 
            3. Set the Certificate Store to personal from the drop down > next.
            4. Bind certificate by going to Default Website > bindings > Edit
                or add an HTTPS binding
(If it doesn't exist)
            5. Find the unique friendly name you chose and click ok.
            6. Run IISRESET
            
          Note:  For expired certificates or having the wrong type of certificate bound.
                     Follow these directions:
                     To obtain a Certificate using web enrollment through IIS -                                                                                                           
                     https://support.citrix.com/article/CTX200292
                     Run IISRESET via command line with administrative
                     rights after you bind the new Certificate in IIS.

 

5. Once you have a certificate installed with the private key in IIS you
    can export it if it was marked as exportable.

           Follow https://support.citrix.com/article/CTX206492 to get a PFX
           certificate to other StoreFront servers in the group.

        Notes: 
        
        1. If you have tested all servers individually and still cannot reproduce the issue,
               then the issue is most likely on the (Netscaler) / ADC LB VIP / Gateway.
         2. testing from more than one client machine could rule out a client side problem.

6.  If you get the following error it may be due to the certificate
      not being marked as exportable.
      You may see this issue after installing a certificate in IIS. 

In the browser you will see the following: 

This page can’t be displayed

Turn on TLS 1.0, TLS 1.1, and TLS 1.2 in Advanced settings and try
connecting to https://<fqdn> again. If this error persists,
it is possible that this site uses an unsupported protocol or cipher
suite such as RC4 (link for the details) <http://go.microsoft.com/fwlink/?LinkId=735074>,
which is not considered secure. Please contact your site administrator.


If you remove the binding and attempt to add it back you may see the error:
hresult

           Most of the time this can be fixed by checking   
           “Allow this certificate to be exported
            However when importing a PFX certificate into IIS and left unchecked
           it results in a broken keyset which is why the error occurs.
           (For more information please visit this link)
            
           User-added image
 

[ √ ] Browsers


All browsers share the same shortcut to clear all cache CTRL+SHIFT+DEL
In control panel > within Internet Properties > Content > Clear SSL State 
Use InPrivate NewPrivate or Incognito modes to test.

    Internet Explorer - Test using In private mode by clicking the top right "gear icon" then click safety > InPrivate browsing.
    FireFox - New Private Window CTRL+SHIFT+P or select the 3 "-"'s top right and select New Private Window
    Edge - Click the "..." on the top right then choose New inPrivate Window
 
    Chrome - You may get an error with chrome with relation to the Error: "Subject Alternative Name Missing" or                                                                 
                    NET::ERR_CERT_COMMON_NAME_INVALID or "Your connection is not private"
                    If the subject alternate name is missing which can be seen in the details of a certificate by inspecting it
                    and scrolling through until you see dns= 
                    This might happen even if Internet Explorer and Edge have no issues.

 

  •         If you need to create a certificate request that contains Subject Alternate Name name:

                
                 1. Open MMC > Add snappin > Certificates > Computer account.
                 2. Right click private > all tasks > advanced Operations > Create Custom Request > next.
                 3. Select Certificate Enrollment Policy page > Active Directory Enrollment Policy > next.
                 4. On the Custom Request page > select the Web Server template and PKCS#10 format > next.
                 5. Click details > Properties
                 6. Subject tab > Subject Name field > select Common Name > add fqdn (I.e. SF baseURL) 
                    > Alternative Name field > DNS > add the fqdn
                     Under the private Key tab > Select Key options > Key size >2048 > 
                     Check to make exportable if you want to use on other Servers. > Select Hash Algorithm > Select sha256
                 7. Use certreq and certutil to submit the CSR request file to the CA or submit via web enrollment(CTX200292).
                     (Reference for commands certreq and certutil)
                  8. Verify that it exists in the private certificates within MMC.
                  9. Open it and view the details to confirm the Subject Alternate Name  DNS= is now complete
                 10. Verify that it has a private key.
                 11. Bind to port 443 in IIS  > default web site > bindings > choose file in dropdown > ok > IISRESET
                 12. Test with Chrome using hosts file to point directly to the StoreFront server's IP in an Incognito window.    

                 How to create a custom SAN Certificate Request.

        Test using Incognito mode CTRL+SHIFT+N or click "..." top right and select New Incognito Window

            1. Check that you’ve updated ChromeOS to the latest version.
                To update Chrome, press the Customize Google Chrome button.
                Select Help > About Google Chrome to open the tab shown directly below.

            2. Open the Chrome browser; and click the Customize Google
                Chrome button at the top right of its window.
                Select Settings to open the tab in the snapshot directly below.
                Press Advanced to open further options.
                Scroll down to and select the Open proxy settings
                option to open the window below.

            3. If you need to recreate the certificate because there is
                a DNS error only with Chrome, you will need to use the MMC
                console to create the certificate and add the DNS properties before
                submitting the Certificate request to the CA (Shown above).


 

[ √ ] Workspace App Clients:



Windows: 

    Run ipconfig /flushdns from command line to flush DNS.
    In control panel > Under internet properties > content > clear SSL state

    You can use https://support.citrix.com/article/CTX134341
     to bypass the Certificate requirement for testing purposes when testing with HTTP.
    
    Note:  
       
       This can only be done with the Windows Platform and
        all other platforms require a full certificate chain. 
        Also, the baseURL needs to be changed to HTTP while testing with CTX134341 
        if it is Set to HTTPS by going to server group and removing the "S" from HTTPS.

      
Mac: 
    May require a root certificate and intermediate imported into the Keychain.
    Use Safari to browse to the Internal or External Site.
    Click on the lock icon to inspect the certificate.
    Drag the "Picture" of the certificate to the desktop or another directory to download it.

        1. From the browser, open the directory in which
             you downloaded the root cert file.
        2. Double-click the certificate.
        3. You are prompted to store the certificate in the login
             keychain or the system keychain. 
        4. To make the certificate available to all users of
             this system, select system keychain.
        5. In Keychain Access, select the System keychain;
            then select Cloud Services Root CA certificate.
        6. Select File > Get Info and expand the Trust section.
        7. Change Secure Sockets Layer (SSL) value to Always Trust.
        8. Close the dialogs and enter your password.

ChromeBook: 

    Check that you have the latest updates to ChromeOS
    At the bottom right, select the time. > Select Settings >
    Select Menu About Chrome OS > Under "Google Chrome OS,"
    you'll see which version of the Chrome operating system
    your Chromebook is using. Select Check for Updates.

HTML5: 

    Must be used with a Netscaler or you will get an error
    referencing https://support.citrix.com/article/CTX134123
    Otherwise StoreFront must be used with HTTP or it will fail.

    Ensure that the latest version is installed on each StoreFront
    server https://www.citrix.com/downloads/workspace-app/html5/

Linux:
        
    How to Import the certificates on a fat client.
     (Thin clients are supported by the thin client vendors directly
       https://support.citrix.com/article/CTX234462)
           
    1. open terminal window
    2. sudo firefox  (this will allow us to save the files we are
        going to export without permission issues)
       *Otherwise save it first and copy it to the cacerts folder path below*
    3. browse to the fqdn for the ADC Gateway or StoreFront
    4. click the lock icon in the browser
    5. browse to see more information on the certificate
    6. when you see the certificate chain click on each one to export
    7. one at a time starting with the root and ending with the server cert
        click the export button after highlighting each
    8. export each one to /opt/Citrix/ICAClient/keystore/cacerts  and append  
        ".pem" to each file name before saving.
    9. when you are finished you can break out ctrl+c in the open
         terminal window to close firefox.
    10. navigate to /opt/Citrix/ICAClient/keystore/cacerts 
    11. run sudo ../../util/ctx_rehash
    12. then try to launch

        If issues persist you may need to do a symbolic link to the
        mozilla keystore or the Linux keystore by doing the following:
        sudo ln -s /usr/share/ca-certificates/mozilla/* /opt/Citrix/ICAClient/keystore/cacerts/
        Then run sudo ../../util/ctx_rehash again

        If that still doesn't help run the following:
        sudo ln -s /etc/ssl/* /opt/Citrix/ICAClient/keystore/cacerts/
        sudo ln -s /etc/ssl/certs/* /opt/Citrix/ICAClient/keystore/cacerts/
        Then run sudo ../../util/ctx_rehash again


        User-added image

[ √ ] External Client testing: (LB VIP/GW VIP on ADC)


    If the issue can't be reproduced going directly to StoreFront the
    issue may be on the ADC/Netscaler.
    The baseURL will normally resolve to the Load Balancer Virtual server IP (LB VIP).
        1. Open Netscaler configuration to Inspect LB VIP 
        2. Goto    Traffic Management > Load Balancer > Virtual Servers.

            Ping the base URL from a machine that doesn't have anything in the hosts or 
            place a "#" at the front of the entry you made in the HOSTS file
            earlier to comment it out and save it. To find the IP of the LB VIP just use
            command line and resolve the IP address.
            Find the Virtual Server and inspect the certificate.

        3. Verify that the correct certificate is installed and is not expired and
            is chained correctly  (Server > Intermediate > Root)
        4. Otherwise install the correct certificate and link.
            (https://support.citrix.com/article/CTX205404)

           Additional configuration and considerations for endpoints:

            When troubleshooting:

            External Gateway certificates can be verified by inputting the
            Gateway FQDN into one of the following sites
            to check that the certificate chain is linked and bound correctly.

                Https://www.digicert.com/help  - Simplified overview with chain.
                Https://ssllabs.com/ssltest - More advanced overview with SSL score and tests against all platforms.
                (to rerun the test clear cache at the top)

                     If you see that the Certificate is not trusted because of a broken chain.
                1. Navigate to Traffic Management > SSL > Server Certificates >
                    Right Click correct certificate and click link
                2. to link the Server Cert to the Intermediate > Repeat for Intermediate to Root.
                3. Then test again with the external sites above to check the certificate chain is correct
                   and that the FQDN of the Gateway matches the certificate.

Issue/Introduction

This guide covers troubleshooting StoreFront certificate issues with configuration and installation from the StoreFront perspective for integration with the following: Web Browsers, Workspace App, ADC Load balancer, Citrix Gateway, and Virtual Desktop Delivery Controllers. It does not include FAS, Smartcard, MFA, SNI, XenMobile / Citrix Endpoint Management integration with StoreFront. For internal smart card authentication to StoreFront please follow: https://support.citrix.com/article/CTX139201.