This article introduces the method to enable DTLS encryption for Linux VDA and provides general troubleshooting tips as well as common error scenarios.

---

Instructions

Datagram Transport Layer Security (DTLS) encryption for UDP-based ICA/HDX connections was previously an experimental feature in Linux VDA 7.17; in the 7.18 release, it is now a fully supported feature.

Enable DTLS encryption

Customers can complete the following 3 major steps to enable this feature: 

1. Verify that Adaptive Transport is enabled

2. Enable DTLS encryption for UDP connections on the Linux VDA

3. Enable SSL encryption on the Linux VDA

Note:
Currently, Linux VDA supports both DTLS 1.0 and DTLS 1.2 (requires Citrix Receiver for Windows 4.12 and above). If the Citrix Receiver you’re using supports only DTLS 1.0 (for example, Citrix Receiver for Windows 4.11), set SSLMinVersion to TLS_1.0 and SSLCipherSuite to COM or ALL on Linux VDA.
 

Troubleshooting

How to confirm DTLS encryption is active

After configuration, you can follow below steps to check if the ICA connection is encrypted using DTLS as expected:

1. Install Wireshark on the Linux VDA
2. Launch the Wireshark on the Linux VDA in a user ICA session, you’ll be able to see the protocol used. It may show DTLSv1.0, DTLSv1.2 or QUIC.

Here are some common error scenarios you might encounter while using DTLS encryption on Linux VDA:

Common error scenario #1: Session fails to launch with error “Socket operation on non-socket (Socket Error 10038)” on latest version of Citrix Receiver.

Problem Cause:
This is usually caused by mis-configuration of SSL encryption part:

• The client host doesn’t trust the CA certificate used by Linux VDA. You may get more specific error message “SSL Error 61: You have not chosen to trust “*.x.x”, the issuer of the server’s security certificate” on older versions of Citrix Receiver for Windows (for example, 4.9.2000).

• The Delivery Controller is configured to enable SSL encryption for ICA sessions while Linux VDA doesn’t. You may get more specific error message "The Citrix SSL Server you have selected is not accepting connections" on older versions of Citrix Receiver for Windows (for example, 4.9.2000).

Solution:

• Follow the instructions on KB CTX230023 to ensure the client host trusts the CA certificate used by the Linux VDA.
• Ensure the SSL encryption is configured correctly, refer to KB CTX230024-Error:"The Citrix SSL Server you have selected is not accepting connections" for detailed instructions.

Common error scenario #2: The ICA connection is established using TLS instead of DTLS.

Problem Cause:
There are multiple possible causes for this issue:

• Adaptive Transport is not correctly configured and EDT is not active.
• DTLS encryption on Linux VDA is disabled (“HKLM\System\CurrentControlSet\Control\Terminal Server\Wds\icawd\Tds\udp\ fDTLSEnabled”).
• Linux VDA supports both DTLS 1.0 and DTLS 1.2 (requires Citrix Receiver for Windows 4.12 and above). By default, the SSLMinVersion is set to TLS_1.2 and SSLCipherSuite is set to ALL. However, older versions of Citrix Receiver for Windows (for example, 4.11) only supports DTLS 1.0. With default SSLMinVersion setting (TLS_1.2) or TLS_1.0 with unsupported SSLCipherSuite (GOV), the DTLS connection will fail and Citrix Receiver will fall back to TCP/TLS connection.

Solution:

• Ensure EDT is active, reference KB CTX232462:How to Troubleshoot Adaptive Transport Issues on Linux VDA for detailed instructions.
• Ensure the value of the registry key “HKLM\System\CurrentControlSet\Control\Terminal Server\Wds\icawd\Tds\udp\ fDTLSEnabled” is set to 0x00000001. Otherwise, use the following command to enable:

• Ensure Citrix Receiver supports Linux VDA configured SSLMinVersion and SSLCipherSuite settings:

• Use Citrix Receiver for Windows 4.12 and above with default SSLMinVersion and SSLCipherSuite settings to use DTLS 1.2.
• For earlier versions of Citrix Receiver which only supports DTLS 1.0, set SSLMinVersion to TLS_1.0 and SSLCipherSuite to COM or ALL on Linux VDA. You can either use the enable_vdassl.sh tool or set the registry key to configure.
  • Use the enable_vdassl.sh tool. As an example, set the SSLMinVersion to TLS_1.0 and SSLCipherSuite to COM. /opt/Citrix/VDA/sbin/enable_vdassl.sh -Enable -Certificate "/root/myCert/myCA/certs/server.pem" -RootCertificate "/root/myCert/myCA/certs/myca.crt" -SSLMinVersion "TLS_1.0" -SSLCipherSuite "COM"

• Set the registry key under HKLM\System\CurrentControlSet\Control\Citrix\WinStations\ssl:

 

How to get more detailed logging of DTLS encryption

• SSH to Linux VDA server
• Set TD/TD_INPUT/TD_OUTPUT log level to VERBOSE use command setlog:

• Look for “TdEdt*” “TdSsl*” lines in /var/log/xdl/hdx.log
• For general ICA connection troubleshooting purpose, you may need to collect CDF traces of Citrix Receiver and Wireshark network traces on both client host and Linux VDA at the same time when reproducing the issue.