• Time of error

• FQDN/IP of Controller(s) / DDC(s)

• FQDN/IP of VDA

• Details of troubleshooting performed so far
  • How to Troubleshoot Virtual Delivery Agent (VDA) Registration Issues (citrix.com)
• Transaction ID

Visual Logs:

• • Video of the issue.
  • Director Screen Shot.

DDC:

• • Scout Diagnostic Logs (e)
  • CDF Trace – All modules, multiple sequential files. Start trace before starting session launch (e)
  • Network trace
    • Wireshark Trace or
    • Netsh trace - https://learn.microsoft.com/en-us/windows/win32/ndf/using-netsh-to-manage-traces
  
  
  • Event Viewer Logs – Export as CSV and as EVTX (2 sets) (e)
    • Application
    • System

VDA

• • Scout Diagnostic Logs (e)
  • CDF Trace – All modules, multiple sequential files. Start trace before starting session launch (e)
  • Network trace
    • Wireshark Trace or
    • Netsh trace - https://learn.microsoft.com/en-us/windows/win32/ndf/using-netsh-to-manage-traces
  
  
  • Event Viewer Logs – Export as CSV and as EVTX (2 sets) (e)
    • Application
    • System
  • Registry Settings export (e)
    • HKLM\Software\Policies\Citrix\VirtualDesktopAgent
    • HKLM\Software\Citrix\VirtualDesktopAgent
    • HKLM\Software\Wow6432Node\Citrix\
  • Citrix Health Assistant output (e)- https://support.citrix.com/article/CTX207624                or      

           Citrix Scout | Citrix Virtual Apps and Desktops

• • Enable Kerberos logging(which will be logged in System event logs) (e) - https://learn.microsoft.com/en-us/troubleshoot/windows-server/active-directory/enable-kerberos-event-logging
  • WCF logs (if requested)

Collect Working Logs and Not Working Logs.

Troubleshooting Articles:

• How to Troubleshoot Virtual Delivery Agent (VDA) Registration Issues (citrix.com)

References:

• VDA registration | Citrix Virtual Apps and Desktops 7 2311

Application/

Desktop Enumeration Fails

• Username
• Application name (executable name) 

• FQDN/IP of Storefront Server(s) 

• FQDN/IP of Controller / DDC 

• FQDN/IP/vServer name of Citrix Gateway (if in use):

• Additionally troubleshoot if storefront was able to contact controller. If not, also collect storefront debug logs
• Details of troubleshooting performed so far 

Visual Logs:

• • Video of the issue.
  • If video not possible, screenshots of errors with time visible.

DDC

• • Scout Diagnostic Logs (e)
  • CDF: All modules, multiple sequential files (e)
  • Network trace
    • Wireshark Trace or
    • Netsh trace - https://learn.microsoft.com/en-us/windows/win32/ndf/using-netsh-to-manage-traces
  • Event Viewer Logs – Export as CSV and as EVTX (2 sets) (e) 
    • Application
    • System

CWA

• • CDF: All modules, multiple sequential files (e)
  • Event Viewer Logs – Export as CSV and as EVTX (2 sets) (e)
    • Application
    • System
  • Network trace
    • Wireshark Trace or
    • Netsh trace - https://learn.microsoft.com/en-us/windows/win32/ndf/using-netsh-to-manage-traces

StoreFront:

• • Storefront logs https://support.citrix.com/article/CTX139592 (e)
  • Network trace
    • Wireshark Trace or
    • Netsh trace - https://learn.microsoft.com/en-us/windows/win32/ndf/using-netsh-to-manage-traces
  
  
  • Event Viewer Logs – Export as CSV and as EVTX (2 sets) (e)
    • Application
    • System
    • Application and Services Logs > Citrix Delivery Services (e) 

Citrix ADC Gateways:

• • Support Bundle - https://support.citrix.com/article/CTX127900
  • NSTrace
  • Network diagram

Collect Working Logs and Not Working Logs. 

References:

• Launching Process of Published Apps and Desktops for the Internal Network Through Citrix Storefront

Application/

Desktop Launch Fails

• Username 
• Application name (executable name) 
• Session Id 
• Time of launch/failure 
• Time of reconnection 

• FQDN/IP of Storefront Server(s): 

• FQDN/IP of Controller / DDC: 

• FQDN/IP/vServer name of Citrix Gateway (if in use) :

• FQDN/IP of VDA :

• FQDN/IP of Endpoint :

• Also troubleshoot where the failure occurred i.e., if ICA file was generated and if launch reached the Server/VDA
• Details of troubleshooting performed so far 
• Transaction ID 

Visual Logs:

• • Video of the issue.
  • If video not possible, screenshots of errors with time visible.

DDC:

• • Scout Diagnostic Logs (e)
  • CDF: All modules, multiple sequential files (e)
  • Network trace
    • Wireshark Trace or
    • Netsh trace - https://learn.microsoft.com/en-us/windows/win32/ndf/using-netsh-to-manage-traces
  
  
  • Event Viewer Logs – Export as CSV and as EVTX (2 sets) (e)
    • Application
    • System

VDA:

• • Scout Diagnostic Logs (e)
  • CDF Trace – All Modules, multiple sequential files (e)
  • Network trace
    • Wireshark Trace or
    • Netsh trace - https://learn.microsoft.com/en-us/windows/win32/ndf/using-netsh-to-manage-traces
  
  
  • Event Viewer Logs – Export as CSV and as EVTX (2 sets) (e)
    • Application
    • System
  • SysInternals ProcMon

FAS (If FAS is used)

• • Scout Diagnostic Logs (e)
  • CDF: All modules, multiple sequential files (e)
  • Network trace
    • Wireshark Trace or
    • Netsh trace - https://learn.microsoft.com/en-us/windows/win32/ndf/using-netsh-to-manage-traces
  
  
  • Event Viewer Logs – Export as CSV and as EVTX (2 sets) (e)
    • Application
    • System

CWA:

• • CDF: All modules, multiple sequential files (e)
  • Network trace
    • Wireshark Trace or
    • Netsh trace - https://learn.microsoft.com/en-us/windows/win32/ndf/using-netsh-to-manage-traces
  
  
  • ICA File logging – Upload the log file
  • Event Viewer Logs – Export as CSV and as EVTX (2 sets) (e)
    • Application
    • System

 Storefront Server(s)

• • Scout Diagnostic Logs (e)
  • Verbose Logs: https://support.citrix.com/article/CTX139592 (e)
  • Network trace
    • Wireshark Trace or
    • Netsh trace - https://learn.microsoft.com/en-us/windows/win32/ndf/using-netsh-to-manage-traces
  
  
  • Event Viewer Logs - Export as CSV and as EVTX (2 sets) (e)
    • Application
    • System
    • Citrix Delivery Services (under Applications and Services)

Citrix ADC Gateway(s) – if in use:

• • Support Bundle - https://support.citrix.com/article/CTX127900
  • NSTrace
  • Network diagram

Collect Working Logs and Not Working Logs.

References:

• Citrix Virtual Apps and Desktops: Session Launch Issues
• Citrix Director - Failure Reasons Troubleshooting Guide

• Username
• Application name (executable name)
• Session Id 
• Time of first launch 
• Time of disconnection 
• Time of reconnection 

• FQDN/IP of Controller / DDC :

• FQDN/IP/vServer name of Citrix Gateway (if in use) :

• FQDN/IP of VDA :

• FQDN/IP of Endpoint :

• Transaction ID 
• Collect complete sequence trace with first launch, disconnect and reconnect.
• Details of troubleshooting performed so far 

Visual Logs:

• • Video of the issue.
  • If video not possible, screenshots of errors with time visible.

DDC

• • Scout Diagnostic Logs (e)
  • CDF: All modules, multiple sequential files (e)
  • Network trace
    • Wireshark Trace or
    • Netsh trace - https://learn.microsoft.com/en-us/windows/win32/ndf/using-netsh-to-manage-traces
  
  
  • Event Viewer Logs – Export as CSV and as EVTX (2 sets) (e)
    • Application
    • System

VDA:

• • Scout Diagnostic Logs (e)
  • CDF Trace – All Modules, multiple sequential files (e)
  • Network trace
    • Wireshark Trace or
    • Netsh trace - https://learn.microsoft.com/en-us/windows/win32/ndf/using-netsh-to-manage-traces
  
  
  • Event Viewer Logs – Export as CSV and as EVTX (2 sets) (e)
    • Application
    • System
  • SysInternals ProcMon

CWA:

• • CDF: All modules, multiple sequential files (e)
  • Network trace
    • Wireshark Trace or
    • Netsh trace - https://learn.microsoft.com/en-us/windows/win32/ndf/using-netsh-to-manage-traces
  
  
  • Event Viewer Logs – Export as CSV and as EVTX (2 sets) (e)
    • Application
    • System

Citrix ADC Gateway(s) – if in use:

• • Support Bundle - https://support.citrix.com/article/CTX127900
  • NSTrace
  • Network diagram

Collect Working Logs and Not Working Logs.

References:

• Citrix Virtual Apps and Desktops: Session Launch Issues
• Citrix Director - Failure Reasons Troubleshooting Guide

• Username
• Application name (executable name)
• Session Id
• Time of first launch
• Time of disconnection
• FQDN/IP of Controller / DDC:

• FQDN/IP/vServer name of Citrix Gateway (if in use):

• FQDN/IP of VDA:

• FQDN/IP of Endpoint:

• Collect complete sequence trace with first launch, disconnect 
• Details of troubleshooing performed so far

Visual Logs:

• • Video of the issue.
  • If video not possible, screenshots of errors with time visible

VDA:

• • Scout Diagnostic Logs (e)
  • CDF Trace – All modules, multiple sequential files. Start trace before starting session launch (e)
  • Network trace
    • Wireshark Trace or
    • Netsh trace - https://learn.microsoft.com/en-us/windows/win32/ndf/using-netsh-to-manage-traces
  
  
  • Event Viewer Logs – Export as CSV and as EVTX (2 sets)
    • Application (e)
    • System (e)

CWA:

• • CDF Trace – All modules, multiple sequential files. Start trace before starting session launch (e)
  • Network trace
    • Wireshark Trace or
    • Netsh trace - https://learn.microsoft.com/en-us/windows/win32/ndf/using-netsh-to-manage-traces
  
  
  • Event Viewer Logs – Export as CSV and as EVTX (2 sets)
    • Application (e)
    • System (e)

Citrix ADC Gateway(s) – if in use:

• • Support Bundle (e) - https://support.citrix.com/article/CTX127900
  • NSTrace
  • Network diagram.

Collect Working Logs and Not Working (e) Logs. 

Application/

Desktop Slow Launch

• Username
• Application name (executable name)
• Session Id
• Time of first launch
• Time when App or Desktop visible to user
• Expected time taken for launch in normal use case
• From users perspective the launch experience starts once they initiate app launch, the delay could be anywhere along the path, identify aera to focus on, e.g. enumeration delay, network delay, logon script delay, profile delay, session creation delay or delay after session created
• Details of troubleshooting performed so far. 

Visual Logs:

• • Video of the issue.
  • If video not possible, screenshots of errors with time visible.

Director

• • Scout Diagnostic Logs (e)
  • Logon Duration snapshot User Details view > Session Logon (e)

DDC

• • Scout Diagnostic Logs (e)
  • CDF Trace – All modules, multiple sequential files. Start trace before starting session launch (e)

VDA

• • Scout Diagnostic Logs (e)
  • CDF Trace – All modules, multiple sequential files. Start trace before starting session launch (e)
  • Event Viewer Logs – Export as CSV and as EVTX (2 sets) (e)
    • Application
    • System
  • Autoruns tool output - Auto start programs configured during boot and log on
  • If Profile issue, enable profile logging Check Profile Management log files | Profile Management (citrix.com)
  • If network issue 
    • Network trace
      • Wireshark Trace or
      • Netsh trace - https://learn.microsoft.com/en-us/windows/win32/ndf/using-netsh-to-manage-traces
    
    
    • Process Monitor

CWA

• • • CDF Trace – All modules, multiple sequential files. Start trace before starting session launch (e)

Collect Working Logs and Not Working Logs.

Troubleshooting Articles:

• How to Troubleshoot Slow Logons on XenApp
• Diagnose user logon issues | Director (citrix.com)

• Username
• Application name (executable name)
• Session Id
• FQDN/IP of VDA:
• Time of first app launch
• Time of second app launch
• Collect complete sequence trace with first launch and second launch
• Details of troubleshooting performed so far
• Any Seamless Flags in Place

Visual Logs:

• • Video of the issue.
  • If video not possible, screenshots of errors with time visible.

DDC

• • Scout Diagnostic Logs (e)
  • CDF Trace – All Modules, multiple sequential files. Start trace before ICA session launch. (e)

VDA:

• • Scout Diagnostic Logs (e)
  • CDF Trace – All Modules, multiple sequential files. Start trace before ICA session launch. (e)

CWA:

• • CDF Trace – All Modules, multiple sequential files. Start trace before ICA session launch. (e)

Collect Working Logs and Not Working Logs.

References:

• Session Sharing Explained with Troubleshooting Steps

• Time of error
• Catalog name
• Details of troubleshooting performed so far. 
• If it's a Cloud customer: CustomerId

Visual Logs:

• • Video of the issue.
  • If video not possible, screenshots of errors with time visible

DDC

• • Scout Diagnostic Logs (e)
  • CDF: Simultaneous all module CDF traces from all the DDCs in the Site. If it's a Cloud customer and the VMs are not hosted in Azure (or if ProxyHypervisorTrafficThroughConnector  is enabled), then also get CDF Traces from all the Cloud Connectors. (e)

VDA:

VM prep logs if this is a problem with catalog creation due to image prep. See the following article (e): https://support.citrix.com/s/article/CTX202656-catalog-creation-mcs-preparation-of-master-vm-image-fails-with-exception-id-xdds7be07c6a?language=en_US 

Troubleshooting articles:

• Catalog Creation (MCS), Preparation of Master VM Image Fails With Exception ID XDDS:7BE07C6A

• Target/server IP address
• MAC address of target and server

• • Network traces from target and from all PVS servers. See How to Capture a Network Trace for a PVS Target Device (citrix.com). (e)
  • If slow or stuck at something, also collect complete memory dump (e)

Collect Working Logs and Not Working Logs.

• (e)For PVS Collect Problem Report rather than CDF. Proper Collection should be:
• Restart Telemetry Service
• Reproduce issue
• Restart Telemetry Service
• Collect Problem Report

References:

• How to Capture a Network Trace for a PVS Target Device (citrix.com)

• Target  device MAC address
• Name and IP address, In case of vdisk  related cases context based on vdisk name,  affected versions etc.
• PVS 7.11 onwards uses Always On Tracing (enable AOT logging while collecting CDF traces

Configuration wizard logs: c:\programdata\citrix\provisioning services\config wizard.log

Streamprocess = https://support.citrix.com/s/article/CTX341570-citrix-provisioning-services-how-to-gather-cdf-traces?language=en_US

DB/Soapserver/console = https://support.citrix.com/s/article/CTX341570-citrix-provisioning-services-how-to-gather-cdf-traces?language=en_US

Target BSOD = Full memory dump - https://support.citrix.com/s/article/CTX322727-how-to-get-a-memory-dump-from-a-pvs-target-device?language=en_US

In the PVS console you can right click on a server object and choose> report a problem> follow the wizard.  This pulls down all the AOT traces, server event logs and general diagnostic and informational data that allows us to gain full context of the software/config in use......

• (e)For PVS Collect Problem Report rather than CDF. Proper Collection should be:
• Restart Telemetry Service
• Reproduce issue
• Restart Telemetry Service
• Collect Problem Report

For Streamprocess crashes, (e) full user crash dump +  (e) Streamprocess debug level logs

Collect Working Logs and Not Working Logs.

References:

• How to Capture a Network Trace for a PVS Target Device (citrix.com)
• How to troubleshoot the Citrix Provisioning Services Server
• PVS Config wizard fails to connect to the database

• Time of error
• Details of troubleshooting performed so far.

Visual Logs:

• • Video of the issue.
  • If video not possible, screenshots of errors with time visible

Storefront Logs:

• • Scout Diagnostic Logs (e)
  • Storefront logs https://support.citrix.com/article/CTX139592 (e)
  • Network trace ( useful for reviewing LDAP/Kerberos calls ) 
    • Wireshark Trace or
    • Netsh trace - https://learn.microsoft.com/en-us/windows/win32/ndf/using-netsh-to-manage-traces
  • Event logs (e):
    •  Application and Services Logs > Citrix Delivery Services or  Windows Logs > Application
    • Security ( useful when authentication failure is observed )
    • System 

Citrix ADC Gateway(s) – if in the path can be used to collect a network trace to decrypt information. Useful in cases where SAML authentication or when we require a decrypted network trace from Storefront to view responses in HTTP.

• • NSTrace
  • Network diagram

Troubleshooting Articles:

• • Citrix StoreFront Troubleshooting: Comprehensive Guide
  • Troubleshoot StoreFront | StoreFront  (citrix.com)

• Example of shortcut that is missing
• Login timestamp

Visual Logs:

• • Video of the issue.
  • If video not possible, screenshots of errors with time visible

CWA:

• • Receiver logs (e)
  • Procmon Log

Storefront Logs:

• • Storefront logs https://support.citrix.com/article/CTX139592 (e)
  • Scout Diagnostic Logs (e)

Collect Working Logs and Not Working Logs.

Troubleshooting Articles:

• • Shortcuts to apps not seen in the start menu of the published desktop

• Time of error
• Media/file/URL/
• Details of troubleshooting performed so far.

Visual Logs:

• • Video of the issue.
  • If video not possible, screenshots of errors with time visible

VDA

• • Scout Diagnostic Logs (e)
  • Policies - BCR Polices configured (VDA/Studio) (e)
  • CDF Trace – All modules, multiple sequential files. Start trace before starting session launch. (e)
  • Enable debug mode in extension (e)
    • Collect Console output
    • HAR data, see Creating a HAR file from your browser to share with support (citrix.com)
  • Event Logs APP (e)
    • Application and Services Logs >Citrx>MultiMedia>BCR
  • For CEF compatibility issues
    • Set HKLM\Software\Citrix\HdxMediaStream\

      RemoteDevConsolePort DWORD  (value between 1024 and 65535)

• • • Configure as per https://developer.chrome.com/docs/devtools/remote-debugging/local-server open chrome://inspect
    • Run test with a CEF sample application, cefclient, https://cef-builds.spotifycdn.com/index.html

• • For Proxy/authentication issues:
    • WireShark - Select both ethernet and localhost interfaces 

    • Set hdxwebproxy log

      HKLM\Software\Citrix\HdxMediaStream\LogFilePath

• • Registry settings export (e)
    • For 32 bit:

HKLM\SOFTWARE\Citrix\HdxMediastream\

• • • For 64-bit:
         HKLM\SOFTWARE\Wow6432Node\Citrix\HdxMediastream\

CWA:

• • Policies - BCR Polices configured (Client) (e)
  • CDF Trace – All modules, multiple sequential files. Start trace before starting session launch (e)
  • For Proxy/authentication issues:
    • WireShark - Select both ethernet and localhost interfaces 

Collect Working Logs and Not Working Logs.

Troubleshooting Articles:

• • How to Troubleshoot Browser Content Redirection

• Time of error
• Media/file/URL/
• Details of troubleshooting performed so far.

Visual Logs:

• • Video of the issue.
  • If video not possible, screenshots of errors with time visible

VDA

• • Scout Diagnostic Logs (e)
  • Policies - Rave Polices configured (VDA/Studio) (e)
  • CDF Trace – All modules, multiple sequential files. Start trace before starting session launch (e)
  • Event Logs APP (e)
    • Application and Services Logs >Citrx>MultiMedia>Rave
  • Registry settings export (e)
    • For 32 bit:

HKLM\SOFTWARE\Citrix\HdxMediastream\

• • • For 64-bit:
         HKLM\SOFTWARE\Wow6432Node\Citrix\HdxMediastream\

CWA:

• • Policies - Rave Polices configured (Client). (e)
  • CDF - All modules, multiple sequential files. Start trace before ICA session launch.(e)

Collect Working Logs and Not Working Logs.

Troubleshooting articles:

• • Flash Redirection Troubleshooting
  • Citrix Workspace App - Linux (RAVE Script) 

• Time of error (gap in audio, choppy, etc...)
• Media/file/
• Details of troubleshooting performed so far.

VDA

• • Scout Diagnostic Logs (e)
  • Policies - Audio Polices configured (VDA/Studio) (e)
  • CDF - All modules, multiple sequential files. Start trace before ICA session launch.
  • Event Logs (e)
    • Application and Services Logs >Citrx>MultiMedia>AudioSvc
  • Registry settings export (e)
    • For 32 bit:

HKLM\SOFTWARE\Citrix\Audio

• • • For 64-bit:
         HKLM\SOFTWARE\WOW6432Node\Citrix\Audio

• • For audio quality issues, a recording using an audio tool such as Audacity (simultaneous capture with client as noted below) when reproducing the issue (e)
  • Output of Citrix Audio Diagnostic Tool with recorded wav file. (e)

CWA:

• • Policies - Audio Polices configured (Client) (e)
  • CDF Trace – All modules, multiple sequential files. Start trace before starting session launch (e)
  • Registry settings export (e)
    • Computer\HKEY_LOCAL_MACHINE\SOFTWARE\(WOW6432Node)\citrix\ICA Client\Engine\Configuration\Advanced\Modules\ClientAudio
    • HKCU\Software\Citrix\ICA Client\Engine\Configuration\Advanced\Modules\ClientAudio
  • Audacity recording of audio that illustrates the issue (simultaneous capture with VDA -- as noted above) (e)

Collect Working Logs and Not Working Logs.

Troubleshooting Articles:

• Citrix Audio Diagnostic Tool
• Multimedia Audio

• Time of error
• User Name
• Call ID
• User Name
  
  
  
  
  

RTOP Logging (e)

Typically HROP logs can be collected by right clicking on the Citrix HDX RealTime Connector icon and clicking on Collect Logs.

Collect Working Logs and Not Working Logs.

• RTOP Analyzer: https://support.citrix.com/article/CTX214237
• https://docs.citrix.com/en-us/hdx-optimization/2-9-ltsr/troubleshooting.html

• Time of error
• User Name
• Call ID
• User Name

MTOP - How to collect logs

Collecting MTOP Logs - Citrix Docs

Teams Logs, CDF Traces and HdxTeams.exe/HdxRtcEngine.exe logs

• VDA:
  • CDF Trace
  • Scout Diagnostic Logs (e)
  • Procmon
    • filter:
      • path contains "9001", "9002"
      • Process Name contains "WebsocketAgent","WebsocketService","Teams","HdxRtcEngine"
    • Drop Filtered Events
  • Wireshark
  • Teams Logs
• Endpoint:
  • CDF Trace
  • Procmon
    • filter:
      • path contains "9001", "9002"
      • Process Name contains "WebsocketAgent","WebsocketService","Teams","HdxRtcEngine"
    • Drop Filtered Events
  • Wireshark
  • WebRPC logs
    • Set the following keys on Endpoint:
      • Key HKCU\Software\Citrix\HDXMediaStream
      • WebrpcLogLevel DWORD 0x0
      • WebrtcLogLevel DWORD 0x0
  • CWA Linux:
    • Enable\Disable Webrpc logging
      • edit /var/.config/citrix/hdx_rtc_engine/config.json
      • {"Webrpclevel" : 0, "WebrtcLogLevel":0}
    • Restart CWA
    • Locations:
      • HdxRtcEngine log: /tmp/hdxrtcengine/<W_M_H_M_S_Y>/hdxrtcengine.log
      • webrpc log : /tmp/webrpc/<W_M_D_H_M_S_Y>/webrpc.log
      •  webrtc log: /tmp/webrpc/<W_M_D_H_M_S_Y>/webrtc.log
      •  Kernel log: /var/log/syslog

Collect Working Logs and Not Working Logs.

References:

• Troubleshooting HDX Optimization for Microsoft Teams
• Monitor, troubleshoot, and support Microsoft Teams
• Optimization for Microsoft Teams (Classic)
• Troubleshooting for Microsoft Teams optimization – Citrix Workspace app for Chrome OS
• Troubleshoot and logs

• Time of error
• User name
• Session Id
• File Name for (reading/writing)
• Application used
• File copy time
• Behavior with RDP
• Details of troubleshooting performed so far

Visual Logs:

• • Video of the issue.
  • If video not possible, screenshots of errors with time visible

VDA: 

• • Scout Diagnostic Logs (e)
  • Policy: CDM polices configured (VDA/Studio) (e)
  • CDF Trace – All modules, multiple sequential files. Start trace before starting session launch (e)
  • Procmon
  • Copy of Tool, Application or file used by customer.

CWA:

• • Policy: CDM polices configured (VDA/Studio) (e)
  • CDF Trace – All modules, multiple sequential files. Start trace before starting session launch (e)
  • Procmon
  • Registry: Export
    • Computer\HKEY_LOCAL_MACHINE\SOFTWARE\(WOW6432Node)\citrix\ICA Client\Engine\Configuration\Advanced\Modules\ClientDrive

Collect Working Logs and Not Working Logs.

References:

• Client Drive Mapping(CDM)
• Client Drive Mapping(CWA-side)

• Time of error
• VID/PID of device

Visual Logs:

• • Video of the issue.
  • Screenshot 

VDA: 

• • Scout Diagnostic Logs (e)
  • Policy: USB polices configured (VDA/Studio) (e)
  • CDF: Simultaneous all module CDF traces (e)
  • Output of USB Diagnostics Tool | Citrix Virtual Apps and Desktops
  • USBView device details within ICA session  (e)

CWA:

• • CDF: Simultaneous all module CDF traces (e)

  Registry: Export

• • • 32 bit OS: HKEY_LOCAL_MACHINE\SOFTWARE\Citrix\ICA Client\GenericUSB
    • 64 bit OS: HKEY_LOCAL_MACHINE\SOFTWARE\Wow6432Node\Citrix\ICA Client\GenericUSB
  • USBView device details on the client without ICA session (e)

Collect Working Logs and Not Working Logs.

References:

• • Troubleshooting USB issues

• Time of error
• User name
• Session Id
• Printer Name
• Printer Drivers being used

VDA:

• • Scout Diagnostic Logs (e)
  • Policy: Printing polices (e)
  • CDF Trace – All modules, multiple sequential files. Start trace before starting session launch (e)
  • Event logs (e)
    • Log Name:      Application
      Source:        MetaFrameEvents
  • Registry:
    • HKEY_LOCAL_MACHINE\SOFTWARE\Policies\Citrix\<session id>\User\PrintingPolicies
    • For additional debug logs set according to EMF or XPS used
      HKLM\Software\Citrix\UniversalPrintDrivers\<EMF or XPS>\
      Set or create DWORD(32 bit) key "EnablePrintTicketCapsDump ", set it's value to to 1
      The filenames of the files that are created under %ProgramData%\Citrix\CDF are:
      ProcessName-DefaultPrintTicket.xml
      ProcessName-PrintTicketModified.xml
      ProcessName-PrintTicketConstraints.xml
      ProcessName-BoundPrinterCapabilities.xml
      ProcessName-UnBoundPrinterCapabilities.xml
      ProcessName-ValidatedPrintTicket.xml
      ProcessName-PrintCapabilities.xml
      Where ProcessName is the name of the process that loaded the Citrix Universal Printer Driver.
      The change will take effect after rebooting the VDA. 
  • Dump:
    • If troubleshooting indicates hung spooler or CpSvc, then multiple full user dumps of the affected service (using ProcDump -ma flag) in the timed intervals when the hang is seen. (e)

CWA: 

• • Policy: Printing polices
  • CDF Trace – All modules, multiple sequential files. Start trace before starting session launch (e)
  • Registry:

     

  • For additional debug logs enable

    • HKEY_LOCAL_MACHINE\Software\Citrix\ICA Client\PrintDebugFlags\
      Set or create DWORD(32 bit) key "EMF" or "XPS", set it's value to to 3
      Debug files will be located in %TEMP%\splXXXX\
      Where XXXX is some value that the windows print spooler generates for each print job.

Collect Working Logs and Not Working Logs.

References:

• Citrix Printing Quick Start Guide

• Time of error
• User name
• Session Id
• Printer Name
• Printer Drivers being used

Visual Logs:

• • Video of the issue.
  • Screenshot 

VDA

• • Scout Diagnostic Logs (e)
  • Policy: Printing polices (e)

• • CDF Trace – All modules, multiple sequential files. Start trace before starting session launch (e)
  • Event logs (e)
    • Log Name:      Application (e)
      Source:        MetaFrameEvents
  • Registry:
    • HKEY_LOCAL_MACHINE\SOFTWARE\Policies\Citrix\<session id>\User\PrintingPolicies

    • For additional debug logs set according to EMF or XPS used
      HKLM\Software\Citrix\UniversalPrintDrivers\<EMF or XPS>\
      Set or create DWORD(32 bit) key "EnablePrintTicketCapsDump ", set it's value to to 1
      The filenames of the files that are created under %ProgramData%\Citrix\CDF are:
      ProcessName-DefaultPrintTicket.xml
      ProcessName-PrintTicketModified.xml
      ProcessName-PrintTicketConstraints.xml
      ProcessName-BoundPrinterCapabilities.xml
      ProcessName-UnBoundPrinterCapabilities.xml
      ProcessName-ValidatedPrintTicket.xml
      ProcessName-PrintCapabilities.xml
      Where ProcessName is the name of the process that loaded the Citrix Universal Printer Driver.
      The change will take effect after rebooting the VDA. 

  • Dump (e)
    • If troubleshooting indicates hung spooler or CpSvc , then multiple full user dumps of the affected service (using ProcDump –ma flag) in timed intervals when the hang is seen

UPS

• • CDF Trace – All modules, multiple sequential files. Start trace before starting session launch (e)
  • Event logs (e)
    • Log Name:      Application
      Source:        MetaFrameEvents
  • Registry Settings export
    • "HKLM\SOFTWARE\\Citrix\\Universal Print Server"

• • • For additional debug logs enable

HKLM\Software\Citrix\Universal Print Server\PrintDebugFlags\
Set or create DWORD(32 bit) key "EMF" or "XPS", set it's value to to 3
Debug files will be located in C:\ProgramData\Citrix\Universal Print Server\Temp\splXXX\
Where XXXX is some value that the windows print spooler generates for each print job.
This change should not require any restart, however if the files don't appear after printing a job then restart.

• • Dump (e): 
    • If troubleshooting indicates hung UPS Server service, then multiple full user dumps of the affected service (using ProcDump –ma flag) in timed intervals when the hang is seen

CWA: 

• • Policy: Printing polices
  • CDF Trace – All modules, multiple sequential files. Start trace before starting session launch (e)
  • Registry:

  • For additional debug logs enable

HKEY_LOCAL_MACHINE\Software\Citrix\ICA Client\PrintDebugFlags\
Set or create DWORD(32 bit) key "EMF" or "XPS", set it's value to to 3
Debug files will be located in %TEMP%\splXXXX\
Where XXXX is some value that the windows print spooler generates for each print job.
Will need to restart receiver for this change to take effect.

Collect Working Logs and Not Working Logs.

References:

• • Troubleshooting Process for Printing Issues with Auto-creation failure and Citrix Print Manager Crashing

• Scanner information with details about scanner drivers for working and nonworking scenario
• Scanner is TWAIN compliant or not 

VDA (e)

• • CDF Trace – All modules, multiple sequential files. Start trace before starting session launch
  • Scout Diagnostic Logs (e)
  • Event Viewer Logs – Export as CSV and as EVTX (2 sets)
    • Application
    • System

       Capture Dump Files during crash

CWA (e)

• • • CDF Trace – All modules, multiple sequential files. Start trace before starting session launch

Video logs 

• • • Install Twacker and check if the twain dlls ( twain_32.dll / TWAINDSM.dll) are getting loaded using Process Explorer 
    • If a crash is happening any module, capture the steps that led to the crash

Collect Working Logs and Not Working Logs.

Info and setup of scanner articles:

• USB Scanner Is Not Being Automatically Redirected Within ICA Sessions
• Scanning

• Time of error
• Port number
• Session ID
• Working Logs
• Not working logs

Visual Logs:

• • Video of the issue.
  • If video not possible, screenshots of errors with time visible

VDA:

• • CDF Trace – All modules, multiple sequential files. Start trace before starting session launch (e)
  • Scout Diagnostic Logs (e)

CWA:

• • CDF Trace – All modules, multiple sequential files. Start trace before starting session launch (e)

Collect Working Logs and Not Working Logs.

• COM Port Redirection: How to map a client COM port to a server COM port
• COM Port Communication Issues when 20 or More Users are Concurrently Using COM Ports in XenApp 6.5
• COM Port Redirection Failure, Error: "Signature Pad Functionality Unavailable"

• Time of error
• Application Name/Pid (copy from/to)
• Details of troubleshooting performed

Visual Logs:

• • Video of the issue.
  • If video not possible, screenshots of errors with time visible

VDA:

• • Scout Diagnostic Logs (e)
  • Policy: Clipboard polices (e)
  • CDF Trace – All modules, multiple sequential files. Start trace before starting session launch (e)
  • Registry: Export
    • HKEY_LOCAL_MACHINE\SOFTWARE\Policies\Citrix\<SessionID>\User\VCPolicies

CWA:

• • CDF Trace – All modules, multiple sequential files. Start trace before starting session launch (e)

Collect Working Logs and Not Working Logs.

References:

• ICA policy settings
• CTX238130 - Clipboard redirection for files from client to VDA and VDA to client is not working
• CTX217933 - Citrix XenApp and XenDesktop : How to configure Uni-directional Clipboard Redirection
• Best Practices For Citrix Virtual Apps & Desktops Clipboard Redirection

• Time of error
• Application Name/Pid
• Does the issue occur locally or in RDP
• Details of troubleshooting performed so far

Visual Logs:

• • Video of the issue with clock showing milliseconds
  • If video not possible, screenshots of errors with time visible

  • On-screen instrumentation can be enabled by setting on the VDA:

    [REG_DWORD] HKLM\Software\Citrix\Graphics\EnableOSD = 1

    (disconnect/reconnect will make it take effect, or you can kill ctxgfx.exe)

VDA:

• • Scout Diagnostic Logs (e)
  • Policy: Graphics Polices (e)
  • CDF Trace – All modules, multiple sequential files. Start trace before starting session launch (e)
  • Event Viewer Logs – Export as CSV and as EVTX (2 sets)
    • Application
    • System
  • Registry: Export
    • Computer\HKEY_LOCAL_MACHINE\SOFTWARE\Citrix\Graphics
    • Computer\HKEY_LOCAL_MACHINE\SOFTWARE\Policies

Dumps: 

• • Mini dumps created by CtxExceptionHandler which is built-in crash detection mechanism located in %ProgramData%\Citrix\CDF\Reports and text file name ReportsCount.txt under %ProgramData%\Citrix\CDF. Please also check the time stamps on these dump files to make sure they were generated in the time frame close enough to the failure reported.

• • The above dumps are not complete dumps, so they are not sufficient to provide the detailed information on the crash.

    It is recommended to disable the handler so procdump will catch the exceptions instead. To do this, please set "Enabled"=dword:00000000 for both of the keys:

    [HKEY_LOCAL_MACHINE\SOFTWARE\Citrix\CtxExceptionHandler]

    [HKEY_LOCAL_MACHINE\SOFTWARE\Wow6423Node\Citrix\CtxExceptionHandler]

    "Enabled"=dword:00000000

    Then please configure procdump to capture a full crash dump on exception.

    E.g. by registering it as default debugger: procdump -ma -i c:\dumps

CWA:

• • Polices: Graphics polices configured (e)
  • CDF Trace – All modules, multiple sequential files. Start trace before starting session launch (e)
  • Event Viewer Logs – Export as CSV and as EVTX (2 sets)
    • Application
    • System

• • Registry: Export 
    • Computer\HKEY_LOCAL_MACHINE\SOFTWARE\WOW6432Node\Citrix\ICA Client
    • Computer\HKEY_LOCAL_MACHINE\SOFTWARE\Citrix\ICA Client
    • Computer\HKEY_CURRENT_USER\SOFTWARE\Citrix\ICA Client
    • Computer\HKEY_LOCAL_MACHINE\SOFTWARE\Policies
    • Computer\HKEY_LOCAL_MACHINE\SOFTWARE\WOW6432Node\Policies

Collect Working Logs and Not Working Logs.

References:

• Frequently overlooked settings to improve graphics performance

• Time of error
• Application Name/Pid
• Window handle (use spy++/use WindowHistory in tooltip mode to find it)
• Make sure you use message history (as per bitness of process) + windows history as per bitness of system and taken from same session (not from console as CDF traces).

Visual Logs:

• • Video of the issue with clock showing milliseconds
  • If video not possible, screenshots of errors with time visible (e)

VDA:

• • Scout Diagnostic Logs (e)
  • CDF Trace – All modules, multiple sequential files. Start trace before starting session launch (e)
  • Registry: Export 
    • HKEY_LOCAL_MACHINE/System/CurrentControlSet/Control/Citrix/wfshell/TWI (e)
  • The Automatic Seamless Tool can be used to collect the following (e):
    • Message history (as per bitness of system and taken from same session)
    • Windows History (as per bitness of system and taken from same session)

CWA:

• • CDF Trace – All modules, multiple sequential files. Start trace before starting session launch (e)
  • The Automatic Seamless Tool can be used to collect the following (e):
    • Message history (as per bitness of system and taken from same session)
    • Windows History (as per bitness of system and taken from same session)

Collect Working Logs and Not Working Logs.

References:

• CTX134941 - Taskbar Focus Issues with Hosted Applications Using RES Workspace Manager
• CTX140095 - Troubleshooting Seamless Focus Issues
• CTX101644 - Seamless Configuration Settings
• CTX138775 - Basic Steps to Determine If an Issue Occurs Only in Seamless Mode
• CTX233121 - Applications Do Not Open in Seamless Mode
• CTX126869 - Seamless Applications Start Slowly and Launch in a Fixed Size Window
• CTX102750 - Seamless Applications Launch in a Fixed Window
• CTX229970 - Seamless Flags Configuration for Epic Hyperspace

Session Recording Issues

• Time of error
• User Name
• Session Id
• Details of troubleshooting performed see 
  • Troubleshoot | Session Recording (citrix.com)

Visual Logs:

• • Video of the issue.
  • If video not possible, screenshots of errors with time visible

VDA: 

• • Scout Diagnostic Logs (e)
  • CDF Trace – All modules, multiple sequential files. Start trace before starting session launch (e)
  • Event Viewer Logs – Export as CSV and as EVTX (2 sets)
    • Application (e)
    • System (e)

Session Recording Server:

• • CDF Trace – All modules, multiple sequential files. Start trace before starting session launch (e)
  • Event Viewer Logs – Export as CSV and as EVTX (2 sets)
    • Application (e)
    • System (e)

Session Recording Player:

• • CDF Trace – All modules, multiple sequential files. Start trace before starting session launch (e)

• • Event Viewer Logs – Export as CSV and as EVTX (2 sets)
    • Application (e)
    • System (e)

Collect Working Logs and Not Working Logs.

Registry export of HKEY_LOCAL_MACHINE\SOFTWARE\Citrix\SmartAuditor for an understanding of current configurations. (e)

Troubleshooting Article:

• Troubleshoot | Session Recording (citrix.com)

References:

• • https://docs.citrix.com/en-us/citrix-virtual-apps-desktops/director/troubleshoot-deployments/user-issues/shadow-users.html
    
    

• Time of error
• User Name
• Clear description of operation performed
• Details of troubleshooting performed see 
  •  Transaction ID for cloud issues along with HAR trace collected from the client side and the public IP.
     

Visual Logs:

• • Video of the issue.
  • If video not possible, screenshots of errors with time visible

Director:

• • Scout Diagnostic Logs (e)
  • CDF Trace – All modules, multiple sequential files. Start trace before starting session launch (e)
  • Har data, see Creating a HAR file from your browser to share with support  (To confirm REST API fired as expected and the API arguments and result are expected) (e)

VDA:

• • CDF Trace – All modules, multiple sequential files. Start trace before starting session launch (e)

DDC:

• • CDF Trace – All modules, multiple sequential files. Start trace before starting session launch (e)

Collect Working Logs and Not Working Logs.

References:

• Citrix Director - Failure Reasons Troubleshooting Guide
• https://docs.citrix.com/en-us/citrix-virtual-apps-desktops/whats-new/known-issues.html
• CTX139382 - Best Practices for Citrix Director
• CTX208206 - "Cannot retrieve the data. View Director server event logs for further information." On Citrix Director
• CTX218695 - Citrix Director - Cannot Retrieve Data - Data Source Unresponsive or Reported an Error
• Director Error "Cannot retrieve data. Data source unresponsive or reported an error".
• "Cannot retrieve the data. Data source unresponsive or reported an error." in Director Trends Panel
• Director: Error "Unexpected error. Check your network .... Further Information" When Retrieving Current Connection or Session Data on Filters Page

• Time of error
• Clear description of operation performed
• Details of troubleshooting performed see 
  • Compare, prioritize, and troubleshoot policies | Citrix Virtual Apps and Desktops 7 2311
  
  
  • How to Troubleshoot Missing or Inconsistent Citrix Policies in Citrix Director

Visual Logs:

• • Video of the issue.
  • If video not possible, screenshots of errors with time visible

Active Directory:

• • GUID of the policy from C:\Windows\SYSVOL\sysvol\customer.com\Policies (e)

VDA:

• • Scout Diagnostic Logs (e)
  •  Policy:
    • Output of GPResult.exe (e)
    • Date/Time of  “UserPolicy” and “ComputerPolicy” gpf files in “C:\ProgramData\CitrixCseCache” (e)
    • “Rsop.gpf” date, time and size from C:\ProgramData\Citrix\GroupPolicy\<session id> (e)

• • Event Viewer Logs – Export as CSV and as EVTX (2 sets)
    • • Application (e) 
      • System (e)
  • CDF Trace – All modules, multiple sequential files. Start trace before starting session launch (e)
  • Registry: Export (e)
    
    • “Computer\HKLM\Software\Wow6432node\Polices\Citrix” or “Computer\HKLM\Software\Polices\Citrix”
  • Procmon logs during user logon - Confirm Rsops.gpf and registries updated.

Collect Working Logs and Not Working Logs.

Troubleshooting Articles:

• • Compare, prioritize, and troubleshoot policies | Citrix Virtual Apps and Desktops 7 2311
  • How to Troubleshoot Missing or Inconsistent Citrix Policies in Citrix Director

• User name
• logon/logoff timestamp
• Details of troubleshooting performed Troubleshoot | Profile Management (citrix.com)
• Environment details - File server OS, is it hosted on cloud or NFS Shares.

Visual Logs:

• • Video of the issue.
  • If video not possible, screenshots of errors with time visible 

VDA:

• • Scout Diagnostic Logs (e)
  • UPMConfigCheck output Profile Management Configuration Checking Tool - UPMConfigCheck (citrix.com) (e)
  • Profile management logs, to enable see Check Profile Management log files | Profile Management (e)  (citrix.com) %SystemRoot%\system32\LogFiles\UserProfileManager
  • Polices: Following files from User store root folder 
    • UPMSettings.ini  (e)
    • UPMFRSettings.ini (e) 
  • Event Viewer Logs – Export as CSV and as EVTX (2 sets)
    • Application (e) 
    • System (e)
  • CDF Trace – All modules, multiple sequential files. Start trace before starting session launch (e)
  • Procmon log

Collect Working Logs and Not Working Logs.

Troubleshooting Articles:

• Troubleshoot | Profile Management (citrix.com)

• User name
• Authentication type,
• Timestamp  of issue (especially if not reproducible every time)
• URL of front-end medium (storefront/web interface)

Visual Logs:

• • Video of the issue.
  • If video not possible, screenshots of errors with time visible

CWA:

• Output of SSON Configuration Checker (e)
• Client Side traces collected from Receiver Diagnostic tool (e)

• Event Viewer Logs – Export as CSV and as EVTX (2 sets) (e)

  • • Application
    • System

VDA:

• Scout Diagnostic Logs (e)
• CDF Trace – All modules, multiple sequential files. Start trace before starting session launch (e)

Event Viewer Logs – Export as CSV and as EVTX (3 sets) (e)

• • Application
  • System
  • Security

Collect Working Logs and Not Working Logs.

Troubleshooting Articles:

• • Troubleshooting Citrix Pass-Through Authentication
  • How To Troubleshoot Pass-Through Authentication to Web Interface (citrix.com)
  • Domain pass-through authentication

• Time of error
• PID of spiking process
• TID of spiking thread (process explorer can help with it).
• Extra data may include WPR or Perfmon Logs.
• ProcDump Recommended. Avoid using Task manager to generate dump files.

Scout Diagnostic Logs (e) (If Spike occurs on VDA,DDC,FAS,SF,License Server)

CDF:

• •  Simultaneous all module CDF in multi-sequential mode (e)

Dump:

• • Several consequent user dumps of spiking process  (e)

Collect Working Logs and Not Working Logs.

Perfmon 

• Perfmon logs with 1 sec interval run for 5-10 mins when the CPU spike is seen 

It is recommended to capture 3 dumps of the process 5 seconds apart when the CPU exceeds a certian threshold. Below is the example of the procdump command that can be used to capture dumps

Example: Write up to 3 full dumps of a process named 'consume' when it exceeds 20% CPU usage for five seconds:

procdump  -ma -n 3 -s 5 -c 20 <ProcessName or PID>

• Download Procdump
• High CPU usage troubleshooting guidance

System Crash/BSOD/

Bug Check

• • Scout Diagnostic Logs (e)
  • Complete memory dump. (e) If not possible, Kernel summary dump may be fine too
  • CDF Trace – All modules, multiple sequential files (e)
  • Event Viewer Logs – Export as CSV and as EVTX (2 sets)
    • Application (e)
    • System (e)

Reference:

• Use Dumpchk.exe to check a memory dump file
• How to verify the integrity of a dump file

Troubleshoot if it’s system hung
Page file size should be bigger than the amount of physical memory for physical memory dumps or you can have a dedicateddumpfile.sys file created. For virtualized systems convert VM snapshots to .DMP. Validate the dump before sending

• • Scout Diagnostic Logs (e)
  • Complete memory dump (e)
  • CDF Trace – All modules, multiple sequential files (e)
  • A matching perfmon can be collected to review increase in CPU/Memory usage or for memory leak issues.
• Perfmon Configuration - Long:
• 1.     Open CMD in Admin mode
• 2.      Copy paste the following command into the command prompt window:
  • Logman.exe create counter PerfLog-Long -o "c:\perflogs\PerfLog-Long.blg" -f bincirc -v mmddhhmm -max 600 -c "\LogicalDisk(*)\*" "\Memory\*" "\.NET CLR Memory(*)\*" "\Cache\*" "\Network Interface(*)\*" "\Paging File(*)\*" "\PhysicalDisk(*)\*" "\Processor(*)\*" "\Processor Information(*)\*" "\Process(*)\*" "\Redirector\*" "\Server\*" "\System\*" "\Server Work Queues(*)\*" "\Terminal Services\*" -si 00:00:301     
  • Start the logs with:
    Logman.exe start PerfLog-Long
• Perfmon Configuration-Short:
• 1.      Open CMD in Admin mode
• 2.      Copy paste the following command into the command prompt window:

Logman.exe create counter PerfLog-Short -o "c:\perflogs\PerfLog-Short.blg" -f bincirc -v mmddhhmm -max 600 -c "\LogicalDisk(*)\*" "\Memory\*" "\.NET CLR Memory(*)\*" "\Cache\*" "\Network Interface(*)\*" "\Paging File(*)\*" "\PhysicalDisk(*)\*" "\Processor(*)\*" "\Processor Information(*)\*" "\Process(*)\*" "\Redirector\*" "\Server\*" "\System\*" "\Server Work Queues(*)\*" "\Terminal Services\*" -si 00:00:011      

• Start the logs with:
  Logman.exe start PerfLog-Short
•  
  • Event Viewer Logs – Export as CSV and as EVTX (2 sets)
    • Application
    • System

Reference:

• Use Dumpchk.exe to check a memory dump file
• How to verify the integrity of a dump file
• How to configure Dedicated dump file for complete memory dump

• Validate the dumps before sending.
• ProcDump Recommended. Avoid using Task manager to generate dump files.

Scout Diagnostic Logs (e) (If the leak occurs on VDA,DDC,FAS,SF,License Server)

CDF:

• •  Simultaneous all module CDF in multi-sequential mode (e)

Dump:

• • Several consequent user dumps of leaking process
    taken while memory consumption gradually grows taken with corresponding UMDH logs. (e)
    • https://learn.microsoft.com/en-us/windows-hardware/drivers/debugger/using-umdh-to-find-a-user-mode-memory-leak
  • Set stack trace database flag.

WPR Logs: https://learn.microsoft.com/en-us/windows-hardware/test/wpt/step-1--starting-and-stopping-a-recording-using-wpr

Collect Working Logs and Not Working Logs.

• Using UMDH to Find a User-Mode Memory Leak
• How to start/stop a Windows Performance Recorder trace

• Confirm Application compatibility issues are ruled out
• Validate the dumps before sending.
• ProcDump Recommended. Avoid using Task manager to generate dump files.

Scout Diagnostic Logs (e) (If the process hang occurs on VDA,DDC,FAS,SF,License Server)

CDF:

• •  Simultaneous all module CDF in multi-sequential mode (e)

Dump:

• • Manual dump of the hanged process. If hang caused by CPU spike in UI thread then get data as for CPU spike issue (e)

Event Viewer Logs – Export as CSV and as EVTX (2 sets) (e)

• • Application
  • System

Windows History/Message History:

• • If hang caused by invisible modal window then WH/MH logs would be required together with new dump

Collect Working Logs and Not Working Logs.

For Regular process hang dump, it is recommended to capture 3 dumps of the process 5 seconds apart in its hung state:

procdump  -ma -n 3 -s 5 <ProcessName or PID>

If the Process hang is due to CPU spike, it is recommended to capture 3 dumps of the process 5 seconds apart. Below is the example of the procdump command that can be used to capture dumps

Example: Write up to 3 full dumps of a process named 'consume' when it exceeds 20% CPU usage for five seconds:

procdump  -ma -n 3 -s 5 -c 20 <ProcessName or PID>

• Download Procdump

Application/

Process Crash

• Confirm Application compatibility issues are ruled out
• Refer Crash Analyser tool recommendations before opening a Dev-A case.
• ProcDump Recommended. Avoid using Task manager to generate dump files.

Scout Diagnostic Logs (e) (If the process crash occurs on VDA,DDC,FAS,SF,License Server)

CDF:

• • Simultaneous all module CDF in multi-sequential mode

Dump:

• • Using ProcDump tool (with –ma & -e flags) (e)
•                         OR 
  •  Using Windows Error Reporting (WER) - https://docs.microsoft.com/en-us/windows/desktop/wer/collecting-user-mode-dump

Event Viewer Logs – Export as CSV and as EVTX (2 sets) (e)

• • Application
  • System

Collect Working Logs and Not Working Logs.

• How to Use Windows Error Reporting (WER) to Capture Application Crash Dumps (User Dumps) On Windows Server 2008 And Windows Vista SP1 Onwards
• Download Procdump
• How to get dump file of random crash process with ProcDump tool

In case of process crash due to heap corruption, enable page heap on the crashing process  before capturing the dump:

• Enabling Page Heap Verification

Dump:

• • Complete memory dump (e)

Scout Diagnostic Logs (e) (If the memory leak occurs on VDA,DDC,FAS,SF,License Server)

CDF:

• • If caused by Citrix driver CDF traces might be required for short period of time during which memory consumption gradually grows

Event Viewer Logs – Export as CSV and as EVTX (2 sets) (e)

• • Application
  • System

Collect Working Logs and Not Working Logs.

Verify if the leak is observed , follow 

• https://docs.microsoft.com/en-us/windows-hardware/drivers/debugger/finding-a-memory-leak

First priority is to identify the process which is causing the leak.If the leak is not process specific , we need to identify which pool tag is consuming memory , follow 

• Using the Kernel Debugger to Find a Kernel-Mode Memory Leak to get the call stack when the allocation happens.
• Use Dumpchk.exe to check a memory dump file
• How to verify the integrity of a dump file

• See if session can be recovered by restarting receiver. If yes, that could be client side hang otherwise its server side.
• Verify if the entire VDA is frozen e.g. if server VDA login via a different account and check if it is responsive.
• Check console login, and verify if it is hard hang or soft hang.

Dump:

• • Complete memory dump if multiple sessions are hung

Scout Diagnostic Logs (e)

CDF: 

• • Full CDF Traces (e)

Event Viewer Logs – Export as CSV and as EVTX (2 sets) (e)

• • Application
  • System

Collect Working Logs and Not Working Logs.

Verify if the session gets hung in a specific scenario such as while printing/logoff/reconnect etc., based on the scenario verify if the complete machine is hung or it can be recovered by killing one of the processes . Share the process details after killing which we can recover from the hang.

Depending on the scenario , we need complete memory dump with session ID or if particular process is hung we need memory dump of the process for analysis. Refer to below links for capturing complete memory dump for different types of machines.

• Collect a complete memory dump on a MCS (non-persistent) machine on XenServer (Citrix Hypervisor)
• How to collect a full memory dump on VMware for Citrix VDAs
• How to trigger a memory dump from a Windows virtual machine
• How To Get A Memory Dump From A PVS Target Device
• How to collect complete memory dump of a virtual machine on Nutanix AHV
• How to capture a full memory dump from a PVS target running on Azure

• Time of error
• App Name
• Session ID
• User Name
• Working Logs
• Not working logs

Exact time stamp of the issue including seconds details, App Name and User name etc.

Chrome client/console logs (e)

https://support.citrix.com/s/article/CTX217352-how-to-collect-logs-in-receiver-for-chrome-and-receiver-for-html5?language=en_US

CWA for Chrome OS (e)

https://docs.citrix.com/en-us/citrix-workspace-app-for-chrome/troubleshoot.html

Collect Working Logs and Not Working Logs.

• How to Collect Logs in Receiver for Chrome and Receiver for HTML5
• Troubleshoot - CWA for ChromeOS

1. iOS version installed on the device.
2. CWA iOS version.
3. Exact error message presented to the user.
4. Does the issue only occur on iOS devices, or does it also occur with other clients (Android, Chrome, Mac, Windows, Linux)?
5. If the issue is also occurring on other clients, then this is most likely not a CWA issue and not a CWA iOS issue specifically.
6. Did it ever work?
7. If yes, then when did the issue start?
8. What changed prior to the issue being seen for the first time (OS update, CWA update, environmental changes/updates, etc)?
9. Does the issue occur when using the CWA app, connecting through a browser, or both?
10. Does the issue occur when connecting through a Gateway/ADC?
11. If yes, does it still occur when connecting directly to StoreFront?
12. Is the desired feature or capability supported on CWA iOS? (check the Feature Matrix - https://docs.citrix.com/en-us/citrix-workspace-app/citrix-workspace-app-feature-matrix.html)

iOS logs, AE Tracing (e)
Crash dump (if crashes)

Citrix Workspace App for iOS: How to Enable and Collect Advanced Logs
https://support.citrix.com/external/article?articleUrl=CTX201884-citrix-receiver-for-ios-common-settings-reference&language=en_US

https://docs.citrix.com/en-us/citrix-workspace-app-for-ios/troubleshooting.html#how-to-collect-logs 

Collect Working Logs and Not Working Logs.

• CWA for iOS: Cannot verify this server's certificate
• CWA for iOS: Cannot verify this server's certificate

• Exact time stamp of the issue including seconds details,
• App Name
• User name

Android Logs (e)

How to collect Receiver for Android Logs:
https://support.citrix.com/article/CTX216900

https://docs.citrix.com/en-us/citrix-workspace-app-for-android/configure.html#how-to-collect-logs

Collect Working Logs and Not Working Logs.

• Troubleshoot - CWA for Android
• CWA for Android: Fail to Launch Apps and Desktops

• Exact time stamp of the issue including seconds details,
• App Name
• User name

Linux Logs

How to collect Logs for Citrix Workspace app for Linux? (e)
https://support.citrix.com/article/CTX217629

https://docs.citrix.com/en-us/citrix-workspace-app-for-linux/configure-xenapp.html#logging

For authentication logs, please enable verbose logging. 

CWA logs and Authman Logs, also grab a copy of the AuthManConfig.xml configuration.
1- Enable CWA logs to be in verbose.
sudo /opt/Citrix/ICAClient/util/setlog level all verbose

2- Enable trace and increase log level for Authman.
-- Edit the Authman config file: /opt/citrix/ICAClient/config/AuthManConfig.xml
-- Changed the lines for "TracingEnabled" to True and "LoggingMode" to verbose

Should look like this:
<key>TracingEnabled</key>
<value>True</value>
<key>LoggingMode</key>
<value>verbose</value>

3- Reproduce the issue

4- Collect the directories and files:
-- ~/.ICAClient/logs/
-- /var/log/citrix/ICAClient.log
--  /opt/citrix/ICAClient/config/AuthManConfig.xml

A quick way of doing this is by compressing thes files in to a tar. You can use the below command.
sudo tar -czvf CWAL-Logs_$HOSTNAME_$(date +%F-%H-%M-%S).tar.gz $HOME/.ICAClient/logs/ /var/log/citrix/ICAClient.log /opt/citrix/ICAClient/config/AuthManConfig.xml

Collect Working Logs and Not Working Logs.

Troubleshooting Articles:

• • Troubleshooting CWAL
  • Troubleshooting HDX MediaStream on Linux
  • Troubleshooting Flash V2 on Linux
  • Troubleshooting Windows Media Redirection on Linux

• Time of error
• User Name
• Call ID
• User Name

Installer logging, Session Logging, Store Logging, Always-on logging and CWA Updater Logging (e)

Collect Working Logs and Not Working Logs.

• How to Enable and Collect Logging for Citrix Workspace App (CWA) for Mac 

• License Server Version
• Username
• Client Hostname

Visual Logs: (e)

• Screen shots of all relevant product and licensing configurations
• Screen shots of license in-use from Citrix License Administration Console or Citrix Licensing 

• Scout Diagnostic Logs (e)

License Server Logs (e)

• Copy of C:\Program Files\Licensing\ folder on an x86 server or C:\Program Files (x86)\Citrix\Licensing\

• Event Viewer Logs – Export as CSV and as EVTX (2 sets)

  • • Application
    • System
• CDF trace

Troubleshooting Articles:

• How to Collect Data for Troubleshooting Licensing Issues
• Error: "Previous Upload to cis.citrix.com Failed. Network Error While Connecting cis.citrix.com" with Code 5

• Time of error
• App Name/Desktop name
• User Name
• Working Logs
• Not working logs

Visual Logs:

• Video of the repro
• Screenshot of the issue

Collect logs using the xdlcollect command and set it to verbose using setlog utility. (e)

• bash /opt/Citrix/VDA/bin/xdlcollect.sh

Troubleshooting Articles:

• • https://docs.citrix.com/en-us/linux-virtual-delivery-agent/current-release/configure/administration/log-collection.html
  • LVDA Troubleshooting
  • Troubleshoot Common Graphics Issues on Linux VDA
  • How to troubleshoot Session Reliability in Linux VDA
  • How to Troubleshoot Adaptive Transport Issues on Linux VDA
  • How to troubleshot Citrix FAS related issues on Linux VDAs
  • How to troubleshoot SSL Encryption for Linux VDA
  • Troubleshooting HDX Session Validation Failure on the Linux VDA

• Time of error
• App Name/Desktop name
• User Name
• Working Logs
• Not working logs
• Any proxy tools installed on macOS

Visual Logs: 

• Video of the repro
• Screenshot of the issue
• Reference link - https://support.apple.com/en-ie/guide/mac-help/mh26782/mac

VDA log collection (e)

• quick guide
  • Open Terminal App, and then run the command: sudo xdlcollect
  • share the zip file
• details: https://docs.citrix.com/en-us/mac-vda/configure/administration/tools-and-utilities#the-xdlcollect-shell-script
• log level configuration: https://docs.citrix.com/en-us/mac-vda/configure/administration/log-collection

References:

macOS overview for IT: 

• https://en.wikipedia.org/wiki/MacOS

macOS critical settings related to Mac VDA (toggles etc.):

• https://support.apple.com/en-hk/guide/mac-help/mchl211c911f/15.0/mac/15.0

Mac VDA product document & guide:

• https://docs.citrix.com/en-us/mac-vda
  • https://docs.citrix.com/en-us/mac-vda/overview
  • https://docs.citrix.com/en-us/mac-vda/installation-overview
  • https://docs.citrix.com/en-us/mac-vda/configure/administration
  • https://docs.citrix.com/en-us/mac-vda/configure/policy-support-list
  • https://docs.citrix.com/en-us/mac-vda/known-issues
• https://community.citrix.com/tech-zone/build/deployment-guides/citrix-mac-vda/ 

Customer ID (e)

Cloud connector name and IP (e)
Cloud connector resource location. (e)
Cloud connector resource location Platform details (e) (Azure\AWS\VMware\Nutanix)

Date/time of the issue (e)
List of steps that has already been taken prior to contacting Support, and dates/times of each thing (if possible) (e)
Is there any antivirus in place on the machine (e)

Connector Installation Logs (e)
Recent installation logs are located here: %LOCALAPPDATA%\Temp\CitrixLogs\CloudServicesSetup.

Additionally, historical install logs are added to:
 %ProgramData%\Citrix\WorkspaceCloud\InstallLogs
 
Connector logs (e)
By default, event logs are located in this directory of the machine hosting the Cloud Connector:
C:\ProgramData\Citrix\WorkspaceCloud\Logs

Application, Setup and System event logs (e)

• How to Troubleshoot Installation Failures on Citrix Cloud Connector for Windows
• Unable to Install Citrix Cloud Connector, Failing with Internet Connectivity Error
• Citrix Cloud Connector Installation does not complete: Unable to validate certificate chain
• Cloud Connector installation error - Make sure the web address http://localhost:8081 is correct 
• Unable to install the Citrix Cloud Connector
• Cloud Connector Connectivity Check Failed
• Cloud Connector Connectivity Check Utility
• Cloud connector cleanup script can be found here:
  https://citrite.sharefile.com/share/view/s3173bbe4d4848a7b
• Cloud Connector Installation
• How Do I Resolve a Failed Cloud Connector Maintenance

Customer ID (e)

Cloud connector name and IP (e)
Cloud connector resource location. (e)
Cloud connector resource location Platform details (e) (Azure\AWS\VMware\Nutanix)

Date/time of the issue (e)
List of steps that has already been taken prior to contacting Support, and dates/times of each thing (if possible) (e)
Is there any antivirus in place on the machine (e)

https://docs.citrix.com/en-us/citrix-cloud/citrix-cloud-resource-locations/citrix-cloud-connector/log-collection.html
Event logs located in the (e) C:\ProgramData\Citrix\WorkspaceCloud\Logs directory of the machine hosting the Cloud Connector.

Connector Appliance diagnostic report. (e)

Network trace while running cloud connector connectivity tool (e)

https://docs.citrix.com/en-us/citrix-cloud/citrix-cloud-resource-locations/connector-appliance.html#generating-a-diagnostic-report

Proxy configuration details, if any (e)

Results from the Advanced Connectivity Check Tool (built in to the Connector installer, you can get to it by re-running cwcconnector.exe on an installed Connector and choosing the advanced check button) (e)

• Citrix Cloud Connector Technical Details
• Cloud Connector Proxy and Firewall Configuration
• Log Collection for Citrix Cloud Connector
• Cloud Connector Connectivity Check Utility
• Citrix Cloud Connector Troubleshooting Guide
• How Do I Restore an Offline Cloud Connector to an Online State
• How to Enable Debug Logging for Proxy Events on Citrix Cloud Connector
• How to Enable SSL on Cloud Connectors to Secure XML Traffic

Customer ID (e)

Client name and IP (e)
VDA name and IP (e)
Gateway details
Storefront details
Connector details

Information to be Gathered:

Questions that help identify problem 
• Customer specific or control plane issue? 
— Helps determine if this is customer related issue or something that affecting various customers from control plan.

• How many users impacted? User names? 
— Needed for starting end to end trace information on user launches. From this information we can get user SID information and session keys -> transaction IDs 

• What resources are being launched (Published Name or Delivery Group )?
— Needed for enumeration issues and tracking ticketing/feed information from Workspace launches

• Specific to Resource Location? External vs Internal? 
- Confirm if NS \ ADC related issues is affecting the users. please do isolation.

On-Prem Storefront vs Citrix Workspace?

Rendezvous launch or not?

If we can get into session/ issue is intermittent ctxsession -v output

Output of below cmd run on client and VDA to understand routing:

nslookup global.g.nssvc.net 

Enumeration:

Client\ Citrix Workspace App CDF trace. (e)
DDC CDF trace (e)
Storefront verbose trace or Splunk logs for workspace service. (e)

Brokering Stage Issues:

Client\ Citrix Workspace App CDF trace. (e)
DDC CDF trace (e)
Storefront verbose trace or Splunk logs for workspace service. (e)

Connector CDF logs if non-rendezvous launch (e)

HDX Stage issue:

Client:

CDF logs (e)

https://docs.citrix.com/en-us/citrix-workspace-app/app-protection/troubleshoot/log-collection.html

Network Traces - netsh trace start capture=yes tracefile=<path>\client.etl maxsize=2048 filemode=circular

VDA

CDF logs (e)

Network trace

• • Wireshark Trace or
  • Netsh trace - https://learn.microsoft.com/en-us/windows/win32/ndf/using-netsh-to-manage-traces

 netsh trace start capture=yes tracefile=<path>\client.etl maxsize=2048 filemode=circular

Connector CDF logs if non-rendezvous launch (e)

• Session Launch Failure: “Your session 'AppName' did not launch successfully due to error code 3500
• Failed to launch session with the error code “Your session <AppName> did not launch successfully due to error code 3501. Please contact your administrator for more information about the error.” on Citrix Workspace App During App/Desktop Launch
• Failing to launch published apps, error “Your session did not launch successfully due to error code 2524.”
• Citrix Workspace App Log Collection

HAR logs from Developer tools (e)

• Citrix Monitor Service OData API Update FAQ
• How to access Citrix Monitor Service data using the OData v4 endpoint in Citrix Cloud
• Monitor - DaaS for Azure
• Citrix Director - Failure Reasons Troubleshooting Guide
• Creating a HAR file from your browser to share with support

Customer ID (e)

machine catalog name

load index values

VDA names which were powered on/off

Transaction ID

Screenshot of the error

HAR log

DDC trace[For citrix engineers only] (e)

• https://docs.citrix.com/en-us/citrix-daas/manage-deployment/autoscale

Customer ID (e)

Caller IP(public IP of endpoint) (e) 

• You can check public IP of the endpoint by accessing websites like whatismyip.com

SAML IDP app configuration e.g. Okta, Azure AD (e)

SAML Config settings in Citrix Cloud (e)

• UI  > SAML > View settings.

Details of the FrontEnd User from IDP Directory (e)
Details of the BackEnd User from Active Directory (e)
Details of UPN suffixes from Active Directory
Is Conditional Auth involved, if yes, details of it.
Desired Logout Behaviour (if any)

Transaction ID (e)

Screenshot of the error (e)

HAR log (e)

SAML Tracer log (e)

• Creating a HAR file from your browser to share with support
• SAML Troubleshooting
• Intermittent Enumeration of Resources [Azure AD]