To provide a comprehensive guide on how to properly collect data for issues with ADC when these issues are related to Authentication. Note that additional data may be needed for SAML, OAuth, client cert, device cert, and EPA.

Instructions

1. If this is SAML or OAuth, or support requests Fiddler, perform these 4 bullet points, otherwise skip to #2
   • Install Fiddler on the Test Device from https://www.telerik.com/download/fiddler. Click the Download for Windows button after filling in the fields. 
   • Open Fiddler. Under Tools > Fiddler Options, click HTTPS tab, Check the Decrypt HTTPS traffic box. Install the Fiddler cert. If this is not completed the Fiddler capture will not be useful.
   • Start the Fiddler capture if not already running (File->Capture Traffic should be Checked)
   • Test that Fiddler is working by going to Google.com. You should see more in the HOST column than "Tunnel To" for the Google requests. A couple will say Tunnel To, but multiple should have google.com in the host column. If not, double-check the HTTPS settings from above.
2. If Support requests browser HAR or Console logs file, preform these bullets:
   • Open Developer Tools in Browser (Press F12)
   • Find the option to "Preserve Log" in the Network Tab of the Dev Tools (In the Gear icon under the Network Tab) and enable it.
3. If this Auth issue is using any of: a) Gateway VPN Plugin, b) EPA, c) Citrix SSO app is used, d) CEM/SecureHub is used, or support requests Plugin logs (do a and b), perform the relevant item(s), otherwise skip to step #4.
   • a. Gateway VPN Plugin: On the Test Device, Enable VPN Plugin Debug logging. Open the VPN plugin on the PC (see https://support.citrix.com/article/CTX138155 if the icon is hidden) then navigate to 3-bar menu->Configuration->Check Enable debug logging.
   • b. EPA: Enable ADC for EPA logs with CLI command "set vpn param -clientSecurityLog ON" and Collect EPA logs: https://support.citrix.com/article/CTX209148/understanding-and-configuring-epa-verbose-logging-on-citrix-gateway
   • c. Citrix SSO: Follow the steps here to enable debug logging: https://help-docs.citrix.com/en-us/citrix-sso/citrix-sso-for-android/use-sso-app-from-your-android-device.html#enable-debug-logs
   • d. SecureHub: Follow steps here to send Logs: https://support.citrix.com/article/CTX228190/how-to-generate-the-securehub-device-side-logs  
4. If this Auth issue occurs only with Workspace, then also collect debug logs for Workspace: 
   • Windows: https://support.citrix.com/article/CTX286820/how-to-collect-logs-for-citrix-workspace-app-for-windows
   • MAC: https://support.citrix.com/article/CTX229846/how-to-enable-and-collect-logging-for-citrix-workspace-app-cwa-for-mac
   • iOS: https://support.citrix.com/article/CTX136380/citrix-workspace-app-for-ios-how-to-enable-and-collect-advanced-logs
   • Android: https://support.citrix.com/article/CTX216900/how-to-collect-receiver-for-android-logs
   • Linux: https://support.citrix.com/article/CTX217629/how-to-collect-logs-for-citrix-workspace-app-for-linux
5. On the ADC, Ensure Debug logging is enabled at System->Auditing->Change Auditing Syslog Settings->Set Log Levels to ALL->Click Ok.
6. On the ADC, Ensure Debug logging for Auth is enabled at Gateway->Global Settings->Under Authentication Settings click Change authentication AAA Settings->For BOTH "AAA Session Log Levels" and "AAAD Log Level", set both to DEBUG -> Click Ok.
7. On the ADC, Ensure Session Reuse is Disabled on the relevant Vserver (Gateway, CS, LB, etc) at Vservers -> Edit Vserver -> SSL Parameters -> Uncheck “Enable Session Reuse”.
   **This should not cause service interruptions and ensures traces will decrypt**
8. On the ADC, if an Authentication Vserver is in use, Ensure Session Reuse is Disabled on the relevant AAA Vserver at Security -> AAA Application Traffic-> Vservers -> Edit Vserver -> SSL Parameters -> Uncheck “Enable Session Reuse”.
   **This should not cause service interruptions and ensures traces will decrypt**
9. On the ADC, in a console window, enter SHELL mode and run this command: cat /tmp/aaad.debug
10. If troubleshooting Kerberos: On the ADC, in another console window, enter SHELL mode and run this command: cat /tmp/nskrb.debug
11. On the ADC, Go to System->Diagnostics->Start a trace, start the trace using the settings in the image; be sure to change all highlighted items.
    **NOTE: Preference is that the trace is unfiltered, however if you absolutely must filter the capture, ensure you capture all these items as well as any authentication ports in use which are using non-standard port numbers: CONNECTION.IP.EQ(ClientIP) || CONNECTION.IP.EQ(VIP_VserverIPs) || CONNECTION.IP.EQ(AAA_VserverIP) || CONNECTION.IP.EQ(Backend_Auth_Server_IPs) || (CONNECTION.PORT.EQ(514) || CONNECTION.PORT.EQ(8766) || CONNECTION.PORT.EQ(88) || CONNECTION.PORT.EQ(636) || CONNECTION.PORT.EQ(389) || CONNECTION.PORT.EQ(1812) || CONNECTION.PORT.EQ(49)
    
    **SSL Master Keys are not your private keys; they are session keys while will only decrypt this trace. This does Not work with FIPS ADCs as the SSL Session Keys cannot be captured from the FIPS HSM.**
    
12. Ensure the trace is running and you acknowledge the decryption message prior to proceeding.
    
13. On the Test Device, Open the Gateway Plugin (or Browser Login page) and attempt to login and replicate the issue you are experiencing.
14. Take note of the exact time when the issue is replicated and provide it to your support representative via Email.
15. On the Test Device, capture the Internal IP (ipconfig) and External IP (whatsmyip.org) and provide them to your support representative via Email.
16. On the ADC, stop the trace and Download the .cap files and the .sslkeys session keys files.
17. On the ADC, Generate a support file at Diagnostics->Generate support file, Select NODE. This will take some time to complete; be sure to download the generated file and provide it at step 22 below.
    **The support file can also be manually collected using CLI command "show techsupport". Then download the file using WinSCP and locate the file at /var/tmp/support/collector_P_IPADDR_DateAndTimeCaptured.tar.gz**
18. On the ADC Shell log for the aaad.debug command, copy all to clipboard and save to Notepad as aaadebug.txt
19. On the Test Device, if Fiddler logging was performed, stop the Fiddler capture. Save the Fiddle trace at File->Save->All Sessions and save the file. The saved file should be in .SAZ format. If not, ensure the correct save method was used.
20. On the Test Device, if HAR logging was preformed, complete the following:
    • Right-Click on the Browser Network logs, click "Save all as HAR with content" and save the HAR file. 
    • Click the Brower Console logs tab, Right click the logs, and click Save as and save the logs.
21. On the Test Device, if client debug logs were enabled, perform the correct bullet below:
    • Gateway Plugin: Open the Gateway Plugin navigate to 3-bar menu->Logging->Collect Log files. Allow this to complete. The file "vpnlogs DateOfCollection.zip" will be created on your desktop.
    • Citrix SSO: Email the logs to your support representative as detailed here: https://help-docs.citrix.com/en-us/citrix-sso/citrix-sso-for-android/use-sso-app-from-your-android-device.html#enable-debug-logs
22. Upload all files listed below to https://cis.citrix.com - If you cannot access the link from within your company, utilize a different internet connection or PC to upload the files.
    ADC aaad.debug log file and if collected nskrb.debug log file
    ADC Trace
    ADC SSLKEYS
    ADC Support File downloaded
    Client logs if collected
    EPA logs if collected
    Workspace logs if collectd
    Fiddler trace and/or HAR Network and Console logs if collected
23. Do not forget to email the information from steps 14 and 15.