If you suspect that your NetScaler ADC or NetScaler Gateway has been compromised, follow these steps to secure your environment and enable effective investigation:

      1. Preserve Evidence 

• Snapshot of Potentially Compromised NetScaler ADC VPX Instance

If the NetScaler is a virtual instance (VPX), take a snapshot of the potentially compromised instance for forensic analysis and further investigation.

Document the system time, timezone settings, and NTP configuration before isolation. 

Preserve logs from remote syslog servers and NetScaler Console in addition to local NetScaler logs.

• Generate Technical Support Bundle
  
  

While Cloud Software Group does not support forensic investigations, collecting a technical support bundle will capture a list of current configuration, running processes, and other data that may help in your analysis later. 

https://support.citrix.com/external/article/472859/how-to-generate-a-support-file-for-adc.html

• Generate a core file for the Packet Engine
  
  

Generate a core file (memory dump) for the Packet Engine by following the steps documented at:  

https://support.citrix.com/external/article?articleUrl=CTX207598-how-to-generate-nsppe-core-dump-on-netscaler&language=en_US

  Note: The system will perform a warm restart during this process and SSH connections will disconnect. You can log in again once the machine boots up.  Copy the core dump files from '/var/core/<N>', where N is the highest numbered folder. The dump files will start with `NSPPE-` and should automatically get compressed with gzip after a short time. The above steps must be performed on the suspected compromised NetScaler. The system will undergo a warm restart as part of this process.

• Forensic Disk Imaging for MPX/SDX Appliances

For hardware appliances, please work with your incident response team to follow local processes. These processes are likely to include steps such as: power down the appliance after memory preservation, remove the physical disks, create bit-for-bit disk images ideally using a hardware write-blocker, retain two copies: one for analysis, one for evidence preservation, and document the chain of custody. 

2. Isolate the Device

Remove the NetScaler ADC/Gateway from the network to prevent further unauthorized access.

3. Revoke Credentials and Access

Ensure:

• All service account passwords and secrets stored on the NetScaler are changed on their respective systems (e.g., LDAP service accounts, RADIUS shared secrets, OAuth tokens, API Keys, SNMP community names).
• All user accounts that may have been authenticated via the suspected compromised platform, including Gateway or AAA virtual servers are changed on their respective systems.
• Certificates and associated private keys stored on the suspected compromised platform are revoked.

4. Investigate Connected Systems 

Investigate all servers and systems that the NetScaler ADC had connected to for any signs of further compromise, particularly authentication servers, sensitive systems, web tier systems, and management jump hosts.

5. Rebuild and Restore

MPX appliances - Refer to the official guide to erase and reinstall the MPX: https://docs.netscaler.com/en-us/netscaler-hardware-platforms/mpx/wiping-your-data-before-sending-your-adc-appliance-to-netscaler .

Alternatively, if your incident response process requires it, please contact your Account Technology Strategist (ATS)

SDX appliances - SDX appliances consist of a XenServer hypervisor, Service Virtual Machine (SVM) management system, and VPX instances. For compromised VPX instances within SDX, follow the VPX remediation steps.

VPX instances -  Cloud Software Group recommends replacing and restoring the instance. For detailed instructions on deploying a new instance on your specific hypervisor, refer to: https://docs.netscaler.com/en-us/vpx/current-release.html

Firmware Upgrade

After wiping or rebuilding, upgrade the NetScaler ADC to the latest available version of firmware before restoring the configuration backup.

Restore Configuration

Restore a known good NetScaler backup using NetScaler Console or your preferred backup platform and ensure the configuration is as expected.  If restoring from backups, verify they pre-date the compromise. For detailed instructions restoring a backup using NetScaler Console, refer to: https://docs.netscaler.com/en-us/netscaler-application-delivery-management-software/current-release/networks/instance-management/backup-restore-netscaler-instances.html

          Important: If law enforcement involvement is anticipated or legally required, consult with legal counsel before proceeding with system rebuild. Evidence preservation requirements may supersede operational recovery needs.

6. Rotate Restored Secrets

After restoring a known good NetScaler backup:   

• Change all local account passwords on the NetScaler ADC.
• Rotate Key Encryption Keys (KEK) to ensure cryptographic security.
• Remove and replace all restored SSL certificates (the older SSL certificates having been revoked at step 3)

7. Harden the Device  

Follow the NetScaler ADC Security Deployment Guide - https://docs.netscaler.com/en-us/citrix-adc-secure-deployment.html to harden your NetScaler ADC. Monitor the rebuilt system closely for suspicious activity for at least 90 days after recovery.

Important: The NetScaler Management Services should never be exposed to the public internet. 

These steps are intended to help contain and remediate a potential compromise. Always consult with your security team and legal team and follow your organization's incident response procedures.

Author: Steven Wright

This document and the information contained in it is provided as-is. Cloud Software Group. makes no warranties or representations, whether express or implied, regarding the document or its contents, including, without limitation, that this document or the information contained in it, is error-free or meets any conditions of merchantability or fitness for a particular purpose.

This article explains about the necessary steps to be taken when  NetScaler ADC or Gateway is suspected to be compromised.