If you suspect that your NetScaler ADC or NetScaler Gateway has been compromised, follow these steps to secure your environment and enable effective investigation:
1. Preserve Evidence
If the NetScaler is a virtual instance (VPX), take a snapshot of the potentially compromised instance for forensic analysis and further investigation.
Document the system time, timezone settings, and NTP configuration before isolation.
Preserve logs from remote syslog servers and NetScaler Console in addition to local NetScaler logs.
While Cloud Software Group does not support forensic investigations, collecting a technical support bundle will capture a list of current configuration, running processes, and other data that may help in your analysis later.
https://support.citrix.com/external/article/472859/how-to-generate-a-support-file-for-adc.html
Generate a core file (memory dump) for the Packet Engine by following the steps documented at:
Note: The system will perform a warm restart during this process and SSH connections will disconnect. You can log in again once the machine boots up. Copy the core dump files from '/var/core/<N>', where N is the highest numbered folder. The dump files will start with `NSPPE-` and should automatically get compressed with gzip after a short time. The above steps must be performed on the suspected compromised NetScaler. The system will undergo a warm restart as part of this process.
For hardware appliances, please work with your incident response team to follow local processes. These processes are likely to include steps such as: power down the appliance after memory preservation, remove the physical disks, create bit-for-bit disk images ideally using a hardware write-blocker, retain two copies: one for analysis, one for evidence preservation, and document the chain of custody.
Remove the NetScaler ADC/Gateway from the network to prevent further unauthorized access.
Ensure:
Investigate all servers and systems that the NetScaler ADC had connected to for any signs of further compromise, particularly authentication servers, sensitive systems, web tier systems, and management jump hosts.
MPX appliances - Refer to the official guide to erase and reinstall the MPX: https://docs.netscaler.com/en-us/netscaler-hardware-platforms/mpx/wiping-your-data-before-sending-your-adc-appliance-to-netscaler .
Alternatively, if your incident response process requires it, please contact your Account Technology Strategist (ATS)
SDX appliances - SDX appliances consist of a XenServer hypervisor, Service Virtual Machine (SVM) management system, and VPX instances. For compromised VPX instances within SDX, follow the VPX remediation steps.
VPX instances - Cloud Software Group recommends replacing and restoring the instance. For detailed instructions on deploying a new instance on your specific hypervisor, refer to: https://docs.netscaler.com/en-us/vpx/current-release.html
Firmware Upgrade
After wiping or rebuilding, upgrade the NetScaler ADC to the latest available version of firmware before restoring the configuration backup.
Restore Configuration
Restore a known good NetScaler backup using NetScaler Console or your preferred backup platform and ensure the configuration is as expected. If restoring from backups, verify they pre-date the compromise. For detailed instructions restoring a backup using NetScaler Console, refer to: https://docs.netscaler.com/en-us/netscaler-application-delivery-management-software/current-release/networks/instance-management/backup-restore-netscaler-instances.html
Important: If law enforcement involvement is anticipated or legally required, consult with legal counsel before proceeding with system rebuild. Evidence preservation requirements may supersede operational recovery needs.
After restoring a known good NetScaler backup:
Follow the NetScaler ADC Security Deployment Guide - https://docs.netscaler.com/en-us/citrix-adc-secure-deployment.html to harden your NetScaler ADC. Monitor the rebuilt system closely for suspicious activity for at least 90 days after recovery.
Important: The NetScaler Management Services should never be exposed to the public internet.
These steps are intended to help contain and remediate a potential compromise. Always consult with your security team and legal team and follow your organization's incident response procedures.
Author: Steven Wright
This document and the information contained in it is provided as-is. Cloud Software Group. makes no warranties or representations, whether express or implied, regarding the document or its contents, including, without limitation, that this document or the information contained in it, is error-free or meets any conditions of merchantability or fitness for a particular purpose.
This article explains about the necessary steps to be taken when NetScaler ADC or Gateway is suspected to be compromised.