Error: "System maintenance in progress. Please try after some time" on NetScaler SDX SVM

Error: "System maintenance in progress. Please try after some time" on NetScaler SDX SVM

book

Article ID: CTX208368

calendar_today

Updated On:

Description

The following error is displayed on NetScaler SDX SVM:
System maintenance in progress. Please try after some time

Resolution

It is a known issue. You need to upgrade to 10.5 59.x and reboot the SVM for fixing it with the current version.

Rebooting the SVM does not bring down the NetScaler VPX instances running on SDX.

Follow the steps as below to reboot the SVM:
  1. Log in to XenServer with root user and password.

  2. Run the following command from shell session on the XenServer:
    xe vm-list
    Note the uuid from above command output for the Management Service VM

  3. Then run the below reboot command for the SVM instance:
    xe vm-reboot uuid=<uuid>

  4. You can check the status of the SVM, by pinging the IP, running the xe vm-list command again, or consoling into it by following CTX136121

Once there is access to the SVM, we can look for the "out of swap space" error in SVM logs under /var/log/messages as below:

Oct 25 18:57:11 nssdx-mgmt kernel: swap_pager_getswapspace(14): failed Oct 25 18:57:11 nssdx-mgmt kernel: swap_pager_getswapspace(15): failed Oct 25 18:57:11 nssdx-mgmt kernel: swap_pager_getswapspace(12): failed Oct 25 18:57:11 nssdx-mgmt kernel: swap_pager_getswapspace(9): failed

Oct 25 18:57:11 nssdx-mgmt kernel: swap_pager_getswapspace(16): failed
Oct 25 18:57:11 nssdx-mgmt kernel: swap_pager_getswapspace(12): failed
Oct 25 18:57:11 nssdx-mgmt kernel: swap_pager_getswapspace(9): failed
Oct 25 18:57:11 nssdx-mgmt kernel: swap_pager_getswapspace(16): failed
Oct 25 18:57:11 nssdx-mgmt kernel: swap_pager_getswapspace(9): failed
Oct 25 18:57:11 nssdx-mgmt kernel: swap_pager_getswapspace(16): failed
Oct 25 18:57:11 nssdx-mgmt kernel: swap_pager_getswapspace(3): failed 
Oct 25 18:57:11 nssdx-mgmt last message repeated 8 times
Oct 25 18:57:11 nssdx-mgmt kernel: pid 340 (svm_stat), uid 0, was killed: out of swap space
Oct 25 18:57:11 nssdx-mgmt kernel: pid 330 (svm_inventory), uid 0, was killed: out of swap space 
Oct 25 18:57:12 nssdx-mgmt svm_event: 10.242.176.12 10/25/2015:18:57:12 GMT : SNMP TRAP_SENT : 127.0.0.1:SubSystemState:svm_stat - SubSystem Down: svm_stat
Oct 25 18:57:12 nssdx-mgmt svm_event: 10.242.176.12 10/25/2015:18:57:12 GMT : EVENT SUBSYSTEMDOWN : 127.0.0.1:SubSystemState:svm_stat - SubSystem Down: svm_stat

The issue is tracked under the following bug number and release notes of 10.5-59.13 for reference to confirm the bug is fixed:
https://download.citrix.com/content/dam/citrix/en_us/documents/downloads/netscaler-adc/NS_10_5_59_13.html 

User Interface * CB management service (SVM) causes failure due to memory leak.. [# 565742, 573215, 576357]


Problem Cause

It is a known issue with early builds of NS10.5 SVM firmware and is resolved in 10.5 59.x and onward

Issue/Introduction

The following error is displayed on NetScaler SDX SVM: System maintenance in progress. Please try after some time

Additional Information


1) Log on to the XenServer using the root credentials. Then tried logging to the SVM through Xenserver and that gave us an error "internal authentication failed.". 

2) Type below command to get the dom id of the SVM:

xe vm-list params=name-label,dom-id name-label="Management Service VM"

The output will be like below, not down the ID for Management Service VM:

[root@netscaler-sdx ~]# xe vm-list params=name-label,dom-id name-label="Management Service VM"
name-label ( RW)    : Management Service VM
        dom-id ( RO): 1


3) Executed below command and crash the SVM:

usr/lib/xen/bin/crash_guest {domID of Management Service VM} 

4) We then got the SVM access back up. Made the same changes on the other SDX also to bring it up. 


Also we can try rebooting the SVM as mentioned below:
 
  1. Log on to the XenServer using the root credentials.

  2. Run the following command to get the UUID of the host machine:
    xe vm-list

    Output of the command will look like below:


    [root@netscaler-sdx ~]# xe vm-list
    uuid ( RO)           : b19e82df-b157-4f7f-81d8-6a583987e208
         name-label ( RW): Control domain on host: netscaler-sdx
        power-state ( RO): running


    uuid ( RO)           : e3b3d9d5-ef2b-6d4d-0312-c45f9eb71e43
         name-label ( RW): Management Service VM
        power-state ( RO): running
    Reboot the SVM using the following command:

  3. xe vm-reboot uuid=<uuid of the Management Service VM>