Citrix ADM, formerly NetScaler MAS

The following section lists some of the frequently asked questions during diagnosis and troubleshooting of NetScaler MAS issues:

• How to verify the NetScaler MAS build version using CLI and support file?
• How does MAS fetch all the dashboard related data from connected devices?
• How to capture NetScaler tcpdumps on MAS?
• How does NetScaler MAS communicate with different devices after adding them?
• How to upgrade different instances through NetScaler MAS?
• How to run techsupport script from CLI of NetScaler MAS?
• How to capture effective debug logs to troubleshoot insight related isues?
• What are the new features which assists in troubleshooting NetScaler MAS issues?
• What are the different processes on NetScaler MAS and how to troubleshoot different issues using mps_*.log in /var/mps/log?
• How to verify NetScaler MAS configuration using the DB?
• What does various NetScaler MAS process states refer to (Output of ‘PS –ax’)?

Q: How to verify the NetScaler MAS build version using CLI and support file?

A: On a live device check the version.conf file under /mps:
bash-2.05b# cat version.conf
11.1-52.15

In support file you can open /var/nslog/dmesg.boot and grep for flash. You should see the build version.
bash-2.05b# grep flash dmesg.boot
/mas-11.1-52.15 -> /flash/mas-11.1-52.15

Back to Top

Q: How does MAS fetch all the dashboard related data from connected devices?

A: MAS does Nitro calls to instances and gets all the details.
To confirm how many vservers each instance has and communicated by NetScaler can be seen in mps_inventory.log:
Wednesday, 22 Mar 17 19:08:02.287 +0530 [Debug] [Emon[#60]] HTTP Request Protocol: https, ContentType: , Method: GET, URL: https://10.107.100.131/nitro/v1/config/lbvserver?attrs=name,ipv46,port,servicetype,effectivestate,curstate,health,tickssincelaststatechange,comment,lbmethod,persistencetype,totalservices,activeservices&pagesize=10000&pageno=1&format=json

Wednesday, 22 Mar 17 19:08:02.371 +0530 [Debug] [Emon[#60]] EMON_LB_VIP: 10.107.100.131, db_objects: 9, new_objects: 8

In the above output you will see differeance in db_objects and new_objects as a vserver was deleted from NetScaler when this call was last made. For performance related dashboard and graphs output look at /var/mps/mps_perf.log

Back to Top

Q: How to capture NetScaler tcpdumps on NetScaler MAS?

A: Use the following command to capture tcpdumps on NetScaler MAS:
Tcpdump –i 1 <options>

The switch -i specifies the interface and 1 represents the first interface on the MAS appliance. For further reference please review https://www.freebsd.org/cgi/man.cgi?tcpdump(1)

For example:
tcpdump -i 1 udp and src 10.107.100.131
tcpdump -i 1 tcp
tcpdump -i 1 src NSIP or dst NSIP

Back to Top

Q: How does NetScaler MAS communicate with different devices after adding them?

A: NetScaler MAS periodically communicates through Nitro calls for each and every instance and resource added to it. You can see these calls in mps_inventory.log and mps_config.log

Back to Top

Q: How to upgrade different instances through NetScaler MAS?

A: Complete the following steps to upgrade different instances through NetScaler MAS:

1. Go to Infrastructure > Configuration Jobs > Maintenance tasks.
2. Now select the device that you want to upgrade.

Back to Top

Q: How to run techsupport script from CLI of NetScaler MAS?

A: Use the following command to run the techsupport script:
cd /mps/scripts
./techsupport.pl

Back to Top

Q: How to capture effective debug logs to troubleshoot insight related isues?

A: Complete the following steps to capture debug logs:

1. Whenever you are capturing a support file to troubleshoot an issue, enable ‘Collect Debug logs’ in the tech-support page:

   

2. When you enable this, additional debug level afdecoder logs are captured. This will help troubleshoot the issue further.
3. The time that you will specify is the amount of time these additional debug logs will be captured after clicking on OK.
4. So if you are troubleshooting HDX insight issue, give a suitable time and click on OK which will start the debugging:

   

5. Now replicate the issue within that time interval so that debug logs pertaining to the issue are captured. Support file will automatically be generated after this timer expires.

Back to Top

Q: What are the new features which assists in troubleshooting NetScaler MAS issues?

This is available only from 12.0 onwards.

• Go to System > Diagnostics > Troubleshooting.
• You can then select the instance you want to diagnose and get the Diagnostics chart.
• Remember, only the vservers and instances where AppFlow is enabled will work with this.
• It will show you different AppFlow related parameters enabled on the device and different vserver states.

Back to Top

Q: What are the different processes on NetScaler MAS and how to troubleshoot different issues using mps_*.log in /var/mps/log?

A: The following are the different processes on NetScaler MAS:

• Control SubSystem (mps_control.log) : Initialize+monitor+stop other subsystems and the database. It is responsible for restarting any subsystem if it crashes.
• Service SubSystem (mps_service.log) : It has an inbuilt HTTP(s) Request/Response handler. It listens on port 80 and 443. Any request from UI/API will hit the Service SubSystem. Based on the type of request, it might process the request itself or pass it on to the other appropriate SubSystem. Response always goes back via Service SubSystem.
• Inventory SubSystem (mps_inventory.log) : It does inventory from NetScaler/SD-WAN instances and updates instances' information in the database. It retrieves build/system information from NetScaler Instances. It runs complete inventory every 30 minutes by default. This subsystem also retrieves statistics from instances to show CPU/Memory usage etc.
• Config SubSystem (mps_config.log) : It processes any configuration request that is received from Service Process. Configuration request can be adding instance, or any other operations on NetScaler instane or on MAS itself. It is also related to admin user management, device profiles, external authentication server config etc.
• Event SubSystem (mps_event.log) : It raises internal events in case of any SubSystem failure or configuration changes. This subsystem also registers itself with NetScaler/SD-WAN instances and SDX appliances to receive syslogs and provide event based reporting. All traps and syslogs events come to this process.
• Perf SubSystem (mps_perf.log) : This subsystem is responsible for performance reporting of NetScaler/SD-WAN instances. This retrieves instance stats every 5 mins and aggregates them on minutely, hourly, daily and weekly basis. There are pre-defined reports.
• afdecoder SubSystem (mps_afdecoder.log) : This subsystem is responsible to receive AppFlow traffic from NetScaler/SD-WAN instances and process that data.
• afanalytics SubSystem (mps_afanalytics.log) : This subsystem is responsible for analytics reporting of NetScaler/SD-WAN instances. This subsystem aggregates the data on minutely, hourly, daily and weekly basis. There are pre-defined reports.

Back to Top

Q: How to verify NetScaler MAS configuration using the DB?

A: All the config is stored in a csv files under /var/mps/mpsdb directory.You can open each file and check the config.

Back to Top

Q: What does various NetScaler MAS process states refer to (Output of ‘PS –ax’)?

A: Here are the different values that the s, stat and state output specifiers (header "STAT" or "S") will display to describe the state of a process:

•        D    uninterruptible sleep (usually IO)
•        R    running or runnable (on run queue)
•        S    interruptible sleep (waiting for an event to complete)
•        T    stopped, either by a job control signal or because it is being traced.
•        W    paging (not valid since the 2.6.xx kernel)
•        X    dead (should never be seen)
•        Z    defunct ("zombie") process, terminated but not reaped by its parent.

For BSD formats and when the stat keyword is used, additional characters may be displayed:

•        <    high-priority (not nice to other users)
•        N    low-priority (nice to other users)
•        L    has pages locked into memory (for real-time and custom IO)
•        s    is a session leader
•        l    is multi-threaded (using CLONE_THREAD, like NPTL pthreads do)
•        +    is in the foreground process group.

Back to Top

This article lists the frequently asked questions during diagnosis and troubleshooting of NetScaler MAS issues.

The troubleshooting sections which was previously present in this article has been moved to Troubleshooting section of NetScaler MAS: How Do I? page.

CTX261729 - Processes, Logs And Configuration Files Participating in Citrix ADM High Availability
CTX261991 - How To Troubleshoot And Fix The Situation When The ADM HA Is Not Working
CTX261932 - How To Fix The Database Replication Issues In Various Situations