Followings are some of the CPU related debugging issues encoutered and the best practices to follow when working with Application Firewall:

• Check Policy hits, Bindings, Network configuration, Application Firewall configuration

               - Identify misconfiguration

               - Identify vserver that is serving the affected traffic

• Inspect logs for security violations and recent configuration changes

- /var/log/ns.log

- /var/nslog/import.log

- /var/nslog/aslearn.log

- tail -f /var/log/ns.log | grep APPFW_SIGNATURE_MATCH

Example: Jun 13 01:11:09 <local0.info> 10.217.31.98 CEF:0|Citrix|NetScaler|NS11.0|APPFW| APPFW_SIGNATURE_MATCH|6|src=10.217.253.62 spt=61141 method=GET request= http://aaron.stratum8.net/FFC/wwwboard/passwd.txt msg=Signature violation rule ID 807: web-cgi /wwwboard/passwd.txt access cn1=140 cn2=841 cs1=pr_ffc cs2=PPE0 cs3=OyTgjbXBqcpBFeENKDlde3OkMQ00001 cs4=ALERT cs5=2015 cs6=web-cgi act=not blocked

• Isolate the traffic that is effected

            - Isolate the profile

            - Isolate the security check

            - Isolate the URL, vserver and traffic parameters

• Conditional profile level trace helps identify the traffic and violation records

          - set appfw profile <profile> -trace ON

          - start nstrace -mode APPFW -size 0

          - stop nstrace

Note: Ensure that the trace is collected with -size 0 option.

• Check appfw, dht, IP reputation activity counters  

          - nsconmsg -g as_ -g appfwreq_ -g iprep -d current

• Monitor window size for resets in connection

          - Appfw sets the window size to 9845 when NetScaler resets the connection due to an invalid http message.

Examples:

• Malformed request received - connection reset
• High CPU related issues
• Check data sheets for system limits
• Inspect for cpu usage, appfw, DHT and memory related activity. Monitor appfw sessions
• nsconmsg -g cc_cpu_use -g appfwreq -g as -g dht -g mem_AS_OBJ -g mem_AS_COMPONENT -d current

• Monitor memory allocated and freed from Application Firewall components and objects during the target time period.  It helps in isolating the protection leading to high CPU usage. 

     - Profiler output

     - Observe logs

 •  Isolate appfw check leading to high CPU

      - startURLClosure

      - Formfiledconsistency

      - CSRF

      - Cookie protections

      - Referer header check

 •  Ascertain that autoupdate of signatures is not leading to high CPU (Disable to confirm)

 Make startURLClosure protection as sessionless using the following CLI option:

     > set appfw profile <profile> -sessionlessuRLClosure ON

Switching to sessionless closureURL will not have any functionality impact.