Netscaler is not caching anything on Netscaler. IC is enabled and sufficient memory is allocated. It was working fine before and suddenly stopped working.

exec: show cache object  (Means there is nothing cached on netscaler)
Done
 

The issue is caused due to a known issue  where IC and AppFw is being used at the same time:

Below is the release notes to confirm the fix details:
 
https://download.citrix.com/content/dam/citrix/en_us/documents/downloads/netscaler-adc/NS_10_5_57_7.html
 
A 64 bit memory leak in the application firewall module might lead to cache misses. The memory leak occurs when the cache is turned on and any of the advanced application firewall security checks are enabled. The application firewall memory leak is now fixed, and the fix resolves the interoperability issue with the cache module.
[# 549466]

Below are the artefacts to confirm if it is similar issue:
===
Run below command from shell prompt by going to /var/nslog directory:

nsconmsg -K newnslog -d statswt0 -g appfw | sort -k3
 
Done.
    5       0             54155 http_tot_appfw_async_kill
    3       0            228352 cacmem_cur_appfw_allocnetbuffs
    1       0         229055691 cactor_tot_appfw_obj_write_failed

From Config, we are seeing the AppFw and IC feature are enabled:
 
enable ns feature WL SP LB CS CMP SSL CF IC SSLVPN REWRITE AppFw RESPONDER HTMLInjection AppFlow



Integrated Cache Statistics - Summary
                                          Rate (/s)                Total
Hits                                               0              2590221                        <---------------Hits ratio is very very low
Misses                                           369           8126419583
Requests                                         369           8129009804
Hit ratio(%)                                      --                    0
Origin bandwidth saved(%)                         --                    0
Cached objects                                    --                    0
Marker objects                                    --                    0
                                          Rate (/s)                Total
Requests                                         369           8129009804
 
 
                Memory Usage Statistics
                                              Total
Maximum memory(KB)                            524288
Maximum memory active value(KB)               524288
Utilized memory(KB)                             4096
Memory allocation failures                4800265551                     <-------------------  High memory allocation failures.
Largest response so far(B)                     81180
Cached objects                                     0                                          <----------------------------- Zero cached objects.
Marker objects                                     0
 
Hits being served                                  0
Misses being handled                               0

 
In ns.conf we will see CSRF is enabled in AppFw config:

set snmp alarm APPFW-CSRF-TAG -state ENABLED -severity Unknown -logging ENABLED -timeout 1
add appfw profile AZTR_Web_WAF_Profile -startURLAction learn log stats -contentTypeAction none -startURLClosure OFF -denyURLAction log stats -RefererHeaderCheck OFF -cookieConsistencyAction learn log stats -cookieTransforms OFF -cookieEncryption none -cookieProxying none -addCookieFlags none -fieldConsistencyAction learn log stats -CSRFtagAction learn log stats -crossSiteScriptingAction learn log stats -crossSiteScriptingTransformUnsafeHTML OFF -crossSiteScriptingCheckCompleteURLs OFF -SQLInjectionAction learn log stats -SQLInjectionTransformSpecialChars OFF -SQLInjectionType SQLSplCharANDKeyword -SQLInjectionCheckSQLWildChars OFF -fieldFormatAction learn log stats -defaultFieldFormatMinLength 0 -defaultFieldFormatMaxLength 65535 -bufferOverflowAction log stats -bufferOverflowMaxURLLength 1024 -bufferOverflowMaxHeaderLength 4096 -bufferOverflowMaxCookieLength 4096 -creditCardAction none -creditCardMaxAllowed 0 -creditCardXOut OFF -responseContentType "application/octet-stream" -XMLDoSAction learn log stats -XMLFormatAction log stats -XMLSQLInjectionAction log stats -XMLSQLInjectionType SQLSplCharANDKeyword -XMLSQLInjectionCheckSQLWildChars OFF -XMLSQLInjectionParseComments checkall -XMLXSSAction log stats -XMLWSIAction learn log stats -XMLAttachmentAction learn log stats -XMLValidationAction log stats -XMLErrorObject " " -signatures az2_signature -XMLSOAPFaultAction log stats -useHTMLErrorObject OFF -errorURL "/" -HTMLErrorObject " " -logEveryPolicyHit OFF -stripHtmlComments none -stripXmlComments none -exemptClosureURLsFromSecurityChecks ON -defaultCharSet iso-8859-1 -postBodyLimit 20000000 -fileUploadMaxNum 65535 -canonicalizeHTMLResponse ON -enableFormTagging ON -sessionlessFieldConsistency OFF -sessionlessURLClosure OFF -semicolonFieldSeparator OFF -excludeFileUploadFromChecks OFF -SQLInjectionParseComments checkall -invalidPercentHandling secure_mode -type HTML XML -checkRequestHeaders OFF -optimizePartialReqs ON -URLDecodeRequestCookies OFF
set appfw learningsettings AZTR_Web_WAF_Profile -startURLMinThreshold 1 -startURLPercentThreshold 0 -cookieConsistencyMinThreshold 1 -cookieConsistencyPercentThreshold 0 -CSRFtagMinThreshold 1 -CSRFtagPercentThreshold 0 -fieldConsistencyMinThreshold 1 -fieldConsistencyPercentThreshold 0 -crossSiteScriptingMinThreshold 1 -crossSiteScriptingPercentThreshold 0 -SQLInjectionMinThreshold 1 -SQLInjectionPercentThreshold 0 -fieldFormatMinThreshold 1 -fieldFormatPercentThreshold 0 -XMLWSIMinThreshold 1 -XMLWSIPercentThreshold 0 -XMLAttachmentMinThreshold 1 -XMLAttachmentPercentThreshold 0


 

---

Problem Cause

AppFw is leaking our 64 bit memory when AppFw (advanced mode) and Cache are turned on at the same time due to not calling clean_appfw_pcb_data in initClientForReuse.