Memory usage increases on appliances continuously and finally causes an access failure.
Memory allocation failure is observed if the following conditions are met:
1.we can see all these allocation errors are happening for NSPPE-2 and not others.
nsconmsg -K newnslog -d finalstatswt0 -s pedist=1 -g mem_err | grep -v Aggr |more
Index reltime counter-value symbol-name&device-no
1 1770995 2013 mem_err_alloc_failed MEM_MISC (PE-2)
3 0 12 mem_err_alloc_failed mempool_MISC(MISC) (PE-2)
5 0 12 mem_err_alloc_failed MEM_WU_MEM (PE-2)
7 0 2165 mem_err_alloc_failed mempool_POLICYENGINE(POLICYENGINE) (PE-2)
9 0 152 mem_err_alloc_failed MEM_PI_RUNTIME (PE-2)
11 0 362314663 mem_err_alloc_failed mempool_SYSTEM(SYSTEM) (PE-2)
13 0 362314663 mem_err_alloc_failed MEM_TBUF (PE-2)
15 0 154169 mem_err_alloc_failed mempool_APPSECURE(APPSECURE) (PE-2)
17 0 154169 mem_err_alloc_failed MEM_APPSECURE (PE-2)
37 0 2110783488 mem_err_alloc_failed_size MEM_MISC (PE-2)
2.Cause of lots of allocation is due to APPSECURE module
nsconmsg -K newnslog -d finalstatswt0 -s pedist=1 -g mem_cur_bm_allocsize | grep "PE-2" | grep mempool | sort -k 3 | more
81 0 64 mem_cur_bm_allocsize mempool_LOGSTREAM(LOGSTREAM) (PE-2)
87 0 1248 mem_cur_bm_allocsize mempool_L2L3(L2L3) (PE-2)
93 0 2432 mem_cur_bm_allocsize mempool_DNS(DNS) (PE-2)
91 0 12032 mem_cur_bm_allocsize mempool_POLICYENGINE(POLICYENGINE) (PE-2)
95 0 16512 mem_cur_bm_allocsize mempool_LOGGING(LOGGING) (PE-2)
97 0 27104 mem_cur_bm_allocsize mempool_SYSTEM(SYSTEM) (PE-2)
89 0 1151744 mem_cur_bm_allocsize mempool_SSL(SSL) (PE-2)
83 0 3651168 mem_cur_bm_allocsize mempool_DHT(DHT) (PE-2)
85 0 931299616 mem_cur_bm_allocsize mempool_APPSECURE(APPSECURE) (PE-2)
3.As per below, we can see mem_AS_COMPONENT_AS_URL and mem_AS_OBJ_SHA_URL_HASH_BUCKET consuming most of memory for PE-2.
nsconmsg -K newnslog -d finalstatswt0 -s pedist=1 -g mem_AS_ | grep "PE-2" | sort -k 3 | more
...
249 0 20029061 mem_AS_COMPONENT_DHT (PE-2)
341 0 23184168 mem_AS_OBJ_HTTP_HEADERS (PE-2)
1057 0 57542473 mem_AS_OBJ_WU_CONTEXT (PE-2)
89 0 64822062 mem_AS_COMPONENT_CONFIG (PE-2)
1093 0 647124688 mem_AS_OBJ_SHA_URL_HASH_BUCKET (PE-2)
117 0 647124784 mem_AS_COMPONENT_AS_URL (PE-2)
The investigation revealed unsafe cleanup of the URL SHA hash table potentially leading to memory allocation failure.
This is a known issue tracked with NSHELP-26435