This article describes how to verify if XenMobile server is hanging/ being unresponsive because of high CPU usage

Collect Support bundle and check the below :

1. Check --- > sys_info -à top.txt

-------------------------

Running Command:/usr/bin/top -n 1 -b

-------------------------

top - 15:52:27 up 16:58,  1 user,  load average: 3.55, 3.29, 3.39

Tasks:  79 total,   1 running,  78 sleeping,   0 stopped,   0 zombie

%Cpu(s): 44.9 us,  2.5 sy,  0.0 ni, 52.3 id,  0.0 wa,  0.0 hi,  0.4 si,  0.0 st

KiB Mem:   8186552 total,  8127948 used,    58604 free,     3824 buffers

KiB Swap:  2104508 total,   280208 used,  1824300 free,   647916 cached

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND

 2809 tomcat    20   0  9.960g 6.004g  18556 S 157.0 76.90   1837:03 java

1. Check --- >sys_info -à running_process.txt and check for tomcat process

tomcat    2809  180 76.9 10443872 6295960 ?    Sl   Jun15 1837:03 /opt/sas/sw/jre/bin/java -agentlib:jdwp=transport=dt_socket,address=8001,server=y,suspend=n -Dinst1=true -DNODE_TYPE=standalone -Xms1299m -Xmx5196m -XX:MaxMetaspaceSize=256m -XX:+UseG1GC -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/opt/sas/core/tc1.hprof.core -Xloggc:/opt/sas/logs/gc.log-20170615_225442 -XX:+PrintGCDateStamps -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=20 -XX:GCLogFileSize=10M -Djsse.enableCBCProtection=false -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Djava.util.logging.config.file=/opt/sas/sw/tomcat/inst1/conf/logging.properties -Djava.endorsed.dirs=/opt/sas/sw/tomcat/inst1/endorsed -classpath /opt/sas/sw/tomcat/shared/bin/bootstrap.jar:/opt/sas/sw/tomcat/inst1/bin/tomcat-juli.jar:/opt/sas/sw/tomcat/inst1/webapps/ROOT/WEB-INF/lib/log4j-core-2.7.jar:/opt/sas/sw/tomcat/inst1/webapps/ROOT/WEB-INF/lib/log4j-api-2.7.jar:/opt/sas/sw/tomcat/inst1/webapps/ROOT/WEB-INF/lib/log4j-1.2-api-2.7.jar:/opt/sas/sw/licensing/MFLic.jar -Djava.library.path=/opt/sas/sw/lib -Dcatalina.base=/opt/sas/sw/tomcat/inst1/ -Dcatalina.home=/opt/sas/sw/tomcat/shared/ -Djava.io.tmpdir=/opt/sas/sw/tomcat/inst1/temp -Dcom.sun.xml.bind.v2.runtime.JAXBContextImpl.fastBoot=true -Djavax.net.ssl.trustStore=/opt/sas/sw/tomcat/inst1/conf/oca_securekeys.truststore -Dlog4j.configurationFile=file:/opt/sas/sw/tomcat/inst1/webapps/ROOT/WEB-INF/classes/log4j2.xml -Dfile.encoding=UTF-8 -Dcom.sun.jndi.ldap.connect.pool.maxsize=0 -Dcom.sun.jndi.ldap.connect.pool.prefsize=10 -Dcom.sun.jndi.ldap.connect.pool.timeout=300000 -Djtds.customTrustManager=com.citrix.security.ssl.HostnameVerifyingTrustManager -Dcom.citrix.xms.apns.enableKeepAlive=true org.apache.catalina.startup.Bootstrap start

 

3) Check Support bundle --- > Threaddumpv2.html

Check if  CPU time enabled = true

Search for the RUNNABLE thread with highest CPU time

Thread "pool-11-thread-1" RUNNABLE

CPU Time: 12h 36min 18s 932ms 325µs 418ns (45378932325418 ns)   ------ > High CPU time shown

User Time: 12h 26min 58s 720ms 0µs 0ns (44818720000000 ns)

Blocked count: 19069

Waited count: 12776

Locked synchronizer:

                hashCode 0x4D19252E, java.util.concurrent.ThreadPoolExecutor$Worker: java.util.concurrent.ThreadPoolExecutor$Worker@4d19252e  ------- >  This is the java thread which is causing high CPU time and the process is ThreadPoolExecutor

Stacktrace:

                org.hibernate.persister.entity.AbstractEntityPersister.getTuplizer(AbstractEntityPersister.java:3474)

                org.hibernate.persister.entity.AbstractEntityPersister.getIdentifier(AbstractEntityPersister.java:3876)

                org.hibernate.event.def.DefaultFlushEntityEventListener.checkId(DefaultFlushEntityEventListener.java:80)

                org.hibernate.event.def.DefaultFlushEntityEventListener.getValues(DefaultFlushEntityEventListener.java:190)

                org.hibernate.event.def.DefaultFlushEntityEventListener.onFlushEntity(DefaultFlushEntityEventListener.java:147)

                org.hibernate.event.def.AbstractFlushingEventListener.flushEntities(AbstractFlushingEventListener.java:219)

                org.hibernate.event.def.AbstractFlushingEventListener.flushEverythingToExecutions(AbstractFlushingEventListener.java:99)

                org.hibernate.event.def.DefaultFlushEventListener.onFlush(DefaultFlushEventListener.java:50)

                org.hibernate.impl.SessionImpl.flush(SessionImpl.java:1216)

                org.hibernate.impl.SessionImpl.managedFlush(SessionImpl.java:383)

                org.hibernate.transaction.JDBCTransaction.commit(JDBCTransaction.java:133)

                org.springframework.orm.hibernate3.HibernateTransactionManager.doCommit(HibernateTransactionManager.java:657)

                org.springframework.transaction.support.AbstractPlatformTransactionManager.processCommit(AbstractPlatformTransactionManager.java:755)

                org.springframework.transaction.support.AbstractPlatformTransactionManager.commit(AbstractPlatformTransactionManager.java:724)

                org.springframework.transaction.interceptor.TransactionAspectSupport.commitTransactionAfterReturning(TransactionAspectSupport.java:475)

                org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:270)

                org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:94)

                org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)

                org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:91)

                org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)

                org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)

                com.sun.proxy.$Proxy234.saveVppLicenseFromApple(Unknown Source)

                com.citrix.ios.vpp.wsclient.cron.VPPImportTask.run(VPPImportTask.java:180)

                com.sparus.nps.util.TaskInSession$Wrapper.runInSession(TaskInSession.java:217)

                com.sparus.nps.util.TaskInSession.run(TaskInSession.java:137)

                com.sparus.nps.cron.Cron$Job.run(Cron.java:218)

                java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)

                java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)

                java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)

                java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)

                java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)

                java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)

                java.lang.Thread.run(Thread.java:745)

The java threads responsible for CPU time consumption can be different .

• The different threads can be C3P0 , which can be fixed by XenMobile Scaling i.e increase the resources dedicated to the environment or infrastructure or by having dedicated vCPU’s for each instance
• The ThreadPool Executor can be fixed by increasing the MaximumPoolSize , we can verify what is the max PoolSize set to and then scale it according to customer’s environment

Note : In the code we can search for the function setMaximumPoolsize(int) to see what is the value set . To retrieve the value the function used is getMaximumPoolSize(int)

KeyWords :

PoolSize : Number of threads in a Pool