![[Note]](images.admon/note.gif)
Out of Memory exceptions in WebSphere Application Server
An Out of Memory exception can be caused by numerous reasons. A memory leak can be indicated by the steady increase in the memory usage without de-allocation, as shown in figure below in IBM Pattern Modeling and Analysis Tool in ISA. It is expected that the Java heap usage fluctuates up and down because the de-allocation of memory is only triggered when the heap usage reaches certain levels.
Figure above is similar to a saw, with slow increase in the memory (objects allocation) and then a sudden decrease (garbage collection). If in every cycle, the heap usage after garbage collector execution shows a steady increase, it can also indicate a memory leak condition.
Another type of out of memory issue can be caused by heap fragmentation. Heap fragmentation is caused by the allocation and de-allocation of large objects in a short amount of time. The heap will have enough space to accommodate new large objects, but that space is not contiguous, causing an Out of Memory exception. In such cases, a steady increase pattern is not seen in the chart, so then the analysis tools will help in determining if heap fragmentation is the problem.
An out of memory condition is most commonly detected when an OutOfMemoryError exception is
shown in the SystemErr.log file, as shown below:
[6/29/12 18:40:31:919 EDT] 0000008f SystemErr R java.lang.OutOfMemoryError: Java heap space [6/29/12 18:40:31:922 EDT] 0000008f SystemErr R at com.ibm.AllocateObject.doWork(AllocateObject.java:45) [6/29/12 18:40:31:924 EDT] 0000008f SystemErr R at com.ibm.BaseServlet.service(BaseServlet.java:72) [6/29/12 18:40:31:925 EDT] 0000008f SystemErr R at javax.servlet.http.HttpServlet.service(HttpServlet.java:668) ... [6/29/12 18:40:32:602 EDT] 0000008f SystemErr R at com.ibm.io.async.AbstractAsyncFuture.invokeCallback(AbstractAsyncFuture.java:217) [6/29/12 18:40:32:603 EDT] 0000008f SystemErr R at com.ibm.io.async.AsyncChannelFuture$1.run(AsyncChannelFuture.java:205) [6/29/12 18:40:32:604 EDT] 0000008f SystemErr R at com.ibm.ws.util.ThreadPool$Worker.run(ThreadPool.java:1783)
After an Out Of Memory exception is detected, WebSphere Application Server by default creates three log files.
The files are one Java Core, one Portable Heap Dump (or simply Heap dump) and one System Dump
(native platform core dump), as shown in the native_stderr.log
JVMDUMP039I Processing dump event "systhrow", detail "java/lang/OutOfMemoryError" at 2012/06/29 18:39:48 - please wait. JVMDUMP032I JVM requested System dump using '/opt/IBM/WebSphere/AppServer/profiles/AppSrv01/core.20120629.183948.19153.0001.dmp' in response to an event JVMDUMP010I System dump written to /opt/IBM/WebSphere/AppServer/profiles/AppSrv01/core.20120629.183948.19153.0001.dmp JVMDUMP032I JVM requested Heap dump using '/opt/IBM/WebSphere/AppServer/profiles/AppSrv01/heapdump.20120629.183948.19153.0002.phd' in response to an event JVMDUMP010I Heap dump written to /opt/IBM/WebSphere/AppServer/profiles/AppSrv01/heapdump.20120629.183948.19153.0002.phd JVMDUMP032I JVM requested Java dump using '/opt/IBM/WebSphere/AppServer/profiles/AppSrv01/javacore.20120629.183948.19153.0003.txt' in response to an event JVMDUMP010I Java dump written to /opt/IBM/WebSphere/AppServer/profiles/AppSrv01/javacore.20120629.183948.19153.0003.txt JVMDUMP032I JVM requested Snap dump using '/opt/IBM/WebSphere/AppServer/profiles/AppSrv01/Snap.20120629.183948.19153.0004.trc' in response to an event JVMDUMP010I Snap dump written to /opt/IBM/WebSphere/AppServer/profiles/AppSrv01/Snap.20120629.183948.19153.0004.trc JVMDUMP013I Processed dump event "systhrow", detail "java/lang/OutOfMemoryError".
Using these files together with the native_stderr.log, which contains the verbose garbage
collection output (if enabled), you can have a first look. These files and logs help diagnose
what is accumulating in the memory and causing the Out of Memory exception.
The following steps help identify the likely cause of the Out Of Memory Exception:
Download the portable heap dump file (
.phd) and thenative_stderr.logfile to a computer with ISA installed. The System Dump file (.dmp) is only needed in case the Out Of Memory exception is being caused by something outside the Java Heap, such as the JVM itself or native libraries.Open the
native_stderr.logfile in the ISA tool IBM Monitoring and Diagnostic Tools for Java - Garbage Collection and Memory Visualizer.The tool provides views of the heap usage and reports possible issues and recommendations, as shown in the figure below:
To understand what objects are getting filled up in memory, use a tool to process the contents of the Portable Heap Dump and allow navigation and drill-down, such as IBM Heap Analyzer or IBM Monitoring and Diagnostic Tools for Java - Garbage Collection and Memory Analyzer. Figure below shows an example of IBM Heap Analyzer:
