Lab: Java OutOfMemoryErrors

1. Request the following web page from your terminal to simulate an OutOfMemoryError:

   1. macOS, Linux, or Windows with Cygwin:

      curl -k -s "https://$(oc get route libertydiag "--output=jsonpath={.spec.host}")/servlet/AllocateObject?size=1048576&iterations=10000&waittime=10&retainData=false"
      

   2. Windows with Command Prompt:

      1. Ensure you have curl for Windows installed

      2. List the application's URL:

         oc get route libertydiag "--output=jsonpath={.spec.host}{'\n'}"
         

      3. Execute the following command, replacing $HOST with the output of the previous command:

         curl -k -s "https://$HOST/servlet/AllocateObject?size=1048576&iterations=10000&waittime=10&retainData=false"
         

1. Craft the following link, replacing $HOST with the hostname of the application from the installation step above when you opened the libertydiag application in your browser (e.g. libertydiag-user1-namespace.apps.was-education-cluster.example.com):

   http://$HOST/servlet/AllocateObject?size=1048576&iterations=10000&waittime=10&retainData=false
   

2. Open this link in the browser

1. List the pods for the example application deployment; for example:

   oc get pods
   

   Example output:

   NAME                          READY   STATUS    RESTARTS   AGE
   libertydiag-b98748954-mgj64   1/1     Running   0          97s
   

2. Print the native logs of the pod by replacing $POD with the pod name from the previous command:

   oc logs $POD
   

   For example:

   oc logs libertydiag-b98748954-mgj64
   

3. You should see OutOfMemoryError lines such as the following. If you don't see them, give it some time and print the logs again.

   JVMDUMP039I Processing dump event "systhrow", detail "java/lang/OutOfMemoryError" at 2022/12/07 16:43:38 - please wait.
   

1. In the Topology view of the Developer perspective, click on the libertydiag circle, then click the Resources tab in the drawer on the right, and then click on View logs for the one pod that's running:
   
2. You should see OutOfMemoryError lines such as the following. If you don't see them, give it some time by watching the bottom of the logs.
   

1. List the pods for the example application deployment; for example:

   oc get pods
   

   Example output:

   NAME                          READY   STATUS    RESTARTS   AGE
   libertydiag-b98748954-mgj64   1/1     Running   0          97s
   

2. Open a shell into the pod by replacing $POD with a pod name from the previous command:

   oc rsh -t $POD
   

   For example:

   oc rsh -t libertydiag-b98748954-mgj64
   

3. Note that the remote shell might timeout after a short period of inactivity, so be aware that you might have been logged out and you'll need to oc rsh back in to continue where you left off.

4. First, we need to find the process ID (PID) of Liberty. Most Liberty images do not have tools like ps or top pre-installed. However, most Liberty images only have a single process in the container which is the Java process running Liberty, and this has the PID of 1. Double check that this is the Liberty process by doing a full listing on PID 1:

   ls -l /proc/1/
   

5. If you see a Liberty current working directory (cwd) such as /opt/ol/wlp or /opt/ibm/wlp then you can assume that is the Liberty process. Otherwise, run ls -l /proc/[0-9]* and then explore each PID to find the Liberty process.

   ls -l /proc/1
   

   Example output:

   [...]
   lrwxrwxrwx.  1 1000830000 root 0 Dec  6 17:45 cwd -> /opt/ibm/wlp/output/defaultServer
   -r--------.  1 1000830000 root 0 Dec  6 17:45 environ
   lrwxrwxrwx.  1 1000830000 root 0 Dec  6 14:57 exe -> /opt/ibm/java/jre/bin/java
   [...]
   

6. Normally, heapdumps for IBM Java and Semeru will be produced as heapdump*phd files in the cwd directory that you found above:

   ls -l /opt/ibm/wlp/output/defaultServer/heapdump*phd
   

   However, in the case of this sample application, this directory is overridden with an -Xdump configuration. You can check JVM configurations by printing the process cmdline and environ files and find the relevant configuration. For example:

   cat /proc/1/cmdline /proc/1/environ
   

   Example output:

   [...] -Xdump:directory=logs/diagnostics/
   

   Therefore, for this application, heapdumps will go into logs/diagnostics/ relative to cwd:

   ls /opt/ibm/wlp/output/defaultServer/logs/diagnostics/heapdump*phd
   

   Example output:

   /opt/ibm/wlp/output/defaultServer/logs/diagnostics/heapdump.20221207.172449.1.0002.phd
   /opt/ibm/wlp/output/defaultServer/logs/diagnostics/heapdump.20221207.172502.1.0005.phd
   /opt/ibm/wlp/output/defaultServer/logs/diagnostics/heapdump.20221207.172504.1.0008.phd
   /opt/ibm/wlp/output/defaultServer/logs/diagnostics/heapdump.20221207.172504.1.0009.phd
   

   Note that overridding the -Xdump directory is common in container deployments so that a directory may be used that's mounted on a permanent disk so that diagnostics are still available if a pod is killed.

1. Download the heaps dumps by replacing $POD with a pod name from above and $DIR with the directory of the heapdumps. Note that oc cp does not support wildcards so the whole directory (or a single file) must be downloaded.

   oc cp $POD:$DIR .
   

   For example:

   oc cp libertydiag-ddf5f95b6-wj6dm:/opt/ibm/wlp/output/defaultServer/logs/diagnostics .
   

Files other than native logs (equivalent to Liberty's console.log) cannot be downloaded through the browser. You must use the command line steps above.

1. Go to https://www.ibm.com/support/pages/eclipse-memory-analyzer-tool-dtfj-and-ibm-extensions
2. Follow the instructions to download and launch MAT. These dumps are produced by IBM Java 8, so choose that download.
3. Click File } Open Heap Dump...
4. Open the first phd file that was downloaded.
5. In the Getting Started Wizard, choose Leak Suspects Report and click Finish

6. Review the leak suspects report which will highlight the cause of the OOME. For example:

   The class com.example.servlet.AllocateObject occupies 426,783,880 (91.14%) bytes. The memory is accumulated in one instance of java.lang.Object[] which occupies 426,779,160 (91.14%) bytes.

1. Check if you have authority for all verbs on all resources:

   oc auth can-i '*' '*'
   

   Example output:

   yes
   

2. If the answer is no, then you do not have cluster-admin permissions.

1. Access your OpenShift web console at https://console-openshift-console.$CLUSTER/. Replace $CLUSTER with your OpenShift cluster domain.
2. Ensure the perspective is set to Administrator in the top left:
   
3. Expand User Management. If you don't see a Users option, then you do not have cluster-admin permissions. If you do see it, click on it, and then click on your user name:
   
4. Click on RoleBindings and check if any binding has a Role ref of cluster-admin. If there are none, then you do not have cluster-admin permissions.
   

1. Request the following web page from your terminal to simulate an OutOfMemoryError:

   1. macOS, Linux, or Windows with Cygwin:

      curl -k -s "https://$(oc get route libertydiag "--output=jsonpath={.spec.host}")/servlet/AllocateObject?size=1048576&iterations=10000&waittime=10&retainData=false"
      

   2. Windows with Command Prompt:

      1. Ensure you have curl for Windows installed

      2. List the application's URL:

         oc get route libertydiag "--output=jsonpath={.spec.host}{'\n'}"
         

      3. Execute the following command, replacing $HOST with the output of the previous command:

         curl -k -s "https://$HOST/servlet/AllocateObject?size=1048576&iterations=10000&waittime=10&retainData=false"
         

1. Craft the following link, replacing $HOST with the hostname of the application from the installation step above when you opened the libertydiag application in your browser (e.g. libertydiag-user1-namespace.apps.was-education-cluster.example.com):

   http://$HOST/servlet/AllocateObject?size=1048576&iterations=10000&waittime=10&retainData=false
   

2. Open this link in the browser

1. List the pods for the example application deployment; for example:

   oc get pods
   

   Example output:

   NAME                          READY   STATUS    RESTARTS   AGE
   libertydiag-b98748954-mgj64   1/1     Running   0          97s
   

2. Print the native logs of the pod by replacing $POD with the pod name from the previous command:

   oc logs $POD
   

   For example:

   oc logs libertydiag-b98748954-mgj64
   

3. You should see OutOfMemoryError lines such as the following. If you don't see them, give it some time and print the logs again.

   JVMDUMP039I Processing dump event "systhrow", detail "java/lang/OutOfMemoryError" at 2022/12/07 16:43:38 - please wait.
   

1. In the Topology view of the Developer perspective, click on the libertydiag circle, then click the Resources tab in the drawer on the right, and then click on View logs for the one pod that's running:
   
2. You should see OutOfMemoryError lines such as the following. If you don't see them, give it some time by watching the bottom of the logs.
   

1. List the pods for the example application deployment; for example:

   oc get pods
   

   Example output:

   NAME                          READY   STATUS    RESTARTS   AGE
   libertydiag-b98748954-mgj64   1/1     Running   0          97s
   

2. Find the worker node of the pod by replacing $POD with the pod name from the previous command:

   oc get pod --output "jsonpath={.spec.nodeName}{'\n'}" $POD
   

   For example:

   oc get pod --output "jsonpath={.spec.nodeName}{'\n'}" libertydiag-b98748954-mgj64
   

3. Start a debug pod on that worker node by replacing $NODE with the node name from the previous command:

   oc debug node/$NODE -t
   

   For example:

   oc debug node/worker0.was-education-cluster.example.com -t
   

4. Note that the remote shell might timeout after a short period of inactivity, so be aware that you might have been logged out and you'll need to re-run oc debug node to continue where you left off.

5. These systems use systemd-coredump which places core dumps in /var/lib/systemd/coredump/ (other systems may use other locations). To see them through the debug pod, prepend /host:

   sh-4.4# ls -l /host/var/lib/systemd/coredump/
   total 115824
   -rw-r-----+ 1 root root 118596950 Feb 28 16:25 'core.Default\x20Executo.1001210000.06a2f2e0997c46b2b50bdac3c25ab072.831941.1677601537000000.lz4'
   

6. Confirm the matching timestamp when the OutOfMemoryError occurred.

7. Now we'll use this debug pod to download the file. First start a looping output so that the debug pod doesn't timeout by executing:

   while true; do echo 'Sleeping'; sleep 8; done
   

8. Next, open a new terminal and find the debug pod and namespace. For example:

   oc get pods --field-selector=status.phase==Running --all-namespaces | grep debug
   

   Example output:

   openshift-debug-node-4whw7mhb8j  worker0was-education-clusterexamplecom-debug   1/1     Running   0     2m46s
   

9. Use the above namespace (first column) and pod name (second column) to download the core dump from the worker node. Unfortunately, there is an open issue with escape characters so we must download the entire dumps directory:

   oc cp --namespace openshift-debug-node-4whw7mhb8j "worker0was-education-clusterexample.com-debug:/host/var/lib/systemd/coredump/" coredumps
   

10. Change directory into coredumps and any subdirectory that was created and then uncompress the core dump:

    1. Install the lz4 utility:

       1. macOS:

          brew install lz4
          

       2. Windows: Download lz4 (under Assets at the bottom)

       3. Linux: Install the lz4 program

    2. Decompress the file (replacing $FILE with the file name):

       lz4 -dc $FILE.lz4 > core.dmp
       

11. Go back to the previous terminal window and type Ctrl^C to stop the while loop, and then exit the debug node.

Files other than native logs (equivalent to Liberty's console.log) cannot be downloaded through the browser. You must use the command line steps above.

1. Go to https://www.ibm.com/support/pages/eclipse-memory-analyzer-tool-dtfj-and-ibm-extensions
2. Follow the instructions to download and launch MAT. These dumps are produced by IBM Java 8, so choose that download.
3. Click File } Open Heap Dump...
4. Open the *.dmp file that was downloaded.
5. In the Getting Started Wizard, choose Leak Suspects Report and click Finish

6. Review the leak suspects report which will highlight the cause of the OOME. For example:

   The class com.example.servlet.AllocateObject occupies 426,783,880 (91.14%) bytes. The memory is accumulated in one instance of java.lang.Object[] which occupies 426,779,160 (91.14%) bytes.