A Java Heap Dump is a snapshot of all the objects that are in memory in the JVM at a certain moment. Typically, a heap dump is created at the moment a Java application crashes because it runs out of memory. The heap dump can then show you what the application was doing in its dying moments, providing insight into potential memory leaks.

By Joep Weijers  | TOPdesk

Likewise, you can also extract a heap dump from a running JVM. This is a useful technique to peak under the hood of a service that is running with abnormal memory usage, but that is not running out of memory (yet). For us at TOPdesk, this is a very valuable tool to investigate performance issues.

The JVM supplies the Attach API to allow external tools to attach to the JVM. Profiling tools like jmap, jcmd and JVisualVM use that API to monitor and troubleshoot the Java process inside JVM.

Unfortunately, we can’t use any of these tools in our containers. JVisualVM is the user friendliest, as it provides a simple graphical user interface, but is not usable from within our headless Docker containers. jmap is not an officially supported tool and is not part of the OpenJ9 distribution. And jcmd is only available in the JDK variant of OpenJ9, not in the JRE.

So we need a way to make the jcmd tool available in an already running OpenJ9 JRE container.

Attempt 1: Using a Kubernetes Ephemeral Debug Container

Kubernetes 1.16 introduces an alpha feature called Ephemeral Containers: a special type of container that runs temporarily in an existing Pod to accomplish user-initiated actions such as troubleshooting.

Could we use such an ephemeral container containing the OpenJ9 JDK to attach to a container running a service on OpenJ9 JRE?

In this example we will use a Docker container running a simple Hello World webserver: topdesk/example-openj9-web-service:1.0.0. You can find the code of this example service in the GitHub repository: https://github.com/TOPdesk/example-openj9-web-service.

Starting the web service container

Let’s start by firing up a minikube cluster. Since ephemeral containers are still an alpha version the EphemeralContainers feature gate has to be enabled:

Now we are going to start a pod with our example-openj9-web-service:

If we exec into the pod, we can verify the Java version is indeed OpenJ9 JRE. And we also see that jcmd is not present in this image:

Starting the ephemeral debug container

Now we are going to attach an ephemeral container to the example-openj9-web-service container. We use the OpenJ9 JDK image, are targeting the process namespace of the other container and start the sh command:

We can use ps to list all running processes and see the web service of the example-openj9-web-service container running:

The additional tools of the JDK are now at our disposal. Let’s use jcmd to create a heap dump of our Java process with PID 7:

It seems we can’t connect to the JVM. Let’s use jps to list all the running JVMs:

That is odd, we only see the tool itself running. So we seem unable to reach the JVM in the container we are attached to. Are we doing something wrong here?

Attempt 2: Switching user in the ephemeral debug container

According to the jps documentation “the tool shows information for every Java process that is owned by the current user ID on the current host”. We are currently running as root, but the Java process is running as user 999. Maybe it works if we create a user 999 and run jps under that user?

Unfortunately, we still can’t connect to the JVM running in the example-openj9-web-service container. I am not sure whether that is caused by something in Kubernetes’ process sharing between the main container and debug container, or maybe the JDK tooling can’t handle this situation.

Attempt 3: Downloading the JDK into the container

A completely different approach is to copy the debug tools into your container. For instance, by downloading the JDK into the running web service container. After downloading and unzipping, you can use jps and jcmd directly:

Success! We have our heap dump. However, we also polluted our container with a JDK. We can of course remove the JDK files afterwards. But the concept of ephemeral debug containers, that disappear when you are done debugging, is much more appealing. If you know of a way to do heap dumps on running containers using ephemeral debug containers, please reach out!