Interpreting Jemaloc Data Possible Off-Heap Leak

Spring boot application not using jemalloc

When you run sudo, Java gets root environment which does not have your previously exported LD_PRELOAD and MALLOC_CONF.

Try

sudo LD_PRELOAD=/usr/local/lib/libjemalloc.so \
     MALLOC_CONF=prof:true,lg_prof_interval:30,lg_prof_sample:17 \
     java -jar application.jar

BTW, jemalloc is not always useful for profiling Java applications, since it cannot show Java stacks (but it is useful for preventing memory leaks caused by the standard allocator).

Try async-profiler as described in this answer.

Also check this post about Java native memory consumption.

Understanding Jeprofile output

1. In your jemalloc output, JVM_FindSignal is accounted for 97% of allocations. This can't be true, since JVM_FindSignal does not allocate anything.

   This must be the result of the issue that JVM has no debug symbols. Install the package with JDK debug symbols as descibed in this answer, or use JDK with built-in debug symbols, e.g. Liberica JDK.

2. jemalloc does not know anything about Java methods. It can't translate the addresses in JIT compiled code to Java method names. Hence so many addresses (hex numbers) in your jemalloc output.

   There is a Java profiler, async-profiler, that can trace native memory allocations down to Java code, and show Java stack traces as Flame Graphs. Profiling malloc, mprotect and mmap calls with async-profiler can be helpful in finding native memory leaks. See this answer for details.

   There is a presentation video showing an example of profiling native allocations with both jemalloc and async-profiler.

jemalloc and JVM_FindSignal

It turns out that

1. openjdk-8-dbg package installs files with debug symbols into /usr/lib/debug/.build-id
2. jeprof looks for debug symbols in /usr/lib/debug/{FULL_SO_PATH}

So, it's a combination of a bug in jeprof that does not parse .note.gnu.build-id section, and a problem of the dbg package that does not include full path symlinks to debug libraries.

To work around this, you may create the corresponding symlink manually:

ln -s /usr/lib/debug/.build-id/16/240e0172c3fc0dd6e974325c8ad1d93723ccac.debug /usr/lib/debug/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/amd64/server/libjvm.so

(the file names are taken from gdb output)

---

However, even when jeprof is able to read debug symbols, this doesn't often help in case of Java applications. The problem is that jemalloc doesn't know how to unwind Java stacks. See this presentation for an example.

Consider trying async-profiler that can show mixed Java+native stacks. Profiling malloc, mprotect and mmap calls with async-profiler can be helpful in finding native memory leaks in Java applications. See this answer for details.

How tp choose the way java allocates memory?

jcmd VM.Native_memory shows up the size that is much smaller than the RSS

I would look at /proc/{pid}/smap to see which regions are resident. You can have a lot of virtual memory used by threads stacks which are typically not resident.

If the java program has allocated a lot of off heap memory you will have DirectByteBuffer objects to wrap each memory region.

---

Related Topics