Trying to Locate a Leak! What Does Anon Mean for Pmap

Trying to locate a leak! What does anon mean for pmap?

Anon blocks are "large" blocks allocated via malloc or mmap -- see the manpages. As such, they have nothing to do with the Java heap (other than the fact that the entire heap should be stored in just such a block).

In my experience, thread stacks also use anon blocks. If you see a lot of anon blocks that all have the same size, and that size is 512k to 4Mb (the example below is repeated over a dozen times for a Tomcat process that I have running), that's the likely cause. Depending on the program, you may have up to a few dozen of these; if you're seeing thousands, it means you have a problem with threading.

b089f000    504K rwx--    [ anon ]
b091d000     12K -----    [ anon ]
b0920000    504K rwx--    [ anon ]
b099e000     12K -----    [ anon ]
b09a1000    504K rwx--    [ anon ]
b0a1f000     12K -----    [ anon ]

But that leaves a question: why are you using pmap to diagnose a Java memory issue?

Using pmap and gdb to find native memory leak

A very basic approach: you could try looking at who is calling mmap (and not munmap).

• attach to the process
• set breakpoint on mmap, with commands to print arguments and backtrace (maybe 5 frames) and continue
• similar thing for munmap
• redirect output
• let it run for a day
• detach
• match mmaps with munmaps in the output

With pmap periodically running on the side, you may be able to match newer anon regions with mmap backtraces (might need playing around with frame count).

---

There is already this nice little article LINUX GDB: IDENTIFY MEMORY LEAKS to get you started.

Note:

• you are looking for mmap and munmap, not malloc and free
• you will have to find out the offset of the return from mmap
• I have not tried the script from the article but I think it would do what the article claims

Finding mmap return instruction offset (from start of mmap):
Just fire up gdb with any executable on the same host

[ aquila ~ ] $ gdb -q /usr/bin/ls
Reading symbols from /usr/bin/ls...Reading symbols from /usr/bin/ls...(no debugging symbols found)...done
.
(no debugging symbols found)...done.
Missing separate debuginfos, use: dnf debuginfo-install coreutils-8.27-5.fc26.x86_64
(gdb) set pagination off
(gdb) set breakpoint pending on
(gdb) b mmap
Function "mmap" not defined.
Breakpoint 1 (mmap) pending.
(gdb) r
Starting program: /usr/bin/ls

Breakpoint 1, 0x00007ffff7df2940 in mmap64 () from /lib64/ld-linux-x86-64.so.2
(gdb) disassemble
Dump of assembler code for function mmap64:
=> 0x00007ffff7df2940 <+0>:     test   %rdi,%rdi
   0x00007ffff7df2943 <+3>:     push   %r15
   0x00007ffff7df2945 <+5>:     mov    %r9,%r15
    :
    :
   0x00007ffff7df2973 <+51>:    mov    $0x9,%eax
    :
   0x00007ffff7df2982 <+66>:    pop    %rbx
    :
   0x00007ffff7df298a <+74>:    pop    %r15
   0x00007ffff7df298c <+76>:    retq
   0x00007ffff7df298d <+77>:    nopl   (%rax)
    :
    :
   0x00007ffff7df29d8 <+152>:   mov    $0xffffffffffffffff,%rax
   0x00007ffff7df29df <+159>:   jmp    0x7ffff7df2982 <mmap64+66>
End of assembler dump.

Note the return instruction here:

0x00007ffff7df298c <+76>:    retq

So, on my machine, the second breakpoint would have to be set at (mmap+76).

Once you determine this offset, you can verify this offset by attaching to your target process and disassembling what is at that offset. E.g. taking my current shell as my target process:

[ aquila ~ ] $ echo $$
9769
[ aquila ~ ] $ gdb -q
(gdb) attach 9769
Attaching to process 9769
Reading symbols from /usr/bin/bash...Reading symbols from /usr/bin/bash...(no debugging symbols found)..
.done.
(no debugging symbols found)...done.
Reading symbols from /lib64/libtinfo.so.6...Reading symbols from /lib64/libtinfo.so.6...(no debugging sy
mbols found)...done.
(no debugging symbols found)...done.
Reading symbols from /lib64/libdl.so.2...(no debugging symbols found)...done.
Reading symbols from /lib64/libc.so.6...(no debugging symbols found)...done.
Reading symbols from /lib64/ld-linux-x86-64.so.2...(no debugging symbols found)...done.
Reading symbols from /lib64/libnss_files.so.2...(no debugging symbols found)...done.
0x00007fcfc67cc18a in waitpid () from /lib64/libc.so.6
Missing separate debuginfos, use: dnf debuginfo-install bash-4.4.12-5.fc26.x86_64
(gdb) x/i mmap+76
   0x7fcfc680375c <mmap64+76>:  retq

I'm not very sure hbreak is required, plain old break might work as well.

Huge Memory allocated outside of Java Heap

Figured out a solution but the reason is still in black box. After adding -XX:MaxDirectMemorySize=1024m, it works.

Seems like java GC for Memory Outside Heap (MOH) is not as smart as heap gc.

The guessed reason is, java gc for MOH is triggered when the MOH usage is reached some level, maybe 50%. So before that, it will only mark the memory as freed but not free it actually. Even the system has very little memory.

By default, the XX:MaxDirectMemorySize is big and even usage is 50%, system will down.

So after add this limitation, the threshold is easy to reach and MOH is freed in time.

So 1024m works for my application. It is big enough for application to run and not too big to make the system crash.

Monitoring Valgrind with pmap

Depending on the tool, valgrind can use a lot more memory than a native run.
Even the "none" tool uses more memory.

Various valgrind tools can do a detailed monitoring/reporting of the memory used by your application.

You might e.g. use the memcheck tool with --xtree-memory=full and visualise the resulting file with kcachegrind.
See e.g. https://www.valgrind.org/docs/manual/manual-core.html#manual-core.xtree for more details.

The massif tool can be used to report peaks of memory usages.

Thread Management Memory Leak

I now use boost::shared_ptr <boost::thread> thr_ev_21; and

if (thr_ev_21.get()) thr_ev_21->interrupt();
thr_ev_21.reset(new boost::thread (boost::bind(&porter::write_units_court_a, this)));

Now my memory usage plateaus instead of constantly growing. The thread object must not be destroyed with this line on the second pass:

tvec.at(21) = boost::thread (boost::bind(&my_class::write_units_court_a, this));

Java still uses system memory after deallocation of objects and garbage collection

Many JVMs never return memory to the operating system. Whether it does so or not is implementation-specific. For those that don't, the memory limits specified at startup, usually through the -Xmx flag, are the primary means to reserve memory for other applications.

I am having a hard time finding documentation on this subject, but the garbage collector documentation for Sun's Java 5 does address this, suggesting that under the right conditions, the heap will shrink if the correct collector is used—by default, if more that 70% of the heap is free, it will shrink so that only 40% is free. The command line options to control these are -XX:MinHeapFreeRatio and -XX:MaxHeapFreeRatio.

---

Related Topics