What Does Anon-Rss and Total-Vm Mean

Why my process has been killed?

You are victim of the Linux OOM Killer.

You can tune the way that the OOM killer handles OOM conditions with certain processes. for example, your my_proj process 3915 that was killed earlier.
If you want it not to be killed by the OOM killer, you can :

echo -15 > /proc/3915/oom_adj

Out of memory: kill process

It's happening because your server is running out of memory. To solve this problem you have 2 options.

1. Update your Server's Ram or use SWAP (But upgrading Physical ram is recommended instead of using SWAP)

2. Limit Nginx ram use.

To limit nginx ram use open the /etc/nginx/nginx.conf file and add client_max_body_size <your_value_here> under the http configuration block. For example:

worker_processes 1;
http {
    client_max_body_size 10M;
    ...
}

Note: use M for MB, G for GB and T for TB

Process using more memory then container itslef

Two pieces. First what you see in the metrics is probably the working set size, which does not include buffers while I think the oom_killer shows rss which does. But more importantly, the data in metrics output is sampled, usually every 30 seconds. So if the memory usage spiked suddenly, or even if it just tried to allocate one huge buffer, then it would be killed.

Trying to locate a leak! What does anon mean for pmap?

Anon blocks are "large" blocks allocated via malloc or mmap -- see the manpages. As such, they have nothing to do with the Java heap (other than the fact that the entire heap should be stored in just such a block).

In my experience, thread stacks also use anon blocks. If you see a lot of anon blocks that all have the same size, and that size is 512k to 4Mb (the example below is repeated over a dozen times for a Tomcat process that I have running), that's the likely cause. Depending on the program, you may have up to a few dozen of these; if you're seeing thousands, it means you have a problem with threading.

b089f000    504K rwx--    [ anon ]
b091d000     12K -----    [ anon ]
b0920000    504K rwx--    [ anon ]
b099e000     12K -----    [ anon ]
b09a1000    504K rwx--    [ anon ]
b0a1f000     12K -----    [ anon ]

But that leaves a question: why are you using pmap to diagnose a Java memory issue?

---

Related Topics