Why The Process Is Getting Killed at 4Gb

Why the process is getting killed at 4GB?

Check with file and ldd that your executable is indeed 64 bits.

Check also the resource limits. From inside the process, you could use getrlimit system call (and setrlimit to change them, when possible). From a bash shell, try ulimit -a. From a zsh shell try limit.

Check also that your process indeed eats the memory you believe it does consume. If its pid is 1234 you could try pmap 1234. From inside the process you could read the /proc/self/maps or /proc/1234/maps (which you can read from a terminal). There is also the /proc/self/smaps or /proc/1234/smaps and /proc/self/status or /proc/1234/status and other files inside your /proc/self/ ...

Check with  free that you got the memory (and the swap space) you believe. You can add some temporary swap space with swapon /tmp/someswapfile (and use mkswap to initialize it).

I was routinely able, a few months (and a couple of years) ago, to run a 7Gb process (a huge cc1 compilation), under Gnu/Linux/Debian/Sid/AMD64, on a machine with 8Gb RAM.

And you could try with a tiny test program, which e.g. allocates with malloc several memory chunks of e.g. 32Mb each. Don't forget to write some bytes inside (at least at each megabyte).

standard C++ containers like std::map or std::vector are rumored to consume more memory than what we usually think.

Buy more RAM if needed. It is quite cheap these days.

What killed my process and why?

If the user or sysadmin did not kill the program the kernel may have. The kernel would only kill a process under exceptional circumstances such as extreme resource starvation (think mem+swap exhaustion).

Application is getting killed without any reason. Suspecting high BSS. How to debug it?

It seems that problem really is huge BSS size.
I have asked you to show output of LD_DEBUG=all /lib64/ld-linux-x86-64.so.2 /path/to/exe in comments.

/lib64/ld-linux-x86-64.so.2 is runtime linker which is used by OS to load your binary in process memory during execve system call. Runtime linker is responsible for parsing executable format, loading all sections and dependencies in memory, performing all required relocations and so on.
Setting environment variable LD_DEBUG to all we instruct runtime linker to generate debug output.

[root@localhost PktBlaster]# LD_DEBUG=all /lib64/ld-linux-x86-64.so.2
/root/Veryx/PktBlaster/PktBlaster
851: file=/root/Veryx/PktBlaster/PktBlaster [0]; generating link map
/root/Veryx/PktBlaster/PktBlaster: error while loading shared
libraries: /root/Veryx/PktBlaster/PktBlaster: cannot map zero-fill
pages: Cannot allocate memory

Searching for this error message in source code of runtime linker(glibc-2.17 elf/dl-load.c, lines ~1400) we see:

1393         if (zeroend > zeropage)
1394           {
1395         /* Map the remaining zero pages in from the zero fill FD.  */
1396         caddr_t mapat;
1397         mapat = __mmap ((caddr_t) zeropage, zeroend - zeropage,
1398                 c->prot, MAP_ANON|MAP_PRIVATE|MAP_FIXED,
1399                 -1, 0);
1400         if (__builtin_expect (mapat == MAP_FAILED, 0))
1401           {
1402             errstring = N_("cannot map zero-fill pages");
1403             goto call_lose_errno;
1404           }

dl-loader is in process of loading BSS segment, which by optimizations is stored in binary format as just number of bytes, that must be initialized to zero. Loader tries to allocate through mmap zero initialized memory block(MAP_ANONYMOUS) and get error from the OS:

 15 #define ENOMEM      12  /* Out of memory */

From man 2 mmap:

ENOMEM No memory is available, or the process's maximum number of
mappings would have been exceeded.

So it seems that for whatever reason OS cannot fulfill loader request for memory. Either some limits are used(systemd, process limit, some security LKM, whatever) or simply there are not enough free memory in kernel.

To determine what object file generates most part of the BSS - use

objdump -j '.bss' -t *.o 

Perl script execution keeps getting killed - running out of memory

You are trying to allocate an array of 90,000,000 elements. Perl, due to its flexible typing and other advanced variable features, uses a lot more memory for this than you would expect.

On my (Windows 7) machine, a program that just allocates such an array and does nothing else eats up 3.5 GB of RAM.

There are various ways to avoid this huge memory usage. Here are a couple:

The PDL module for scientific data processing, which is designed to efficiently store huge numeric arrays in memory. This will change the syntax for allocating and using the array, though (and it messes around with Perl's syntax in various other ways).

DBM::Deep is a module that allocates a database in a file--and then lets you access that database through a normal array or hash:

use DBM::Deep;
my @array;
my $db = tie @array, "DBM::Deep", "array.db";

#Now you can use @array like a normal array, but it will be stored in a database.

Return code when OS kills your process

A process' return status (as returned by wait, waitpid and system) contains more or less the following:

• Exit code, only applies if process terminated normally
• whether normal/abnormal termination occured
• Termination signal, only applies if process was terminated by a signal

The exit code is utterly meaningless if your process was killed by the OOM killer (which will apparently send you a SIGKILL signal)

for more information, see the man page for the wait command.

Java process killed due to out of memory after running job in container for an hour

We had this problem. It is a limit in CircleCI (or the VM, really). Only solution is to make your app use less memory.

---

Related Topics