Fork() Failing with Out of Memory Error

fork() is causing memory leak

This memory leak is related to the recursion. You do not close the directory handles further up the stack:

==4075==    by 0x4EEB902: opendir_tail (opendir.c:145)
==4075==    by 0x108977: dfsdirectory (in /home/alex/Desktop/main)
==4075==    by 0x108AE6: main (in /home/alex/Desktop/main)

==4075==    by 0x4EEB902: opendir_tail (opendir.c:145)
==4075==    by 0x108977: dfsdirectory (in /home/alex/Desktop/main)
==4075==    by 0x108A78: dfsdirectory (in /home/alex/Desktop/main)
==4075==    by 0x108AE6: main (in /home/alex/Desktop/main)

It is a leak of the still reachable kind, and these can be harmless, either because you call exit early (which applies here), or because they refer to data structures that are rooted in global variables. There is considerable disagreement among C developers whether these leaks need to be addressed, or whether it is acceptable to leave them in a program.

Please also note that the EINTR check for closedir is wrong with glibc. It introduces a double-free vulnerability in case of an EINTR failure.

fork failures: Cannot allocate memory

To check for memory leaks you might like to run the program under Valgrind: http://valgrind.org

---

To get/set limits from the console/shell there is the ulimit command available.

From inside the program the system calls getrlimit()/setrlimit() provide this functionality.

---

Another workaround for situations where memory might get tight due to fork()ing would be to use vfork() immediately followed by a call to a member of the exec*() family of functions.

From man vfork:

vfork() is a special case of clone(2). It is used to create new processes without copying the page tables of the parent process. It may be useful in performance-sensitive applications where a child is created which then immediately issues an execve(2).

vfork() differs from fork(2) in that the parent is suspended until the child terminates (either normally, by calling _exit(2), or abnormally, after delivery of a fatal signal), or it makes a call to execve(2). Until that
point, the child shares all memory with its parent, including the stack. The child must not return from the current function or call exit(3), but may call _exit(2).

What might be possible reasons that I get Cannot Allocate Memory error on forking()?

Change your first if branch to this:

if((pid=fork())<0){
 perror("forking failed");
 exit(0);
}

This will tell you what went wrong.

R multicore mcfork(): Unable to fork: Cannot allocate memory

The issue might be exactly what the error message suggests: there isn't enough memory to fork and create parallel processes.

R essentially needs to create a copy of everything that's in memory for each individual process (to my knowledge it doesn't utilize shared memory). If you are already using 51% of your RAM with a single process, then you don't have enough memory to create a second process since that would required 102% of your RAM in total.

Try:

1. Using fewer cores - If you were trying to use 4 cores, it's possible you have enough RAM to support 3 parallel threads, but not 4. registerDoMC(2), for example, will set the number of parallel threads to 2 (if you are using the doMC parallel backend).
2. Using less memory - without seeing the rest of your code, it's hard to suggest ways to accomplish this. One thing that might help is figuring out which R objects are taking up all the memory (Determining memory usage of objects?) and then removing any objects from memory that you don't need (rm(my_big_object))
3. Adding more RAM - if all else fails, throw hardware at it so you have more capacity.
4. Sticking to single threading - multithreaded processing in R is a tradeoff of CPU and memory. It sounds like in this case you may not have enough memory to support the CPU power you have, so the best course of action might be to just stick to a single core.

proc_open(): fork failed - Out of memory

Try this:
https://getcomposer.org/doc/articles/troubleshooting.md#proc-open-fork-failed-errors

/bin/dd if=/dev/zero of=/var/swap.1 bs=1M count=1024 
/sbin/mkswap /var/swap.1 
/sbin/swapon /var/swap.1

---

Related Topics