Ruby: Cannot Allocate Memory

Ruby: Cannot allocate memory

When Ruby calls fork the OS will make a copy of the entire parent processes address space, even if fork is only being called to exec another small process like ls. Momentarily, your system needs to be able to allocate a chunk of memory at least the size of the Ruby parent process before collapsing it down to what the child process actually needs.

So rails is generally quite memory hungry. Then if something uses fork, you need twice as much memory.

TL;DR Use posix-spawn instead of fork if you are in control of the code. Otherwise give your VM 1024MB or a bit of extra swap space to take up the slack for the fork call

Example Ruby Memory Usage withfork

Take a random VM, this one has swap space disabled:

$ free -m
             total       used       free     shared    buffers     cached
Mem:          1009        571        438          0          1         35
-/+ buffers/cache:        534        475
Swap:            0          0          0

Look at the Mem: row and free column. This is around about your size limit for a new process, in my case 438MiB

My buffers/cached have already been flushed for this test so that my free memory is at it's limit. You may need to take the buffers/cache values into account if they are large. Linux has the ability to evict stale cache when memory is needed by a process.

Use up some memory

Create a ruby process with a string around the size of your free memory. There is some overhead for the ruby process so it's not going to exactly match free.

$ ruby -e 'mb = 380; a="z"*mb*2**20; puts "=)"'
=)

Then make the string slightly larger:

$ ruby -e 'mb = 385; a="z"*mb*2**20; puts "=)"'
-e:1:in `*': failed to allocate memory (NoMemoryError)
        from -e:1:in `<main>'

Add a fork to the ruby process, reducing mb until it runs.

$ ruby -e 'mb = 195; a="z"*mb*2**20; fork; puts "=)"'
=)

A slightly larger fork process will produce the ENOMEM error:

$ ruby -e 'mb = 200; a="z"*mb*2**20; fork; puts "=)"'
-e:1:in `fork': Cannot allocate memory - fork(2) (Errno::ENOMEM)
        from -e:1:in `<main>'

Running a command with backticks launches that process with a fork so has the same outcome:

$ ruby -e 'mb = 200; a="z"*mb*2**20; `ls`'
-e:1:in ``': Cannot allocate memory - ls (Errno::ENOMEM)
        from -e:1:in `<main>'

So there you go, you need about twice the parent processes memory available on the system to fork a new process. MRI Ruby relies heavily on fork for it's multi process model, this is due to the design of Ruby which uses a global interpreter lock (GIL) that only allows one thread to execute at a time per ruby process.

I believe Python has a lot less use of fork internally. When you do use os.fork in Python, the same occurs though:

python -c 'a="c"*420*2**20;'
python -c 'import os; a="c"*200*2**20; os.fork()'

Oracle have a detailed article on the problem and talk about using the alternative of posix_spawn(). The article is directed at Solaris but this is a general POSIX Unix issue so applies to Linux (if not most Unices).

There is also a Ruby implementation of posix-spawn which you could use if you are in control of the code. This module doesn't replace anything in Rails, so it won't help you here unless you replaced the calls to fork yourself.

Getting error Cannot allocate memory for Rails

Both IO.popen and Kernel#system can be expensive operations in terms of memory because they both rely on fork(2). Fork(2) is a Unix system call which creates a child process that clones the parent's memory and resources. That means, if your parent process uses 500mb of memory, then your child would also use 500mb of memory. Each time you do Kernel#system or IO.popen you increase your application's memory usage by the amount of memory it takes to run your Rails app.

If your development machine has more RAM than your production server or if your production server produces a lot more output, there are two things you could do:

1. Increase memory for your production server.
2. Do some memory management using something like Resque.

You can use Resque to queue those operations as jobs. Resque will then spawn "workers"/child processes to get a job from the queue, work on it and then exit. Resque still forks, but the important thing is that the worker exits after working on the task so that frees up memory. There'll be a spike in memory every time a worker does a job, but it will go back to the baseline memory of your app every after it.

You might have to do both options above and look for other ways to minimize the memory-usage of your app.

Errno::ENOMEM: Cannot allocate memory - cat

So it seems that your system is running pretty low on memory and spawning a shell + calling cat is too much for the few memory left.

If you don't mind loosing some speed, you can merge the files in ruby, with small buffers.
This avoids spawning a shell, and you can control the buffer size.

This is untested but you get the idea :

buffer_size = 4096
output_file = File.open(final_output_file, 'w')

Dir["#{processing_directory}/*.csv"].sort_by {|file| [file.count("/"), file]}.each do |file|
  f = File.open(file)
  while buffer = f.read(buffer_size)
    output_file.write(buffer)
  end
  f.close
end

Cannot allocate memory for activeadmin in ubuntu 14.04 with DigitalOcean

I have found solution that I do need some more memory for bundle install. That's why I have created swap in harddisk with 512MB memory and it is working fine for me.

You can create swap from here.

Rails app Cannot allocate memory when saving photo using Paperclip

Have you checked your current memory usage on your server? This issue often occurs when there are only a little of memory left. As you said, this does not happen all the time so I think lack of memory is the main reason.

If you can not add more memory right away to your server, try adding a swap partition on your server can also be helpful.

---

Related Topics