Get current ruby process memory usage
When trying to solve this problem a year ago, I did a lot of online research and API digging and was only able to solve it via a system call to ps.
In both OS X 10.7.2 and Red Hat 4.1.2-13 (on EC2):
pid, size = `ps ax -o pid,rss | grep -E "^[[:space:]]*#{$$}"`.strip.split.map(&:to_i)
This fetches and places the resident memory size of the process in kilobytes into the size variable.
With a little effort this could be cleaned up, but most of the time is spend calling ps and capturing its output, so I don't think it is worth the time.
A good way to get memory usage of a Ruby process in Linux from within itself?
You could use the proc-wait3 library, which adds Process.getrusage
. It would save you implementing the low-level stuff yourself, but if you only want memory usage, including a library might be overkill, especially given that hasn't been updated in a while.
Ruby process memory structure
It is likely that your application is allocating objects that are then groomed by the Garbage Collector. You can check this with a call to GC.stat
Ruby does not release memory back to the operating system in any meaningful way. (if you're running MRI) Consequently, if you allocate 18GB of memory and 15GB gets garbage collected, you'll end up with your ~3GB of heap data.
The Ruby MRI GC is not a compacting garbage collector, so as long as there is any data in the heap the heap will not be released. This leads to memory fragmentation and the values that you see in your app.
Getting memory usage of my process from OSX using Ruby
Referring to this answer it seems like you need to call proc_pidinfo()
. I don't think there's a Ruby equivalent, so either you'll have to write a C-extension or use the ruby-ffi gem.
Other sources indicate Ruby 1.9.2 ships with a built in FFI -- but that version is not delivered with OS X.
Finding the cause of a memory leak in Ruby
It looks like you are entering The Lost World here. I don’t think the problem is with c-bindings in racc
either.
Ruby memory management is both elegant and cumbersome. It stores objects (named RVALUE
s) in so-called heaps of size of approx 16KB. On a low level, RVALUE
is a c-struct, containing a union
of different standard ruby object representations.
So, heaps store RVALUE
objects, which size is not more than 40 bytes. For such objects as String
, Array
, Hash
etc. this means that small objects can fit in the heap, but as soon as they reach a threshold, an extra memory outside of the Ruby heaps will be allocated.
This extra memory is flexible; is will be freed as soon as an object became GC’ed. That’s why your testcase with big_string
shows the memory up-down behaviour:
def report
puts 'Memory ' + `ps ax -o pid,rss | grep -E "^[[:space:]]*#{$$}"`
.strip.split.map(&:to_i)[1].to_s + 'KB'
end
report
big_var = " " * 10000000
report
big_var = nil
report
ObjectSpace.garbage_collect
sleep 1
report
# ⇒ Memory 11788KB
# ⇒ Memory 65188KB
# ⇒ Memory 65188KB
# ⇒ Memory 11788KB
But the heaps (see GC[:heap_length]
) themselves are not released back to OS, once acquired. Look, I’ll make a humdrum change to your testcase:
- big_var = " " * 10000000
+ big_var = 1_000_000.times.map(&:to_s)
And, voilá:
# ⇒ Memory 11788KB
# ⇒ Memory 65188KB
# ⇒ Memory 65188KB
# ⇒ Memory 57448KB
The memory is not released back to OS anymore, because each element of the array I introduced suits the RVALUE
size and is stored in the ruby heap.
If you’ll examine the output of GC.stat
after the GC was run, you’ll find that GC[:heap_used]
value is decreased as expected. Ruby now has a lot of empty heaps, ready.
The summing up: I don’t think, the c
code leaks. I think the problem is within base64 representation of huge image in your css
. I have no clue, what’s happening inside parser, but it looks like the huge string forces the ruby heap count to increase.
Hope it helps.
ruby/ruby on rails memory leak detection
Some tips to find memory leaks in Rails:
- use the Bleak House plugin
- implement Scout monitoring specifically the memory usage profiler
- try another simple memory usage logger
The first is a graphical exploration of memory usage by objects in the ObjectSpace.
The last two will help you identify specific usage patterns that are inflating memory usage, and you can work from there.
As for specific coding-patterns, from experience you have to watch anything that's dealing with file io, image processing, working with massive strings and the like.
I would check whether you are using the most appropriate XML library - ReXML is known to be slow and believed to be leaky (I have no proof of that!). Also check whether you can memoize expensive operations.
Related Topics
In Rails - Is There a Rails Method to Convert Newlines to <Br>
Problems with the Rails Console, Rvm and Readline
Ruby Hash Default Value Behavior
How to Find the Namespace/Module Name Programmatically in Ruby on Rails
Colon (:) Appears as Forward Slash (/) When Creating File Name
How to Require Active Record Working Outside of Rails
Warning: Constant ::Fixnum Is Deprecated When Generating New Model
How to Set Default Ruby Version with Rvm
How to Generate a Human Readable Time Range Using Ruby on Rails
Why Should One Use a Http Server in Front of a Framework Web Server
How to Check to See If My Array Includes an Object
How to Compare Strings Ignoring the Case