How to Know Whether Enough Memory Is Free to Deploy a New Application on a Linux MAChine

How to know whether enough memory is free to deploy a new application on a Linux machine?

(It's probably a bit late for the OP, but this is asked quite often, so I'll give it a shot)

free normally shows something like this:

             total       used       free     shared    buffers     cached
Mem:       8195284    8137708      57576          0    1232328    2651156
-/+ buffers/cache:    4254224    3941060
Swap:     18892216     759852   18132364

People tend to look at the Mem: line when trying to find out how much free memory they have. Unfortunately that line is quite misleading, because the Linux kernel tries to make optimal use of the available memory in (at least) these ways:

• It will cache data from the I/O subsystem (e.g. the disk), so that it will be readily available if needed.

• It will actively evict processes that have been inactive for some time to the swap space, in favour of caching data for active processes. This tends to favour throughput over responsiveness, so some people tune their kernel to change this behaviour.

The first point is the source of confusion regarding free, because the Mem: line includes the memory used for caching in the used memory amount. The kernel, though, will cache as much as possible for performance reasons. In fact, on any Linux system that has been up for some time, the free memory tends to be close to zero - unused memory is wasted memory.

The cache memory, though, can be freed by the kernel if needed by another process. While it will impact I/O performance to a degree, other processes can have more memory without using the swap space. Therefore, for most intents and purposes, that memory is free.

That's why free includes a second line, where the cache memory is considered free:

-/+ buffers/cache:    4254224    3941060

This second line is what people should be looking at when they want to know if they have enough free memory for a certain purpose.

In the example above, according to the Mem: line there are ~57 MB of free memory. If one reads the second line, though, there are in fact about 3.9 GB that can be used without forcing active processes to swap. As a sidenote, there are also about 760 MB of rarely-used data that have been swapped out, to make more space in the main memory for processes and caching.

At roughly the same time, the contents of /proc/meminfo:

MemTotal:        8195284 kB
MemFree:           57660 kB
Buffers:         1232352 kB
Cached:          2651156 kB
SwapCached:       119936 kB
.
.
.

MemTotal: the available physical memory detected by the kernel.

MemFree: the unused physical memory - the free memory shown in the Mem: line of free.

Buffers: relatively temporary storage of raw disk blocks.

Cached: in-memory cache for files read from the disk. It does not include SwapCached memory.

SwapCached: memory that was once swapped out, then swapped back in but is still in the swap space. If needed, its contents can be just discarded (very fast!), without having to swap them out (slower).

So, to have a semi-accurate estimate of the memory that is actually available

MemFree + Buffers + Cached + SwapCached

is a good starting point - and the one free shows in that second line.

Naturally, memory management and the related statistics and measurements are more complicated than this. The numbers shown by free are mere estimates at best, since there are a lot of other variables to take into account if you want to go deeper. For people who regularly perform memory usage optimization, this is almost a form of art.

EDIT:

A somewhat humorous link about this "issue":

http://www.linuxatemyram.com/

EDIT 2:

To confirm the comment about memory use analysis almost being a form of art:

Even free misses a major chunk of cached data on modern Linux systems. From /proc/meminfo on my system:

SReclaimable:    2253576 kB

That's about 2GB of memory that is used by the system slab allocator for caching directory entries and such and it is reclaimable (i.e. it can be cleared and used by processes if necessary). Yet free does not consider it cache memory and does not enter it in any of its calculations and therefore it shows up as used memory.

The slabtop utility, if available, allows the system administrator to find out what the slab cache is used for.

A way (for the root user only) to have free show the actual memory use of the system is the following:

# swapoff -a
# sync
# echo 3 > /proc/sys/vm/drop_caches 
# free
             total       used       free     shared    buffers     cached
Mem:       8195284    3181468    5013816          0       8656     228832
-/+ buffers/cache:    2943980    5251304
Swap:            0          0          0
# swapon -a

The first command disables the swap space. It should not be issued if the available memory may not be enough to hold the data that have been swapped out - in that case one has to take into account the Swap: line of free in their memory usage calculations.

The second command pushes all buffered data to the disk. It allows more cache memory to be freed in the next step.

The third command is the most important of the set - it forces the kernel to discard as much cached data as possible (page cache, directory entries, inodes etc).

Then free finally shows what the running processes actually use in its -/+ buffers/cache: line. It is quite noticeable that even after dropping all cached data the kernel quickly starts caching again - in this case it has already reached almost 250MB of cached data within a few seconds.

The final command enables the swap space again - it is only necessary if the first command was used too.

It should be noted that these commands should be executed by the root user in order to have the necessary privileges.

who eat my memory?

From my system:

$ grep ^S[^wh] /proc/meminfo 
Slab:            4707412 kB
SReclaimable:    4602900 kB
SUnreclaim:       104512 kB

These three metrics are data structures held by the slab alocator. While SUnreclaimable is, well, unreclaimable, SReclaimable is just like any other cache in the system - it will be made available to processes under memory pressure. Unfortunately free does not seem to take it into account, as mentioned in detail in this older answer of mine, and this part of memory can easily grow to several GB of memory...

If you really want to see how much memory your processes are using you could try going through the cache-emptying procedure described in my post - you can skip the swap-related parts, since your system does not appear to be using any swap memory anyway.

Reclaim memory after program exit

Yes, memory used by your program is freed after your program exits.

The statistics in "free" are confusing, but the fact is that the memory IS available to other programs:

http://kevinclosson.wordpress.com/2009/11/17/linux-free-memory-is-it-free-or-reclaimable-yes-when-i-want-free-memory-i-want-free-memory/

http://sourcefrog.net/weblog/software/linux-kernel/free-mem.html

Here's an event better link:

http://www.linuxatemyram.com/

How to find user memory usage in linux

Per-user memory usage in percent using standard tools:

for _user in $(ps haux | awk '{print $1}' | sort -u)
do
    ps haux | awk -v user=${_user} '$1 ~ user { sum += $4} END { print user, sum; }'            
done

or for more precision:

TOTAL=$(free | awk '/Mem:/ { print $2 }')
for _user in $(ps haux | awk '{print $1}' | sort -u)
do
    ps hux -U ${_user} | awk -v user=${_user} -v total=$TOTAL '{ sum += $6 } END { printf "%s %.2f\n", user, sum / total * 100; }'
done

The first version just sums up the memory percentage for each process as reported by ps. The second version sums up the memory in bytes instead and calculates the total percentage afterwards, thus leading to a higher precision.

How to see top processes sorted by actual memory usage?

First, repeat this mantra for a little while: "unused memory is wasted memory". The Linux kernel keeps around huge amounts of file metadata and files that were requested, until something that looks more important pushes that data out. It's why you can run:

find /home -type f -name '*.mp3'
find /home -type f -name '*.aac'

and have the second find instance run at ridiculous speed.

Linux only leaves a little bit of memory 'free' to handle spikes in memory usage without too much effort.

Second, you want to find the processes that are eating all your memory; in top use the M command to sort by memory use. Feel free to ignore the VIRT column, that just tells you how much virtual memory has been allocated, not how much memory the process is using. RES reports how much memory is resident, or currently in ram (as opposed to swapped to disk or never actually allocated in the first place, despite being requested).

But, since RES will count e.g. /lib/libc.so.6 memory once for nearly every process, it isn't exactly an awesome measure of how much memory a process is using. The SHR column reports how much memory is shared with other processes, but there is no guarantee that another process is actually sharing -- it could be sharable, just no one else wants to share.

The smem tool is designed to help users better gage just how much memory should really be blamed on each individual process. It does some clever work to figure out what is really unique, what is shared, and proportionally tallies the shared memory to the processes sharing it. smem may help you understand where your memory is going better than top will, but top is an excellent first tool.

Using LINUX top command to compute used memory percentage

You have a few problems here.

First, this doesn't do what you want it to do.

USEDMEM=/home/modadm/top-output.txt | grep "Mem" | cut -c 25-31

You can't pipe a filename into a command. You actually want to pipe the contents of the file into the command. You can do that with 'cat'. However, grep is actually designed to search within a file so you can do

USEDMEM=$(grep "Mem" /home/modadm/top-output.txt | cut -c 25-31)

Note that $(cmd) is how you execute a command in a subshell. i.e., you can run some commands to compute the value of a variable in your script. You can also use `cmd` (backticks; usually on the tilde key) but that syntax is less clear.

Again, you probably want to calculate this result in a subshell. Also, don't use $ when assigning to variables.

$USEDPCT='echo $USEDMEM / $MAXMEM * 100 | bc'

This can be rewritten as

USEDPCT=$(echo "scale=3; $USEDMEM / $MAXMEM * 100" | bc)

Finally, you want to pipe the contents of the variable into the mail program. The pipe is expecting a program to be on the left hand side. You do this by echo'ing the value of the variable into the pipe.

echo "$USEDPCT" | mail -s "Test Email from MOD Server" test@test.com

To put everything back together:

#!/bin/bash
top -n 1 -b | grep "Mem" > /home/modadm/top-output.txt
MAXMEM=$(grep "Mem" /home/modadm/top-output.txt | cut -c 7-14)
USEDMEM=$(grep "Mem" /home/modadm/top-output.txt | cut -c 25-31)
USEDPCT=$(echo "$USEDMEM / $MAXMEM * 100" | bc -l)
echo "$USEDPCT" | mail -s "Test Email from MOD Server" test@test.com

---

Related Topics