How to See Top Processes Sorted by Actual Memory Usage

How to see top processes sorted by actual memory usage?

First, repeat this mantra for a little while: "unused memory is wasted memory". The Linux kernel keeps around huge amounts of file metadata and files that were requested, until something that looks more important pushes that data out. It's why you can run:

find /home -type f -name '*.mp3'
find /home -type f -name '*.aac'

and have the second find instance run at ridiculous speed.

Linux only leaves a little bit of memory 'free' to handle spikes in memory usage without too much effort.

Second, you want to find the processes that are eating all your memory; in top use the M command to sort by memory use. Feel free to ignore the VIRT column, that just tells you how much virtual memory has been allocated, not how much memory the process is using. RES reports how much memory is resident, or currently in ram (as opposed to swapped to disk or never actually allocated in the first place, despite being requested).

But, since RES will count e.g. /lib/libc.so.6 memory once for nearly every process, it isn't exactly an awesome measure of how much memory a process is using. The SHR column reports how much memory is shared with other processes, but there is no guarantee that another process is actually sharing -- it could be sharable, just no one else wants to share.

The smem tool is designed to help users better gage just how much memory should really be blamed on each individual process. It does some clever work to figure out what is really unique, what is shared, and proportionally tallies the shared memory to the processes sharing it. smem may help you understand where your memory is going better than top will, but top is an excellent first tool.

How to display processes that are using memory in an given range

This will list processes as expected. Remember that ps shows memory size in kilobytes.

ps -u 1000 -o pid,user,stime,rss \
  | awk '{if($4 > 50000 && $4 < 100000){ print $0 }}' \
  | sort -n -k 4,4

Command output:

 3407 luis.mu+ 10:30 51824
 3523 luis.mu+ 10:30 66108
 3410 luis.mu+ 10:30 71060
 3595 luis.mu+ 10:30 74340
 3609 luis.mu+ 10:30 77772
18550 luis.mu+ 16:47 93616

In that case it's showing only 4 fields for user id 1000. To show all processes use

ps -e -o pid,user,stime,rss

From the ps(3) man page under STANDARD FORMAT SPECIFIERS:

rss

resident set size, the non-swapped physical memory that a task has used (inkiloBytes)

If you want to show more fields, check the man page and add fields to -o option.

How to check the (absolute) memory usage of a process?

To convert Nate Eldredge's comments to an answer, firstly, the 0.3 output is a percentage, so 0.3% of 64 GB is 192 MB.

Secondly, the rss output should give the resident set size:

       rss         RSS       resident set size, the non-swapped physical
                             memory that a task has used (in kilobytes).
                             (alias rssize, rsz).

Running this command gives

> ps -p 24257 -o %mem,rss
%MEM    RSS
 0.3 209908

The output of ~21 Mb agrees pretty well with the estimate of ~19 Gb derived from the %MEM output.

How can I measure the actual memory usage of an application or process?

With ps or similar tools you will only get the amount of memory pages allocated by that process. This number is correct, but:

does not reflect the actual amount of memory used by the application, only the amount of memory reserved for it
can be misleading if pages are shared, for example by several threads or by using dynamically linked libraries

If you really want to know what amount of memory your application actually uses, you need to run it within a profiler. For example, Valgrind can give you insights about the amount of memory used, and, more importantly, about possible memory leaks in your program. The heap profiler tool of Valgrind is called 'massif':

Massif is a heap profiler. It performs detailed heap profiling by taking regular snapshots of a program's heap. It produces a graph showing heap usage over time, including information about which parts of the program are responsible for the most memory allocations. The graph is supplemented by a text or HTML file that includes more information for determining where the most memory is being allocated. Massif runs programs about 20x slower than normal.

As explained in the Valgrind documentation, you need to run the program through Valgrind:

valgrind --tool=massif <executable> <arguments>

Massif writes a dump of memory usage snapshots (e.g. massif.out.12345). These provide, (1) a timeline of memory usage, (2) for each snapshot, a record of where in your program memory was allocated. A great graphical tool for analyzing these files is massif-visualizer. But I found ms_print, a simple text-based tool shipped with Valgrind, to be of great help already.

To find memory leaks, use the (default) memcheck tool of valgrind.

Why do 'htop' and 'top' show different memory usage?

Top counts the cache aswell.

So if you do:

X = Used

Y = Buffers

Z = Cached

X - Y - Z = HTOPMEM

Find exact physical memory usage in Ubuntu/Linux

TL;DR - Virtual memory is complicated.

The best measure of a Linux processes current usage of physical memory is RES.

The RES value represents the sum of all of the processes pages that are currently resident in physical memory. It includes resident code pages and resident data pages. It also includes shared pages (SHR) that are currently RAM resident, though these pages cannot be exclusively ascribed to >>this<< process.

The VIRT value is actually the sum of all notionally allocated pages for the process, and it includes pages that are currently RAM resident, pages that are currently swapped to disk.

See https://stackoverflow.com/a/56351211/1184752 for another explanation.

Note that RES is giving you (roughly) instantaneous RAM usage. That is what you asked about ...

The "actual" memory usage over time is more complicated because the OS's virtual memory subsystem is typically be swapping pages in and out according to demand. So, for example, some of your application's pages may not have been accesses recently, and the OS may then swap them out (to swap space) to free up RAM for other pages required by your application ... or something else.

The VIRT value while actually representing virtual address space, is a good approximation of total (virtual) memory usage. However, it may be an over-estimate:

Some pages in a processes address space are shared between multiple processes. This includes read-only code segments, pages shared between parent and child processes between vfork and exec, and shared memory segments created using mmap.
Some pages may be set to have illegal access (e.g. for stack red-zones) and may not be backed by either RAM or swap device pages.
Some pages of the address space in certain states may not have been committed to either RAM or disk yet ... depending on how the virtual memory system is implemented. (Consider the case where a process requests a huge memory segment and neither reads from it or writes to it. It is possible that the virtual memory implementation will not allocate RAM pages until the first read or write in the page. And if you use lazy swap reservation, swap pages not be committed either. But beware that you can get into trouble with lazy swap reservation.)

VIRT can also be under-estimate because the OS usually reserves swap space for all pages ... whether they are currently swapped in or swapped out. So if you count the RAM and swap versions of a given page as separate units of storage, VIRT usually underestimates the total storage used.

Finally, if your real goal is to limit your application to using at most
700 MB (of virtual address space) then you can use ulimit -v ... to do this. If the application tries to request memory beyond its limit, the request fails.

Check Shared Memory(SHR) consumption for process

The S or share key is listed as obsolete, in the ps man page, so that doesn't seem to be possible:

https://www.man7.org/linux/man-pages/man1/ps.1.html

Your next best option may be something like this:

awk '{print $3}' < /proc/<PID>/statm

That's the number of resident shared pages (so multiple by the page size).

However, there's a big caveat: https://man7.org/linux/man-pages/man5/proc.5.html

       /proc/[pid]/statm
              Provides information about memory usage, measured in
              pages.  The columns are:

                  size       (1) total program size
                             (same as VmSize in /proc/[pid]/status)
                  resident   (2) resident set size
                             (inaccurate; same as VmRSS in /proc/[pid]/status)
                  shared     (3) number of resident shared pages
                             (i.e., backed by a file)
                             (inaccurate; same as RssFile+RssShmem in
                             /proc/[pid]/status)
                  text       (4) text (code)
                  lib        (5) library (unused since Linux 2.6; always 0)
                  data       (6) data + stack
                  dt         (7) dirty pages (unused since Linux 2.6; always 0)

How to See Top Processes Sorted by Actual Memory Usage