How to Find Which Process Is Leaking Memory

How to find which process is leaking file handles in Linux?

Probably the root cause is a bug in NFSv4 implementation: https://stackoverflow.com/a/5205459/280758

They have similar symptoms.

How OS handles memory leaks

Recommended reading : Operating Systems: Three Easy Pieces

On common OSes (e.g. Linux, Windows, MacOSX, Android) each process has its own virtual address space (and the heap memory, e.g. used for malloc or mmap, is inside that virtual address space), and when the process terminates, its entire virtual address space is destroyed.

So memory leaks don't survive the process itself.

There could be subtle corner cases (e.g. leaks on using shm_overview(7) or shmget(2)).

Read (for Linux) proc(5), try cat /proc/self/maps, and see also this. Learn to use valgrind and the Address Sanitizer.

Read also about Garbage Collection. It is quite relevant.

How to check for memory leaks in a large scale c++ Linux application?

First of all...

and we reached the point when it is mandatory to make a roundup of checks for memory leaks.

This, actually, is a problem of methodology. Correctness should be the primary goal of any software piece and not an afterthought.

I will suppose though that you now realize this and how much easier it would have been to identify the problems had you been running an instrumented unit test at each commit.

---

So, what to do now ?

• Runtime detection:

  • Try to make Valgrind work, you probably have some environmental issues
  • Try ASan, ThreadSan and MemSan; they are not trivial to setup under Linux but oh so impressive!
  • Try instrumented builds: tcmalloc includes a heap-checker for example
  • ...

• Compile time detection:

  • Turn on the warnings (preferably with -Werror) (not specific to your issue)
  • Use static analysis, such as Clang's, it may spot unpaired allocation routines
  • ...

• Human detection:

  • Code reviews: make sure all resources are allocated within RAII classes
  • ...

Note: using only RAII classes helps removing memory leaks, but does not help with dangling references. Thankfully, detecting dangling references is what ASan does.

---

And once you have patched all the issues, make sure that this becomes part of the process. Changes should be reviewed and tested, always, so that rotten eggs are culled immediately rather than left to stink up the code base.

Windows memory metric to detect memory leak

GlobalMemoryStatusEx is wrong. You do not want to fill up the machine memory until 5 GB are left in total.

You need GetProcessMemoryInfo.

BOOL WINAPI GetProcessMemoryInfo(
  __in   HANDLE Process,
  __out  PPROCESS_MEMORY_COUNTERS ppsmemCounters,
  __in   DWORD cb
);

From an example using GetProcessMemoryInfo:

#include <windows.h>
#include <stdio.h>
#include <psapi.h>

// To ensure correct resolution of symbols, add Psapi.lib to TARGETLIBS
// and compile with -DPSAPI_VERSION=1

void PrintMemoryInfo( DWORD processID )
{
    HANDLE hProcess;
    PROCESS_MEMORY_COUNTERS pmc;

    // Print the process identifier.

    printf( "\nProcess ID: %u\n", processID );

    // Print information about the memory usage of the process.

    hProcess = OpenProcess(  PROCESS_QUERY_INFORMATION |
                                    PROCESS_VM_READ,
                                    FALSE, processID );
    if (NULL == hProcess)
        return;

    if ( GetProcessMemoryInfo( hProcess, &pmc, sizeof(pmc)) )
    {
        printf( "\tWorkingSetSize: 0x%08X\n", pmc.WorkingSetSize );
        printf( "\tPagefileUsage: 0x%08X\n", pmc.PagefileUsage ); 
    }

    CloseHandle( hProcess );
}

int main( void )
{
    // Get the list of process identifiers.

    DWORD aProcesses[1024], cbNeeded, cProcesses;
    unsigned int i;

    if ( !EnumProcesses( aProcesses, sizeof(aProcesses), &cbNeeded ) )
    {
        return 1;
    }

    // Calculate how many process identifiers were returned.

    cProcesses = cbNeeded / sizeof(DWORD);

    // Print the memory usage for each process

    for ( i = 0; i < cProcesses; i++ )
    {
        PrintMemoryInfo( aProcesses[i] );
    }

    return 0;
}

Although unintuitive you need to read PagefileUsage which gets you the committed memory which was allocated by your process. WorkingSetSize is unreliable because if the machine gets tight on memory the OS will write all data to the page file. That can cause WorkingSetSize to be small (e.g. 100 MB) but in reality you leaked already 20 GB of memory. This would result in a saw tooth pattern in memory consumption until the page file is full. Working set is only the actively used memory which might hide the multi GB memory leak if the machine is under memory pressure.

---

Related Topics