Why Isn't Malloc Filling Up Memory

Why isn't malloc filling up memory?

By default, Linux allocates physical memory lazily, the first time it's accessed. Your call to malloc will allocate a large block of virtual memory, but there are not yet any pages of physical memory mapped to it. The first access to an unmapped page will cause a fault, which the kernel will handle by allocating and mapping one or more pages of physical memory.

To allocate all the physical memory, you'll have to access at least one byte on every page; or you could bypass malloc and go straight to the operating system with something like

mmap(0, 1000000000, PROT_NONE, MAP_PRIVATE | MAP_ANONYMOUS | MAP_POPULATE, -1, 0);

Note the use of MAP_POPULATE to populate the page tables, i.e. to allocate physical memory, immediately. According to the manpage, this only works on fairly recent versions of Linux.

malloc zeroing out memory?

malloc itself doesn't zero out memory but it many operating systems will zero the memory that your program requests for security reasons (to keep one process from accessing potentially sensitive information that was used by another process).

Is malloc() initializing allocated array to zero?

The man page of malloc says:

The malloc() function allocates size bytes and returns a pointer to
the allocated memory. The memory is not initialized. If size is 0,
then malloc() returns either NULL, or a unique pointer value that can
later be successfully passed to free().

So malloc() returns uninitialized memory, the contents of which is indeterminate.

 if (arr[i] != 0)

In your program, You have tried to access the content of a memory block, which is invoked undefined behavior.

C - calloc() v. malloc()

From http://wiki.answers.com/Q/Is_it_better_to_use_malloc_or_calloc_to_allocate_memory

malloc() is faster, since calloc() initializes the allocated memory to contain all zeros. Since you typically would want to use and initialize the memory yourself, this additional benefit of calloc() may not be necessary.

I can use more memory than how much I've allocated with malloc(), why?

You've asked a very good question and maybe this will whet your appetite about operating systems. Already you know you've managed to achieve something with this code that you wouldn't ordinarily expect to do. So you would never do this in code you want to make portable.

To be more specific, and this depends entirely on your operating system and CPU architecture, the operating system allocates "pages" of memory to your program - typically this can be in the order of 4 kilobytes. The operating system is the guardian of pages and will immediately terminate any program that attempts to access a page it has not been assigned.

malloc, on the other hand, is not an operating system function but a C library call. It can be implemented in many ways. It is likely that your call to malloc resulted in a page request from the operating system. Then malloc would have decided to give you a pointer to a single byte inside that page. When you wrote to the memory from the location you were given you were just writing in a "page" that the operating system had granted your program, and thus the operating system will not see any wrong doing.

The real problems, of course, will begin when you continue to call malloc to assign more memory. It will eventually return pointers to the locations you just wrote over. This is called a "buffer overflow" when you write to memory locations that are legal (from an operating system perspective) but could potentially be overwriting memory another part of the program will also be using.

If you continue to learn about this subject you'll begin to understand how programs can be exploited using such "buffer overflow" techniques - even to the point where you begin to write assembly language instructions directly into areas of memory that will be executed by another part of your program.

When you get to this stage you'll have gained much wisdom. But please be ethical and do not use it to wreak havoc in the universe!

PS when I say "operating system" above I really mean "operating system in conjunction with privileged CPU access". The CPU and MMU (memory management unit) triggers particular interrupts or callbacks into the operating system if a process attempts to use a page that has not been allocated to that process. The operating system then cleanly shuts down your application and allows the system to continue functioning. In the old days, before memory management units and privileged CPU instructions, you could practically write anywhere in memory at any time - and then your system would be totally at the mercy of the consequences of that memory write!

How to get malloc/calloc to fail if request exceeds free physical memory (i.e., don't use swap)

Expanding on a comment I made to the original question:

If you want to disable swapping, use the swapoff command (sudo swapoff -a). I usually run my machine that way, to avoid it freezing when firefox does something it shouldn't. You can use setrlimit() (or the ulimit command) to set a maximum VM size, but that won't properly compensate for some other process suddenly deciding to be a memory hog (see above).

Even if you choose one of the above options, you should read the rest of this answer to see how to avoid unnecessary initialisation on the first call to calloc().

---

As for your precise test harness, it turns out that you are triggering an unfortunate exception to GNU calloc()'s optimisation.

Here's a comment (now deleted) I made to another answer, which turns out to not be strictly speaking accurate:

I checked the glibc source for the default gnu/linux malloc library, and verified that calloc() does not normally manually clear memory which has just been mmap'd. And malloc() doesn't touch the memory at all.

It turns out that I missed one exception to the calloc optimisation. Because of the way the GNU malloc implementation initialises the malloc system, the first call to calloc always uses memset() to set the newly-allocated storage to 0. Every other call to calloc() passes through the entire calloc logic, which avoids calling memset() on storage which has been freshly mmap'd.

So the following modification to the test program shows radically different behaviour:

#include <stdio.h>
#include <stdlib.h>
int main ( int argc, char *argv[] ) {
  /* These three lines were added */
  void* tmp = calloc(1000, 1); /* force initialization */
  printf("Allocated 1000 bytes at %p\n", tmp);
  free(tmp);
  /* The rest is unchanged */
  unsigned int nmalloc = (argc>1? atoi(argv[1]) : 10000000 ),
               size    = (argc>2? atoi(argv[2]) : (0) );
  unsigned char *pmalloc = (size>0? calloc(nmalloc,size):malloc(nmalloc));
  fprintf( stdout," %s malloc'ed %d elements of %d bytes each.\n",
    (pmalloc==NULL? "UNsuccessfully" : "Successfully"),
    nmalloc, (size>0?size:1) );
  if ( pmalloc != NULL ) free(pmalloc);
}

Note that if you set MALLOC_PERTURB_ to a non-zero value, then it is used to initialise malloc()'d blocks, and forces calloc()'d blocks to be initialised to 0. That's used in the test below.

In the following, I used /usr/bin/time to show the number of page faults during execution. Pay attention to the number of minor faults, which are the result of the operating system zero-initialising a previously unreferenced page in an anonymous mmap'd region (and some other occurrences, like mapping a page already present in Linux's page cache). Also look at the resident set size and, of course, the execution time.

$ gcc -Og -ggdb -Wall -o mall mall.c

$ # A simple malloc completes instantly without page faults
$ /usr/bin/time ./mall 4000000000
Allocated 1000 bytes at 0x55b94ff56260
 Successfully malloc'ed -294967296 elements of 1 bytes each.
0.00user 0.00system 0:00.00elapsed 100%CPU (0avgtext+0avgdata 1600maxresident)k
0inputs+0outputs (0major+61minor)pagefaults 0swaps

$ # Unless we tell malloc to initialise memory
$ MALLOC_PERTURB_=35 /usr/bin/time ./mall 4000000000
Allocated 1000 bytes at 0x5648c2436260
 Successfully malloc'ed -294967296 elements of 1 bytes each.
0.19user 1.23system 0:01.43elapsed 99%CPU (0avgtext+0avgdata 3907584maxresident)k
0inputs+0outputs (0major+976623minor)pagefaults 0swaps

# Same, with calloc. No page faults, instant completion.
$ /usr/bin/time ./mall 1000000000 4
Allocated 1000 bytes at 0x55e8257bb260
 Successfully malloc'ed 1000000000 elements of 4 bytes each.
0.00user 0.00system 0:00.00elapsed 100%CPU (0avgtext+0avgdata 1656maxresident)k
0inputs+0outputs (0major+62minor)pagefaults 0swaps

$ # Again, setting the magic malloc config variable changes everything
$ MALLOC_PERMUTE_=35 /usr/bin/time ./mall 1000000000 4
Allocated 1000 bytes at 0x5646f391e260
 Successfully malloc'ed 1000000000 elements of 4 bytes each.
0.00user 0.00system 0:00.00elapsed 100%CPU (0avgtext+0avgdata 1656maxresident)k
0inputs+0outputs (0major+62minor)pagefaults 0swaps

---

Related Topics