Allocating More Memory Than There Exists Using Malloc

Allocating more memory than there exists using malloc

It is called memory overcommit. You can disable it by running as root:

 echo 2 > /proc/sys/vm/overcommit_memory

and it is not a kernel feature that I like (so I always disable it). See malloc(3) and mmap(2) and proc(5)

^{NB: echo 0 instead of echo 2 often -but not always- works also. Read the docs (in particular proc man page that I just linked to).}

I can use more memory than how much I've allocated with malloc(), why?

You've asked a very good question and maybe this will whet your appetite about operating systems. Already you know you've managed to achieve something with this code that you wouldn't ordinarily expect to do. So you would never do this in code you want to make portable.

To be more specific, and this depends entirely on your operating system and CPU architecture, the operating system allocates "pages" of memory to your program - typically this can be in the order of 4 kilobytes. The operating system is the guardian of pages and will immediately terminate any program that attempts to access a page it has not been assigned.

malloc, on the other hand, is not an operating system function but a C library call. It can be implemented in many ways. It is likely that your call to malloc resulted in a page request from the operating system. Then malloc would have decided to give you a pointer to a single byte inside that page. When you wrote to the memory from the location you were given you were just writing in a "page" that the operating system had granted your program, and thus the operating system will not see any wrong doing.

The real problems, of course, will begin when you continue to call malloc to assign more memory. It will eventually return pointers to the locations you just wrote over. This is called a "buffer overflow" when you write to memory locations that are legal (from an operating system perspective) but could potentially be overwriting memory another part of the program will also be using.

If you continue to learn about this subject you'll begin to understand how programs can be exploited using such "buffer overflow" techniques - even to the point where you begin to write assembly language instructions directly into areas of memory that will be executed by another part of your program.

When you get to this stage you'll have gained much wisdom. But please be ethical and do not use it to wreak havoc in the universe!

PS when I say "operating system" above I really mean "operating system in conjunction with privileged CPU access". The CPU and MMU (memory management unit) triggers particular interrupts or callbacks into the operating system if a process attempts to use a page that has not been allocated to that process. The operating system then cleanly shuts down your application and allows the system to continue functioning. In the old days, before memory management units and privileged CPU instructions, you could practically write anywhere in memory at any time - and then your system would be totally at the mercy of the consequences of that memory write!

is Malloc allocating more memory then needed?

Usually malloc allocates memory in chunks exactly divisible by the paragraph size equal to 16 bytes (that is malloc allocates chunks with a fundamental alignment requirement). However you should not rely on this internal behavior of malloc. Otherwise the behavior is undefined.

Why is malloc() allocating more memory than demanded and how can I disable malloc from doing this in Mac OS X?

Unlike what most people think, malloc does not allocate the memory directly from the kernel. Instead, it takes a chunk of memory from the kernel, and then manages that chunk in user space with a data structure. That data structure used to be a heap, hence calling it "heap memory".

The practical upshot of the above is that the memory allocated using malloc does not have invalid memory around it. Overstepping the memory by a little will not, in fact, cause your program to crash.

In addition to the above, malloc itself does not allocate in any size you wish. It usually allocates chunks in multiples of 8, 16 or 32 bytes, depending on the data structure used to manage the heap. When you allocate 1 byte with malloc, you are actually getting around 15 bytes that will never be used by anyone. Overwriting those bytes will not cause any noticeable ill effect.

If you want to make sure your program works with a small amount of memory, rlimit/ulimit and co are your friends. If you want to make sure you are not overstepping your allocated buffers, I strongly recommend the address sanitizer. Compile (both clang and gcc) with -fsanitize=address, and your program will crash as soon as you overstep your buffers. This comes at a performance cost, of course.

I'll also add that hardware access control are only possible on page boundaries. On Intel, that's 4096 bytes. Even for memory allocated from the kernel, accesses past the end of the allocation will not trigger a segmentation fault if they do not step outside the allocated page.

Using malloc is giving me more memory than expected?

C isn't required to perform any bounds checking on array access. It can allow you read/write past that without any warning or error.

You're invoking undefined behavior by reading and writing past the end of allocated memory. This means the behavior of your program can't be predicted. It could crash, it could output strange results, or it could (as in your case) appear to work properly.

Just because the program can crash doesn't mean it will.

Why does Malloc allocate more memory than I requested?

The malloc call only set aside as much storage as you specified (10 elements of struct user). C doesn't do any bounds checking on array accesses; the language assumes that you know how big your array is, and that you're smart enough not to go past the end of it.

Attempting to access items past the end of the array invokes undefined behavior, which doesn't have to result in a segfault or a crash. It just so happened that the memory at index 2815 wasn't protected or being used for anything "important" (although it may have affected execution somewhere else).

For reference, here's the definition for "undefined behavior", taken from the latest standard:

3.4.3

1 undefined behavior
behavior, upon use of a nonportable or erroneous program construct or of erroneous data,
for which this International Standard imposes no requirements

2 NOTE Possible undefined behavior ranges from ignoring the situation completely with unpredictable
results, to behaving during translation or program execution in a documented manner characteristic of the
environment (with or without the issuance of a diagnostic message), to terminating a translation or
execution (with the issuance of a diagnostic message).

3 EXAMPLE An example of undefined behavior is the behavior on integer overflow

Maximum memory which malloc can allocate

I read that the maximum memory malloc can allocate is limited to physical memory (on heap).

Wrong: most computers/OSs support virtual memory, backed by disk space.

Some questions: does malloc allocate memory from HDD also?

malloc asks the OS, which in turn may well use some disk space.

What was the reason for above behavior? Why didn't the loop break at any time?

Why wasn't there any allocation failure?

You just asked for too little at a time: the loop would have broken eventually (well after your machine slowed to a crawl due to the large excess of virtual vs physical memory and the consequent super-frequent disk access, an issue known as "thrashing") but it exhausted your patience well before then. Try getting e.g. a megabyte at a time instead.

When a program exceeds consumption of memory to a certain level, the
computer stops working because other applications do not get enough
memory that they require.

A total stop is unlikely, but when an operation that normally would take a few microseconds ends up taking (e.g.) tens of milliseconds, those four orders of magnitude may certainly make it feel as if the computer had basically stopped, and what would normally take a minute could take a week.

---

Related Topics