Numa-Aware Named Shared Memory for Linux

NUMA aware cache aligned memory allocation

If you're just looking to get the alignment functionality around a NUMA allocator, you can easily build your own.

The idea is to call the unaligned malloc() with a little bit more space. Then return the first aligned address. To be able to free it, you need to store the base address at a known location.

Here's an example. Just substitute the names with whatever is appropriate:

pint         //  An unsigned integer that is large enough to store a pointer.
NUMA_malloc  //  The NUMA malloc function
NUMA_free    //  The NUMA free function

void* my_NUMA_malloc(size_t bytes,size_t align, /* NUMA parameters */ ){

    //  The NUMA malloc function
    void *ptr = numa_malloc(
        (size_t)(bytes + align + sizeof(pint)),
        /* NUMA parameters */
    );

    if (ptr == NULL)
        return NULL;

    //  Get aligned return address
    pint *ret = (pint*)((((pint)ptr + sizeof(pint)) & ~(pint)(align - 1)) + align);

    //  Save the free pointer
    ret[-1] = (pint)ptr;

    return ret;
}

void my_NUMA_free(void *ptr){
    if (ptr == NULL)
        return;

    //  Get the free pointer
    ptr = (void*)(((pint*)ptr)[-1]);

    //  The NUMA free function
    numa_free(ptr); 
}

To when you use this, you need to call my_NUMA_free for anything allocated with my_NUMA_malloc.

linux kernel module memory layout under NUMA

It depends on the OS. If you are using Linux a "First Touch" policy will be applied. This means all the data of you program will be allocated on a node close to the one containing the CPU on which your program is running.

Logically your program will see a unique shared memory. In a lower lever the OS may allocate the data on different NUMA node causing some performance issues.

It depends on you if you want to write a NUMA-aware code to avoid NUMA latency and contention. Although some Linux distribution are already implementing some NUMA balancing features. (e.g. Automatic NUMA Balancing)

Related Topics