Multiprocessing: Use Only the Physical Cores

Multiprocessing: use only the physical cores?

I found a solution that doesn't involve changing the source code of a python module. It uses the approach suggested here. One can check that only
the physical cores are active after running that script by doing:

lscpu

in the bash returns:

CPU(s):                8
On-line CPU(s) list:   0,2,4,6
Off-line CPU(s) list:  1,3,5,7
Thread(s) per core:    1

[One can run the script linked above from within python]. In any case, after running the script above, typing these commands in python:

import multiprocessing
multiprocessing.cpu_count()

returns 4.

Multiprocessing pool size - cpu_count or cpu_count/2?

There are many rules-of-thumb that you may follow, depending on the task as you already figured out

• Number of physical cores
• Number of logical cores
• Number of phyiscal or logical cores minus one (supposedly reserving one core for the logic and control)

To avoid counting logical cores instead of physical ones, I suggest using the psutil library:

import psutil
psutil.cpu_count(logical=False)

As for what using in the end, for numerically intensive applications I tend to go with the number of physical cores. Bear in mind that some BLAS implementations use multithreading by default, which may hurt a lot the scalability of data-parallel pipelines. Use MKL_NUM_THREADS=1 or OPENBLAS_NUM_THREADS=1 (depending on your BLAS backend) as environment variables whenever doing batch processing and you should have quasi-linear speedups w.r.t. the number of physical cores.

How many processors should be used with multiprocessing.Pool?

The difference between the two is clearly stated in the doc:

multiprocessing.cpu_count()
Return the number of CPUs in the system.

This number is not equivalent to the number of CPUs the current process can use. The number of usable CPUs can be obtained with len(os.sched_getaffinity(0)).

So even if you are on a 128-core system, your program could have been somehow limited to only run on a specific set of 10 out of the 128 available CPUs. Since affinity also applies to child threads and processes, it doesn't make much sense to spawn more than 10. You could however try to increase the number of available CPUs through os.sched_setaffinity() before starting your pool.

import os
import multiprocessing as mp

cpu_count = mp.cpu_count() 

if len(os.sched_getaffinity(0)) < cpu_count:
    try:
        os.sched_setaffinity(0, range(cpu_count))
    except OSError:
        print('Could not set affinity')

n = max(len(os.sched_getaffinity(0)), 96)
print('Using', n, 'processes for the pool')

pool = mp.Pool(n)
# ...

See also man 2 sched_setaffinity.

multiprocessing.cpu_count returning wrong number of cores

You have 4 physical cores, but 8 logical processors. For example, you may have a processor with hyper-threading.

From SuperUser: Difference Between Cores and Processors

Why my CPU core number is 2, but use multiprocessing.cpu_count() get 4?

Yes multiprocessing.cpu_count() and os.cpu_count() will return logical processors. If you want to check logical and physical processors seperately you can use psutil They can be uses as shown below.

import psutil

print(psutil.cpu_count(logical = False))
print(psutil.cpu_count(logical = True))

output will be

2

4

---

Related Topics