Multiprocessing a for Loop

Multiprocessing a for loop?

You can simply use multiprocessing.Pool:

from multiprocessing import Pool

def process_image(name):
    sci=fits.open('{}.fits'.format(name))
    <process>

if __name__ == '__main__':
    pool = Pool()                         # Create a multiprocessing Pool
    pool.map(process_image, data_inputs)  # process data_inputs iterable with pool

Multiprocessing a for loop in Python

I suppose the quickest / simplest way to get there is to use a multiprocessing pool and let it run across iterable (of your files)... A minimal example with fixed number of workers and a little extra info to observe behavior would be:

import datetime
import time

from multiprocessing import Pool

def long_running_task(filename):
    time.sleep(1)
    print(f"{datetime.datetime.now()} finished: {filename}")

filenames = range(15)

with Pool(10) as mp_pool:
    mp_pool.map(long_running_task, filenames)

This creates a pool of 10 workers and will call long_running_task with each item from filenames (here just series of 0..14 ints as a stand-in) as a task finishes and the worker becomes available.

Alternatively, if you wanted to iterate over the inputs yourself, you could do something like:

with Pool(10) as mp_pool:
    for fn in range(15):
        mp_pool.apply_async(long_running_task, (fn,))
    mp_pool.close()
    mp_pool.join()

This would pass fn as first positional argument for each long_running_task call... when assigning all the work, we need to close the pool to stop accepting any more requests and join to wait for any outstanding jobs to finish.

Start while loop in one multiprocessing function, from another multiprocessing function

i added two comments (with print statements) to highlight the error.

basically action=None in func2() so that is why...

from multiprocessing import Process

running = True
action = None


def func1():
    global action
    if 1+1 == 2:
        action = "fn2"
        print(action)


def func2():
    while running:
        print('got here')  # <--- loops infinitly here
        print(action) # <--- this is none
        while action == "fn2":
            print("fn2")


if __name__ == '__main__':
    p1 = Process(target=func1)
    p1.start()
    p2 = Process(target=func2)
    p2.start()
    p1.join()
    p2.join()

In order to share values when multiprocessing, which is called Sharing state between processes you need to use value or array for a single device shared memory or alternatively, Manager for networks of servers.

Here is a link:
https://docs.python.org/3/library/multiprocessing.html

The basic format looks like this:

from multiprocessing import Process, Value, Array

def f(n, a):
    n.value = 3.1415927
    for i in range(len(a)):
        a[i] = -a[i]

if __name__ == '__main__':
    num = Value('d', 0.0)
    arr = Array('i', range(10))

    p = Process(target=f, args=(num, arr))
    p.start()
    p.join()

    print(num.value)
    print(arr[:])

So in the case of the question what the variable action is equivalent to n (variable) or a (list) etc.. and this can be shares across functions.

Also note that one can parse arguments into multiprocess functions with the args keyword: args=(num, arr ...).

---

Related Topics