How to Share Data Between a Parent and Forked Child Process in Python

How do you share data between a parent and forked child process in Python?

Subprocess replaces os.popen, os.system, os.spawn, popen2 and commands. A simple example for piping would be:

p1 = Popen(["dmesg"], stdout=PIPE)
p2 = Popen(["grep", "hda"], stdin=p1.stdout, stdout=PIPE)
output = p2.communicate()[0]

You could also use a memory mapped file with the flag=MAP_SHARED for shared memory between processes.

multiprocessing abstracts both pipes and shared memory and provides a higher level interface. Taken from the Processing documentation:

from multiprocessing import Process, Pipe

def f(conn):
    conn.send([42, None, 'hello'])
    conn.close()

if __name__ == '__main__':
    parent_conn, child_conn = Pipe()
    p = Process(target=f, args=(child_conn,))
    p.start()
    print parent_conn.recv()   # prints "[42, None, 'hello']"
    p.join()

Communication between parent and child process when forking in Python

I'd recommend using multiprocessing rather than os.fork(), as it handles a lot of details for you. In particular it provides the Manager class, which provides a nice way to share data between processes. You'd start one Process to handle getting the data, and another for doing the web serving, and pass them both a shared data dictionary provided by the Manager. The main process is then just responsible for setting all that up (and waiting for the processes to finish - otherwise the Manager breaks).

Here's what this might look like:

import time
from multiprocessing import Manager, Process

def get_data():
    """ Does the actual work of getting the updating value. """

def update_the_data(shared_dict):
    while not shared_dict.get('server_started'):
        time.sleep(.1)
    while True:
        shared_dict['data'] = get_data()
        shared_dict['data_timestamp'] = time.time()
        time.sleep(LOOP_DELAY)


def serve_the_data(shared_dict):
    server = initialize_server() # whatever this looks like
    shared_dict['server_started'] = True
    while True:
        server.serve_with_timeout()
        if time.time() - shared_dict['data_timestamp'] > 30:
            # child hasn't updated data for 30 seconds; problem?
            handle_child_problem()


if __name__ == '__main__':
    manager = Manager()
    shared_dict = manager.dict()
    processes = [Process(target=update_the_data, args=(shared_dict,)),
        Process(target=serve_the_data, args=(shared_dict,))]
    for process in processes:
        process.start()
    for process in processes:
        process.join()

Python: Share variable between forked child and parent

As suggested by @polku multiprocessing.Value works well

With these modifications

availables_cpu = multiprocessing.Value('I',psutil.cpu_count())

and replace all the occurences of availables_cpu by availables_cpu.value

how to fork a child process from a child process respectively

Thanks to AsTeR and chrk. After i read the answers and some playing around, i ended up with this. I used os.wait().

import os

reply = int(input("Enter no of proc:   "))
pid = 0

for i in range(reply):

    if pid == 0:
        pid = os.fork()

if pid != 0:
    os.wait()
    print("PID = {}, PPID = {}".format(os.getpid(), os.getppid()))

And the result for num_process = 3 is:

Enter no of proc:   3
PID = 49312, PPID = 49311
PID = 49311, PPID = 49309
PID = 49309, PPID = 20928

---

Related Topics