Can Multiprocessing Process Class Be Run from Idle

Can multiprocessing Process class be run from IDLE

Yes. The following works in that function f is run in a separate (third) process.

from multiprocessing import Process

def f(name):
    print('hello', name)

if __name__ == '__main__':
    p = Process(target=f, args=('bob',))
    p.start()
    p.join()

However, to see the print output, at least on Windows, one must start IDLE from a console like so.

C:\Users\Terry>python -m idlelib
hello bob

(Use idlelib.idle on 2.x.) The reason is that IDLE runs user code in a separate process. Currently the connection between the IDLE process and the user code process is via a socket. The fork done by multiprocessing does not duplicate or inherit the socket connection. When IDLE is started via an icon or Explorer (in Windows), there is nowhere for the print output to go. When started from a console with python (rather than pythonw), output goes to the console, as above.

CPU idle when parallelising using Python multiprocessing

It seemed to be because the child processes were memory-intensive and Python silently killed them. The parent process simply waited for the child processes to return. I checked the reason for killed processes using dmesg -T| grep -E -i -m10 -B20 'killed process', where -m specifies max number of matches to return and -B specifies number of lines before the match "killed process".

For more possible reasons and troubleshooting tips, look at the question description and the comments in the question. They include a few things to look out for and the suggestion of viztracer to trace what happened. Quoting from @minker:

viztracer will output the file if the script is stuck. Just Ctrl-C out of it. Just remember you need --log_multiprocess for multiprocessing library when you use viztracer.

CPU (all cores) become idle during python multiprocessing on windows

I confess to not understanding the subtleties of map_async, but I'm not sure whether you can use it like that (I can't seem to get it to work at all)...

I usually use the following recipe (a list comprehension of the calls I want doing):

In [11]: procs = [multiprocessing.Process(target=f, args=()) for _ in xrange(4)]
   ....: for p in procs: p.start()
   ....: for p in procs: p.join()
   ....:

It's simple and waits until the jobs are finished before continuing.

This works fine with pandas objects provided you're not doing modifications... (I think) copies of the object are passed to each thread and if you perform mutations they not propogate and will be garbage collected.

You can use multiprocessing's version of a dict or list with the Manager class, this is useful for storing the result of each job (simply access the dict/list from within the function):

mgr = multiproccessing.Manager()
d = mgr.dict()
L = mgr.list()

and they will have shared access (as if you had written a Lock). It's hardly worth mentioning, that if you are appending to a list then the order will not just be the same as procs!

You may be able to do something similar to the Manager for pandas objects (writing a lock to objects in memory without copying), but I think this would be a non-trivial task...

Module won't work in IDLE

You need to set childprocess=False as IDLE has problems with multiprocessing. IDLE runs user code in a separate process so with pyscreenshot IDLE hangs and seems like the module won't work. You may go through this to know more.

An option is to turn off multiprocessing by setting childprocess=False.

Try:

import pyscreenshot as getImage 
im = getImage.grab(bbox=(1300,800,1500,850), childprocess=False) 
im.save("screen.png")

---

Related Topics