Python Multiprocessing Linux Windows Difference

Python multiprocessing linux windows difference

Windows lacks a fork() system call, which duplicates current process. This has many implications, including those listed on the windows multiprocessing documentation page. More specifically:

Bear in mind that if code run in a child process tries to access a global variable, then the value it sees (if any) may not be the same as the value in the parent process at the time that Process.start was called.

In internals, python creates a new process on windows by starting a new process from scratch, and telling it to load all modules again. So any change you have done in current process will not be seen.

In your example, this means that in the child process, your module will be loaded, but the if __name__ == '__main__' section will not be run. So T.init will not be called, and T.val won't exist, thus the error you see.

On the other hand, on POSIX systems (that includes Linux), process creation uses fork, and all global state is left untouched. The child runs with a copy of everything, so it does not have to reload anything and will see its copy of T with its copy of val.

This also means that Process creation is much faster and much lighter on resources on POSIX systems, especially as the “duplication” uses copy-on-write to avoid the overhead of actually copying the data.

There are other quirks when using multiprocessing, all of which are detailed in the python multiprocessing guidelines.

Python Multiprocess diff between Windows and Linux

You shouldn't expect the values of global variables that you set in the parent process to be automatically propagated to the child processes.

Your code happens to work on Unix-like platforms because on those platforms multiprocessing uses fork(). This means that every child processes gets a copy of the parent process's address space, including all global variables.

This isn't the case on Windows; every variable from the parent process that needs to be accessed by the child has to be explicitly passed down or placed in shared memory.

Once you do this, your code will work on both Unix and Windows.

Multiprocessing output differs between Linux and Windows - Why?

As mentioned in one of the posts automatically linked on the sidebar, windows does not have the fork systemcall present on *NIX systems.

This implies that instead of sharing global state (like NIX Processes can do), a Windows child process is basically completely separate. This includes modules.

What I suspect is happening is that the module gets loaded anew and the module1 you access inside module2.start isn't quite the module you expected.

The multiprocessing guidelines explicitly mention that module-level constants are exempt from the rule: "variables may not contain what you expect". Well in either case, the solution is to explicitly pass the module you want to the child process like so:

module 2

def start(mod1):
    print(mod1.x.value)

main.py

if __name__ == '__main__':
    module1.init()
    process = multiprocessing.Process(target=module2.start, args=(module1,))
    process.start()
    process.join()

Why multiprocessing.Process behave differently on windows and linux for global object and function arguments

Adding to @Blckknght's answer: on Windows, each process imports the original module "from scratch", while on Unix-y systems only the main process runs the whole module, while all other processes see whatever exists at the time fork() is used to create the new processes (no, you're not calling fork() yourself - multiprocessing internals call it whenever it creates a new process).

In detail, for your import_mock:

• On all platforms, the main process calls func(), which sets import_mock.to_mock to 1.

• On Unix-y platforms, that's what all new processes see: the fork() occurs after that, so 1 is the state all new processes inherit.

• On Windows, all new processes run the entire module "from scratch". So they each import their own, brand new version of import_mock. Only the main process calls func(), so only the main process sees to_mock change to 1. All other processes see the fresh None state.

That's all expected, and actually easy to understand the second time ;-)

What's going on with passing a is subtler, because it depends more on multiprocessing implementation details. The implementation could have chosen to pickle arguments on all platforms from the start, but it didn't, and now it's too late to change without breaking stuff on some platforms.

Because of copy-on-write fork() semantics, it wasn't necessary to pickle Process() arguments on Unix-y systems, and so the implementation never did. However, without fork() it is necessary to pickle them on Windows - and so the implementation does.

Before Python 3.4, which allows you to force "the Windows implementation" (spawn) on all platforms, there's no mechanical way to avoid possible cross-platform surprises.

But in practice, I've rarely been bothered by this. Knowing that, for example, multiprocessing can depend heavily on pickling, I stay completely clear of getting anywhere near playing tricks with pickles. The only reason you had "a problem" passing an A() instance is that you are playing pickle tricks (via overriding the default __getstate__()).

python multiprocessing vs threading for cpu bound work on windows and linux

Processes are much more lightweight under UNIX variants. Windows processes are heavy and take much more time to start up. Threads are the recommended way of doing multiprocessing on windows.

Python parallel processing - Different behaviors between Linux and Windows

You can do parallel processing under Windows (I have a script running now doing heavy computations and using 100% of all 8 cores) but the way it works is by creating parallel processes, not threads (which won't work because of the GIL except for I/O operations). A few important points:

• you need to use concurrent.futures.ProcessPoolExecutor() (note it's the process pool not the thread pool). See https://docs.python.org/3/library/concurrent.futures.html. In a nutshell, the way it works is that you put the code you want to parallelize in a function and then you call executor.map() which will done the split.
• note that on Windows each parallel process will start from scratch, so you'll probably need to use if __name__ == '__main__:' in a few places to distinguish between what you do in the main process vs the others. The data that you load in the main script will be replicated to the child processes so it has to be serializable (pickl'able in Python lingo).
• in order to efficiently use the core, avoid writing data to objects shared across all processes (eg. a progress counter or a common data structure). Otherwise the synchronization between the processes will kill performance. So monitor the execution from the task manager.

---

Related Topics