Launch a Completely Independent Process

Launch a completely independent process

You open your long-running process and keep a pipe to it. So you expect to talk to it. When yor launcher script exits, you can no longer talk to it. The long-running process receives a SIGPIPE and exits.

The following just worked for me (Linux, Python 2.7).

Create a long-running executable:

$ echo "sleep 100" > ~/tmp/sleeper.sh

Run Python REPL:

$ python
>>>

import subprocess
import os
p = subprocess.Popen(['/bin/sh', os.path.expanduser('~/tmp/sleeper.sh')])
# look ma, no pipes!
print p.pid
# prints 29893

Exit the REPL and see the process still running:

>>> ^D
$ ps ax | grep sleeper
29893 pts/0    S      0:00 /bin/sh .../tmp/sleeper.sh
29917 pts/0    S+     0:00 grep --color=auto sleeper

If you want to first communicate to the started process and then leave it alone to run further, you have a few options:

• Handle SIGPIPE in your long-running process, do not die on it. Live without stdin after the launcher process exits.
• Pass whatever you wanted using arguments, environment, or a temporary file.
• If you want bidirectional communication, consider using a named pipe (man mkfifo) or a socket, or writing a proper server.
• Make the long-running process fork after the initial bi-direcional communication phase is done.

Launch a totally independent process from Python

I think I found the answer. By using Popen with close_fds = True I was able to start up a process that was independent and without handles to the parent.

For docs look here and search for close_fds.

Or, on Windows, if close_fds is true then no handles will be inherited
by the child process. Note that on Windows, you cannot set close_fds
to true and also redirect the standard handles by setting stdin,
stdout or stderr.

Note this solution only works on Windows. I have no idea about any *nix system.

How do I launch a completely independent process from a Java program?

It may help if you post a test section of minimal code needed to reproduce the problem. I tested the following code on Windows and a Linux system.

public class Main {

    /**
     * @param args the command line arguments
     */
    public static void main(String[] args) throws Exception {
        Runtime.getRuntime().exec(args[0]);
    }
}

And tested with the following on Linux:

java -jar JustForTesting.jar /home/monceaux/Desktop/__TMP/test.sh

where test.sh looks like:

#!/bin/bash
ping -i 20 localhost

as well as this on Linux:

java -jar JustForTesting.jar gedit

And tested this on Windows:

java -jar JustForTesting.jar notepad.exe

All of these launched their intended programs, but the Java application had no problems exiting. I have the following versions of Sun's JVM as reported by java -version :

• Windows: 1.6.0_13-b03
• Linux: 1.6.0_10-b33

I have not had a chance to test on my Mac yet. Perhaps there is some interaction occuring with other code in your project that may not be clear. You may want to try this test app and see what the results are.

Launch an independent process with python

Solution for Windows: os.startfile()

Works as if you double clicked an executable and causes it to launch independently. A very handy one liner.

http://docs.python.org/library/os.html?highlight=startfile#os.startfile

Subprocess.popen not creating independent process

In windows, you could make use of creationflags option of Popen.

DETACHED_PROCESS = 8 # bit mask taken from the linked doc below
subprocess.Popen(executable, creationflags=DETACHED_PROCESS, close_fds=True)

Which will create a detached process (demon process).

https://learn.microsoft.com/en-us/windows/desktop/ProcThread/process-creation-flags

For console processes, the new process does not inherit its parent's
console (the default). The new process can call the AllocConsole
function at a later time to create a console. For more information,
see Creation of a Console.

Java launch independent process

Your problem is due to what is called Unix job control.

Like many shells do, /bin/sh intercepts SIGHUP and SIGINT signals, and before exiting, it sends signals to some of its child processes groups, depending on its configuration and on their state (for instance, stopped background processes receive a SIGCONT).

So, when your main java app is closed, the /bin/sh shell that your app had forked is terminated, and just before exiting, it sends a SIGHUP signal to its subprocesses corresponding to the command java -jar myjar.jar.

So, the answer to your question is: just use the huponexit /bin/sh shell option to avoid killing subprocesses. They will be detached from the controlling terminal, if any, but they will not be killed.

So, replace this java -jar myjar.jar by shopt -u huponexit; java -jar myjar.jar:

ProcessBuilder processBuilder =
  new ProcessBuilder(new String[] {
    "su", "-s", "/bin/sh", "myuser", "-c",
    "shopt -u huponexit; java -jar myjar.jar"
  });
Process p = processBuilder.start();

How to initialize parallel independent process within function?

If you don't wan't to use a 3rd-party lib like daemonocle implementing a "well-behaved" Unix-Daemon, you could
use subprocess.Popen() to create an independent process. Another option would be to modify multiprocessing.Process to prevent auto-joining of the child when the parent exits.

---

subprocess.Popen()

With subprocess.Popen() you start the new process with specifying commands and arguments like manually from terminal. This means you need to make funcs.py or another file a top-level script which parses string-arguments from stdin and then calls funcs.calculate() with these arguments.

I boiled your example down to the essence so we don't have to read too much code.

funcs.py

#!/usr/bin/env python3
# UNIX: enable executable from terminal with: chmod +x filename
import os
import sys
import time

import psutil  # 3rd party for demo

def print_msg(msg):
    print(f"[{time.ctime()}, pid: {os.getpid()}] --- {msg}")

def calculate(data, *args):
    print_msg(f"parent pid: {psutil.Process().parent().pid}, start calculate()")
    for _ in range(int(500e6)):
        pass
    print_msg(f"parent pid: {psutil.Process().parent().pid}, end calculate()")

if __name__ == '__main__':

    if len(sys.argv) > 1:
        calculate(*sys.argv[1:])

subp_main.py

#!/usr/bin/env python3
# UNIX: enable executable from terminal with: chmod +x filename
if __name__ == '__main__':

    import time
    import logging
    import subprocess
    import multiprocessing as mp

    import funcs

    mp.log_to_stderr(logging.DEBUG)

    filename = funcs.__file__
    data = ("data", 42)

    # in case filename is an executable you don't need "python" before `filename`:
    subprocess.Popen(args=["python", filename, *[str(arg) for arg in data]])
    time.sleep(1)  # keep parent alive a bit longer for demo
    funcs.print_msg(f"exiting")

And important for testing, run from terminal, e.g. not PyCharm-Run, because it won't show what the child prints. In the last line below you see the child process' parent-id changed to 1 because the child got adopted by systemd (Ubuntu) after the parent exited.

$> ./subp_main.py
[Fri Oct 23 20:14:44 2020, pid: 28650] --- parent pid: 28649, start calculate()
[Fri Oct 23 20:14:45 2020, pid: 28649] --- exiting
[INFO/MainProcess] process shutting down
[DEBUG/MainProcess] running all "atexit" finalizers with priority >= 0
[DEBUG/MainProcess] running the remaining "atexit" finalizers
$> [Fri Oct 23 20:14:54 2020, pid: 28650] --- parent pid: 1, end calculate()

---

class OrphanProcess(multiprocessing.Process)

If you search for something more convenient, well you can't use the high-level multiprocessing.Process as is, because it doesn't let the parent process exit before the child, as you asked for. Regular child-processes are either joined (awaited) or terminated (if you set the daemon-flag for Process) when the parent shuts down. This still happens within Python. Note that the daemon-flag doesn't make a process a Unix-Daemon. The naming is a somewhat frequent source of confusion.

I subclassed multiprocessing.Process to switch the auto-joining off and spend some time with the source and observing if zombies might become an issue. Because the modification turns off automatic joining in the parent, I recommend using "forkserver" as start-method for new processes on Unix (always a good idea if the parent is already multi-threaded) to prevent zombie-children from sticking around as long the parent is still running. When the parent process terminates, its child-zombies get eventually reaped by systemd/init. Running multiprocessing.log_to_stderr() shows everything shutting down cleanly, so nothing seems broken so far.

Consider this approach experimental, but it's probably a lot safer than using raw os.fork() to re-invent part of the extensive multiprocessing machinery, just to add this one feature. For error-handling in the child, write a try-except block and log to file.

orphan.py

import multiprocessing.util
import multiprocessing.process as mpp
import multiprocessing as mp

__all__ = ['OrphanProcess']

class OrphanProcess(mp.Process):
    """Process which won't be joined by parent on parent shutdown."""
    def start(self):
        super().start()
        mpp._children.discard(self)

    def __del__(self):
        # Finalizer won't `.join()` the child because we discarded it,
        # so here last chance to reap a possible zombie from within Python.
        # Otherwise systemd/init will reap eventually.
        self.join(0)

orph_main.py

#!/usr/bin/env python3
# UNIX: enable executable from terminal with: chmod +x filename
if __name__ == '__main__':

    import time
    import logging
    import multiprocessing as mp
    from orphan import OrphanProcess
    from funcs import print_msg, calculate

    mp.set_start_method("forkserver")
    mp.log_to_stderr(logging.DEBUG)

    p = OrphanProcess(target=calculate, args=("data", 42))
    p.start()
    time.sleep(1)
    print_msg(f"exiting")

Again test from terminal to get the child print to stdout. When the shell appears to be hanging after everything was printed over the second prompt, hit enter to get a new prompt. The parent-id stays the same here because the parent, from the OS-point of view, is the forkserver-process, not the initial main-process for orph_main.py.

$> ./orph_main.py
[INFO/MainProcess] created temp directory /tmp/pymp-bd75vnol
[INFO/OrphanProcess-1] child process calling self.run()
[Fri Oct 23 21:18:29 2020, pid: 30998] --- parent pid: 30997, start calculate()
[Fri Oct 23 21:18:30 2020, pid: 30995] --- exiting
[INFO/MainProcess] process shutting down
[DEBUG/MainProcess] running all "atexit" finalizers with priority >= 0
[DEBUG/MainProcess] running the remaining "atexit" finalizers
$> [Fri Oct 23 21:18:38 2020, pid: 30998] --- parent pid: 30997, end calculate()
[INFO/OrphanProcess-1] process shutting down
[DEBUG/OrphanProcess-1] running all "atexit" finalizers with priority >= 0
[DEBUG/OrphanProcess-1] running the remaining "atexit" finalizers
[INFO/OrphanProcess-1] process exiting with exitcode 0

---

Related Topics