How to Start a Background Process in Python

How to start a background process in Python?

Note: This answer is less current than it was when posted in 2009. Using the subprocess module shown in other answers is now recommended in the docs

(Note that the subprocess module provides more powerful facilities for spawning new processes and retrieving their results; using that module is preferable to using these functions.)

---

If you want your process to start in the background you can either use system() and call it in the same way your shell script did, or you can spawn it:

import os
os.spawnl(os.P_DETACH, 'some_long_running_command')

(or, alternatively, you may try the less portable os.P_NOWAIT flag).

See the documentation here.

How to run a background process and do *not* wait?

Here is verified example for Python REPL:

>>> import subprocess
>>> import sys
>>> p = subprocess.Popen([sys.executable, '-c', 'import time; time.sleep(100)'], stdout=subprocess.PIPE, stderr=subprocess.STDOUT); print('finished')
finished

How to verify that via another terminal window:

$ ps aux | grep python

Output:

user           32820   0.0  0.0  2447684   3972 s003  S+   10:11PM   0:00.01 /Users/user/venv/bin/python -c import time; time.sleep(100)

Run Background Process in Python

I think what would be more beneficial to you would be threading, you are able to start a process in another thread without blocking the main thread which runs your gui. Once the other thread has completed its task it will join the main thread

How can I start a process and put it to background in python?

First it is easy not to block the Python script in communicate... by not calling communicate! Just read from output or error output from the command until you find the correct message and just forget about the command.

# to avoid waiting for an EOF on a pipe ...
def getlines(fd):
    line = bytearray()
    c = None
    while True:
        c = fd.read(1)
        if c is None:
            return
        line += c
        if c == '\n':
            yield str(line)
            del line[:]

p = subprocess.Popen(startCommand, shell=True, stdout=subprocess.PIPE,
               stderr=subprocess.STDOUT) # send stderr to stdout, same as 2>&1 for bash
for line in getlines(p.stdout):
    if "Server started in RUNNING mode" in line:
        print("STARTUP OK")
        break
else:    # end of input without getting startup message
     print("STARTUP FAILED")
     p.poll()    # get status from child to avoid a zombie
     # other error processing

The problem with the above, is that the server is still a child a the Python process and could get unwanted signals such as SIGHUP. If you want to make it a daemon, you must first start a subprocess that next start your server. That way when first child will end, it can be waited by caller and the server will get a PPID of 1 (adopted by init process). You can use multiprocessing module to ease that part

Code could be like:

import multiprocessing
import subprocess

# to avoid waiting for an EOF on a pipe ...
def getlines(fd):
    line = bytearray()
    c = None
    while True:
        c = fd.read(1)
        if c is None:
            return
        line += c
        if c == '\n':
            yield str(line)
            del line[:]

def start_child(cmd):
    p = subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.STDOUT,
                         shell=True)
    for line in getlines(p.stdout):
        print line
        if "Server started in RUNNING mode" in line:
            print "STARTUP OK"
            break
    else:
        print "STARTUP FAILED"

def main():
    # other stuff in program
    p = multiprocessing.Process(target = start_child, args = (server_program,))
    p.start()
    p.join()
    print "DONE"
    # other stuff in program

# protect program startup for multiprocessing module
if __name__ == '__main__':
    main()

One could wonder what is the need for the getlines generator when a file object is itself an iterator that returns one line at a time. The problem is that it internally calls read that read until EOF when file is not connected to a terminal. As it is now connected to a PIPE, you will not get anything until the server ends... which is not what is expected

run bash script in background with process name

You can run programs with a specific command name by using the bash buildin exec. Note that exec replaces the shell with the command so you have to run it in a subshell environment like:

( exec -a my_new_name my_old_command ) &

However, it probably won't help you much because this sets the command line name, which is apparently different from the command name. So executing the above snippet will show your process as "my_new_name" for example in top or htop, but pkill and killall are filtering by the command name and will thus not find a process called "my_new_name".

While it is interesting, how one can start a command with a different name than the executable, it is most likely not the cause of your problem. PIDs never change, so I assume that the problem lays somewhere different.

My best guess is that the server binds a socket to listen on a specific port. If the program is not shutdown gracefully but killed the port number remains occupied and is only freed by the kernel after some time during some kind of kernel garbage collect. If the program is restarted after a short period of time it finds the port already been occupied and prints a misleading message, that says it is already running. If that is indeed the cause of your problem I would strongly consider implementing a way to graceful shutdown the server. (may be already closing the socket in a destructor or something similar could help)

Execute Subprocess in Background

& is a shell feature. If you want it to work with subprocess, you must specify shell=True like:

subprocess.call(command, shell=True)

This will allow you to run command in background.

Notes:

1. Since shell=True, the above uses command, not command_list.

2. Using shell=True enables all of the shell's features. Don't do this unless command including thingy comes from sources that you trust.

Safer Alternative

This alternative still lets you run the command in background but is safe because it uses the default shell=False:

p = subprocess.Popen(command_list)

After this statement is executed, the command will run in background. If you want to be sure that it has completed, run p.wait().

---

Related Topics