How to Terminate a Python Subprocess Launched With Shell=True

How to terminate a python subprocess launched with shell=True

Use a process group so as to enable sending a signal to all the process in the groups. For that, you should attach a session id to the parent process of the spawned/child processes, which is a shell in your case. This will make it the group leader of the processes. So now, when a signal is sent to the process group leader, it's transmitted to all of the child processes of this group.

Here's the code:

import os
import signal
import subprocess

# The os.setsid() is passed in the argument preexec_fn so
# it's run after the fork() and before  exec() to run the shell.
pro = subprocess.Popen(cmd, stdout=subprocess.PIPE, 
                       shell=True, preexec_fn=os.setsid) 

os.killpg(os.getpgid(pro.pid), signal.SIGTERM)  # Send the signal to all the process groups

How to cleanly kill subprocesses in python

There are 2 main issues here:

First issue: If you're using shell=True, so you're killing the shell running the process, not the process itself. With its parent killed, the child process goes defunct / isn't killed immediately.

In your case, you're using sleep which is not built-in, so you could drop shell=True, and Popen would yield the actual process id: p.terminate() would work.

You can (and you should) avoid shell=True most of the time, even if it requires extra python coding effort (piping 2 commands together, redirecting input/output, all those cases can be nicely handled by one or several Popen without shell=True.

And (second issue) if the process is still defunct when terminating after that fix, you could call p.wait() (from this question). Seems that calling terminate isn't enough. The Popen object needs to be garbage collected.

How to kill a subprocess in python

Hope it will help you.

import os
import signal
import subprocess

process = subprocess.Popen(cmd, shell = True)
os.killpg(os.getpgid(process.pid), signal.SIGTERM)

How to terminate the script started with subprocess.Popen() based on specific condition

UPDATED
The easiest way to perform this is to pass the termination condition as a parameter to test.py.

Otherwise, you can use printing and reading from stdout and stdin If you want to preserve the output and still use Popen, see below. As an example, consider a simple test.py that calculates (in a very inefficient way) some primes:

test.py

import time

primes = [2, 3]

if __name__ == "__main__":
    for p in primes:
        print(p, flush=True)

    i = 5
    while True:
        for p in primes:
            if i % p == 0:
                break
        if i % p:
            primes.append(i)
            print(i, flush=True)
        i += 2
        time.sleep(.005)

You can read the output and choose to terminate the process when you achieve the desired output. As an example, I want to get primes up to 1000.

import subprocess

proc = subprocess.Popen("python test.py",
                        stdout=subprocess.PIPE, stdin=subprocess.PIPE,
                        bufsize=1, universal_newlines=True,
                        shell=True, text=True)
must_stop = False
primes = []
while proc.poll() is None:
    line = proc.stdout.readline()
    if line:
        new_prime = int(line)
        primes.append(new_prime)
        if  new_prime > 1000:
            print("Threshold achieved", line)
            proc.terminate()
        else:
            print("new prime:", new_prime)
print(primes)

please notice that since there is a delay in the processing and communication, you might get one or two more primes than desired. If you want to avoid that, you'd need bi-directional communication and test.py would be more complicated. If you want to see the output of test.py on screen, you can print it and then somehow parse it and check if the condition is fulfilled. Other options include using os.mkfifo (Linux only, not very difficult), which provides an easy communication path between two processes:

os.mkinfo version

test.py

import time
import sys

primes = [2, 3]

if __name__ == "__main__":
    outfile = sys.stdout
    if len(sys.argv) > 1:
        try:
            outfile = open(sys.argv[1], "w")
        except:
            print("Could not open file")
    for p in primes:
        print(p, file=outfile, flush=True)
    i = 5
    while True:
        for p in primes:
            if i % p == 0:
                break
        if i % p:
            primes.append(i)
            print("This will be printed to screen:", i, flush=True)
            print(i, file=outfile, flush=True) # this will go to the main process
        i += 2
        time.sleep(.005)

main file

import subprocess
import os
import tempfile


tmpdir = tempfile.mkdtemp()
filename = os.path.join(tmpdir, 'fifo')  # Temporary filename
os.mkfifo(filename)  # Create FIFO
proc = subprocess.Popen(["python3", "test.py", filename], shell=False)
with open(filename, 'rt', 1) as fifo:
    primes = []
    while proc.poll() is None:
        line = fifo.readline()
        if line:
            new_prime = int(line)
            primes.append(new_prime)
            if new_prime > 1000:
                print("Threshold achieved", line)
                proc.terminate()
            else:
                print("new prime:", new_prime)
    print(primes)

    pass

os.remove(filename)
os.rmdir(tmpdir)

Python 2.6 on Windows: how to terminate subprocess.Popen with shell=True argument?

Based on the tip given in Thomas Watnedal's answer, where he points out that just the shell is actually being killed in the example, I have arranged the following function which solves the problem for my scenario, based on the example given in Mark Hammond's PyWin32 library:

procname is the name of the process as seen in Task Manager without the extension, e.g. FFMPEG.EXE would be killProcName("FFMPEG"). Note that the function is reasonably slow as it performs enumeration of all current running processes so the result is not instant.

import win32api
import win32pdhutil
import win32con

def killProcName(procname):
    """Kill a running process by name.  Kills first process with the given name."""
    try:
        win32pdhutil.GetPerformanceAttributes("Process", "ID Process", procname)
    except:
        pass

    pids = win32pdhutil.FindPerformanceAttributesByName(procname)

    # If _my_ pid in there, remove it!
    try:
        pids.remove(win32api.GetCurrentProcessId())
    except ValueError:
        pass

    handle = win32api.OpenProcess(win32con.PROCESS_TERMINATE, 0, pids[0])
    win32api.TerminateProcess(handle, 0)
    win32api.CloseHandle(handle)

---

Related Topics