Python Sigkill Catching Strategies

python sigkill catching strategies

SIGKILL by its very nature cannot be trapped.

See http://en.wikipedia.org/wiki/Unix_signal#SIGKILL:

SIGKILL

The SIGKILL signal is sent to a process to cause it to terminate
immediately (kill). In contrast to SIGTERM and SIGINT, this signal cannot
be caught or ignored, and the receiving process cannot perform any
clean-up upon receiving this signal.

The best thing to do is the next time your process launches, look for anything that needs to be cleaned up.

And yes, kill -9 <pid> will send a SIGKILL to the process. (To be precise, it sends the 9th signal - it just happens that SIGKILL has the number 9 on pretty much every system. You could alternatively write kill -KILL <pid>, which lets you specify the signal by name instead of by number in a portable way.)

Python script terminated by SIGKILL rather than throwing MemoryError

It sounds like you've run into the dreaded Linux OOM Killer. When the system completely runs of out of memory and the kernel absolutely needs to allocate memory, it kills a process rather than crashing the entire system.

Look in the syslog for confirmation of this. A line similar to:

kernel: [884145.344240] mysqld invoked oom-killer:

followed sometime later with:

kernel: [884145.344399] Out of memory: Kill process 3318

Should be present (in this example, it mentions mysql specifically)

You can add these lines to your /etc/sysctl.conf file to effectively disable the OOM killer:

vm.overcommit_memory = 2
vm.overcommit_ratio = 100

And then reboot. Now, the original, memory hungry, process should fail to allocate memory and, hopefully, throw the proper exception.

Setting overcommit_memory means that Linux won't over commit memory, meaning memory allocations will fail if there isn't enough memory for them. See this answer for details on what effect the overcommit_ratio has: https://serverfault.com/a/510857

How to send SIGINT to python running in a subprocess

Maybe try:

#!/usr/local/cpython-3.8/bin/python3

import signal
import subprocess
import time

proc = subprocess.Popen(['python3', 'B.py'], shell=False)
time.sleep(1)
proc.send_signal(signal.SIGINT)
proc.wait()

...as your A.py?

I suspect the shell is ignoring SIGINT rather than passing it down to B.py.

Killing the children with the parent

prctl's PR_SET_DEATHSIG can only be set for this very process that's calling prctl -- not for any other process, including this specific process's children. The way the man page I'm pointing to expresses this is "This value is cleared upon a fork()" -- fork, of course, is the way other processes are spawned (in Linux and any other Unix-y OS).

If you have no control over the code you want to run in subprocesses (as would be the case, essentially, for your gnuchess example), I suggest you first spawn a separate small "monitor" process with the role of keeping track of all of its siblings (your parent process can let the monitor know about those siblings' pids as it spawns them) and sending them killer signals when the common parent dies (the monitor needs to poll for that, waking up every N seconds for some N of your choice to check if the parent's still alive; use select to wait for more info from the parent with a timeout of N seconds, within a loop).

Not trivial, but then such system tasks often aren't. Terminals do it differently (via the concept of a "controlling terminal" for a process group) but of course it's trivial for any child to block THAT off (double forks, nohup, and so on).

Logs from signal handler hidden when redirecting stdout to file via tee

If you don't want this to happen, put tee in the background so it isn't part of the process group getting a SIGINT. For example, with bash 4.1 or newer, you can start a process substitution with an automatically-allocated file descriptor providing a handle:

#!/usr/bin/env bash
#              ^^^^ NOT /bin/sh; >(...) is a bashism, likewise automatic FD allocation.

exec {log_fd}> >(exec tee log.txt)  # run this first as a separate command
python3 -u myfile >&"$log_fd" 2>&1  # then here, ctrl+c will only impact Python...
exec {log_fd}>&-                    # here we close the file & thus the copy of tee.

Of course, if you put those three commands in a script, that entire script becomes your foreground process, so different techniques are called for. Thus:

python3 -u myfile > >(trap '' INT; exec tee log.txt) 2>&1

Kill or terminate subprocess when timeout?

You could do something like this:

import subprocess as sub
import threading

class RunCmd(threading.Thread):
    def __init__(self, cmd, timeout):
        threading.Thread.__init__(self)
        self.cmd = cmd
        self.timeout = timeout

    def run(self):
        self.p = sub.Popen(self.cmd)
        self.p.wait()

    def Run(self):
        self.start()
        self.join(self.timeout)

        if self.is_alive():
            self.p.terminate()      #use self.p.kill() if process needs a kill -9
            self.join()

RunCmd(["./someProg", "arg1"], 60).Run()

The idea is that you create a thread that runs the command and to kill it if the timeout exceeds some suitable value, in this case 60 seconds.

SIGKILL to a subprocess tree on parent termination

Creating a subreaper is not useful in this case, your grandchildren would be reparented to and reaped by init anyway.

What you could do however is:

1. Start a parent process and fork a child immediately.
2. The parent will simply wait for the child.
3. The child will carry out all the work of your actual program, including spawning any other children via fork + execve.
4. Upon exit of the child for any reason (including deathly signals e.g. a crash) the parent can issue kill(0, SIGKILL) or killpg(getpgid(0), SIGKILL) to kill all the processes in its process group. Issuing a SIGINT/SIGTERM before SIGKILL would probably be a better idea depending on what child processes you want to run, as they could handle such signals and do a graceful cleanup of used resources (including children) before exiting.

Assuming that none of the children or grandchildren changes their process group while running, this will kill the entire tree of processes upon exit of your program. You could also keep the PR_SET_PDEATHSIG before any execve to make this more robust. Again depending on the processes you want to run a PR_SET_PDEATHSIG with SIGINT/SIGTERM could make more sense than SIGKILL.

You can issue setpgid(getpid(), 0) before doing any of the above to create a new process group for your program and avoid killing any parents when issuing kill(0, SIGKILL).

The logic of the "parent" process should be really simple, just a fork + wait in a loop + kill upon the right condition returned by wait. Of course, if this process crashes too then all bets are off, so take care in writing simple and reliable code.

Ensuring subprocesses are dead on exiting Python program

You can use atexit for this, and register any clean up tasks to be run when your program exits.

atexit.register(func[, *args[, **kargs]])

In your cleanup process, you can also implement your own wait, and kill it when a your desired timeout occurs.

>>> import atexit
>>> import sys
>>> import time
>>> 
>>> 
>>>
>>> def cleanup():
...     timeout_sec = 5
...     for p in all_processes: # list of your processes
...         p_sec = 0
...         for second in range(timeout_sec):
...             if p.poll() == None:
...                 time.sleep(1)
...                 p_sec += 1
...         if p_sec >= timeout_sec:
...             p.kill() # supported from python 2.6
...     print 'cleaned up!'
...
>>>
>>> atexit.register(cleanup)
>>>
>>> sys.exit()
cleaned up!

Note -- Registered functions won't be run if this process (parent process) is killed.

The following windows method is no longer needed for python >= 2.6

Here's a way to kill a process in windows. Your Popen object has a pid attribute, so you can just call it by success = win_kill(p.pid) (Needs pywin32 installed):

    def win_kill(pid):
        '''kill a process by specified PID in windows'''
        import win32api
        import win32con

        hProc = None
        try:
            hProc = win32api.OpenProcess(win32con.PROCESS_TERMINATE, 0, pid)
            win32api.TerminateProcess(hProc, 0)
        except Exception:
            return False
        finally:
            if hProc != None:
                hProc.Close()

        return True

---

Related Topics