How to Process Sigterm Signal Gracefully

How to process SIGTERM signal gracefully?

A class based clean to use solution:

import signal
import time

class GracefulKiller:
  kill_now = False
  def __init__(self):
    signal.signal(signal.SIGINT, self.exit_gracefully)
    signal.signal(signal.SIGTERM, self.exit_gracefully)

  def exit_gracefully(self, *args):
    self.kill_now = True

if __name__ == '__main__':
  killer = GracefulKiller()
  while not killer.kill_now:
    time.sleep(1)
    print("doing something in a loop ...")
   
  print("End of the program. I was killed gracefully :)")

How to process SIGTERM signal gracefully in Java?

I rewritten the registerShutdownHook() method and now it works as I wanted.

private static void registerShutdownHook() {
    final Thread mainThread = Thread.currentThread();
    Runtime.getRuntime().addShutdownHook(new Thread() {
        public void run() {
            try {
                System.out.println("Tralala");
                Hellow.setShutdownProcess();
                mainThread.join();
            } catch (InterruptedException ex) {
                System.out.println(ex);
            }

        }
    });  
}

How to process SIGTERM and still have a working process.terminate()

The solution very much depends on what platform you are running on as is often the case for Python questions tagged with [multiprocessing] and it is for that reason one is supposed also tag such questions with the specific platform, such as [linux], too. I am inferring that your platform is not Windows since signal.SIGQUIT is not defined for that platform. So I will go with Linux.

1. For Linux you do not want your subprocesses to handle the signals at all (and it's sort of nonsensical for them to be calling function term on an Ctrl-C interrupt, for example). For Windows, however, you want your subprocesses to ignore these interrupts. That means you want your main process to call signal only after it has created the subprocesses.
2. Instead of using FLAG to indicate that the main process should terminate and have to have the main process loop testing this value periodically, it is simpler, cleaner and more efficient to have the main process just wait on a threading.Event instance, done_event. Although. for some reason, this does not seem to work on Windows; the main process wait call does not get satisfied immediately.
3. You would like some provision to terminate gracefully if and when your processes complete normally and there has been so signal triggered. The easiest way to accomplish all your goals including this is to make your subprocesses daemon processes that will terminate when the main process terminates. Then create a daemon thread that simply waits for the subprocesses to normally terminate and sets done_event when that occurs. So the main process will fall through on the call to done_event.wait() on either an interrupt of some sort or normal completion. All it has to do now is just end normally; there is no need to call terminate against the subprocesses since they will end when the main process ends.

import multiprocessing as mp
from threading import Thread, Event
import signal
import time
import sys

IS_WINDOWS = sys.platform == 'win32'

def f(x):
    if IS_WINDOWS:
        signal.signal(signal.SIGTERM, signal.SIG_IGN)
        signal.signal(signal.SIGINT, signal.SIG_IGN)
        signal.signal(signal.SIGABRT, signal.SIG_IGN)

    time.sleep(5)
    print(x)
    return x * x

def term(signum, frame):
    print(f'Received Signal {signum}')
    if IS_WINDOWS:
        globals()['FLAG'] = True
    else:
        done_event.set()

def process_wait_thread():
    """
    wait for processes to finish normally and set done_event
    """
    for process in workers:
        process.join()

    if IS_WINDOWS:
        globals()['FLAG'] = True
    else:
        done_event.set()

if __name__ == '__main__':

    if IS_WINDOWS:
        globals()['FLAG'] = False
    else:
        done_event = Event()

    workers = [
        mp.Process(
            target=f,
            args=(i,),
            daemon=True
        ) for i in range(4)
    ]
    for process in workers:
        process.start()

    # We don't want subprocesses to inherit these so
    # call signal after we start the processes:
    signal.signal(signal.SIGTERM, term)
    signal.signal(signal.SIGINT, term)
    if not IS_WINDOWS:
        signal.signal(signal.SIGQUIT, term) # Not supported by Windows at all
    signal.signal(signal.SIGABRT, term)

    Thread(target=process_wait_thread, daemon=True).start()

    if IS_WINDOWS:
        while not globals()['FLAG']:
            time.sleep(0.1)
    else:
        done_event.wait()

    print('Done')

In what order should I send signals to gracefully shutdown processes?

SIGTERM tells an application to terminate. The other signals tell the application other things which are unrelated to shutdown but may sometimes have the same result. Don't use those. If you want an application to shut down, tell it to. Don't give it misleading signals.

Some people believe the smart standard way of terminating a process is by sending it a slew of signals, such as HUP, INT, TERM and finally KILL. This is ridiculous. The right signal for termination is SIGTERM and if SIGTERM doesn't terminate the process instantly, as you might prefer, it's because the application has chosen to handle the signal. Which means it has a very good reason to not terminate immediately: It's got cleanup work to do. If you interrupt that cleanup work with other signals, there's no telling what data from memory it hasn't yet saved to disk, what client applications are left hanging or whether you're interrupting it "mid-sentence" which is effectively data corruption.

For more information on what the real meaning of the signals is, see sigaction(2). Don't confuse "Default Action" with "Description", they are not the same thing.

SIGINT is used to signal an interactive "keyboard interrupt" of the process. Some programs may handle the situation in a special way for the purpose of terminal users.

SIGHUP is used to signal that the terminal has disappeared and is no longer looking at the process. That is all. Some processes choose to shut down in response, generally because their operation makes no sense without a terminal, some choose to do other things such as recheck configuration files.

SIGKILL is used to forcefully remove the process from the kernel. It is special in the sense that it's not actually a signal to the process but rather gets interpreted by the kernel directly.

Don't send SIGKILL. - SIGKILL should certainly never be sent by scripts. If the application handles the SIGTERM, it can take it a second to cleanup, it can take a minute, it can take an hour. Depending on what the application has to get done before it's ready to end. Any logic that "assumes" an application's cleanup sequence has taken long enough and needs to be shortcut or SIGKILLed after X seconds is just plain wrong.

The only reason why an application would need a SIGKILL to terminate, is if something bugged out during its cleanup sequence. In which case you can open a terminal and SIGKILL it manually. Aside from that, the only one other reason why you'd SIGKILL something is because you WANT to prevent it from cleaning itself up.

Even though half the world blindly sends SIGKILL after 5 seconds it's still horribly wrong thing to do.

How does a python process exit gracefully after receiving SIGTERM while waiting on a semaphore?

You can install a signal handler which throws an exception which is then caught in the subprocess to handle exits gracefully.

Here is an example of a script which waits in a semaphore in a subprocess and terminates gracefully when sent a SIGTERM.

#!/usr/bin/env python

import signal
import time
import multiprocessing

class GracefulExit(Exception):
    pass

def signal_handler(signum, frame):
    raise GracefulExit()

def subprocess_function():
    try:
        sem = multiprocessing.Semaphore()
        print "Acquiring semaphore"
        sem.acquire()
        print "Semaphore acquired"

        print "Blocking on semaphore - waiting for SIGTERM"
        sem.acquire()
    except GracefulExit:
        print "Subprocess exiting gracefully"

if __name__ == "__main__":

    # Use signal handler to throw exception which can be caught to allow
    # graceful exit.
    signal.signal(signal.SIGTERM, signal_handler)

    # Start a subprocess and wait for it to terminate.
    p = multiprocessing.Process(target=subprocess_function)
    p.start()

    print "Subprocess pid: %d" % p.pid

    p.join()

An example run of this script is as follows:

$ ./test.py 
Subprocess pid: 7546
Acquiring semaphore
Semaphore acquired
Blocking on semaphore - waiting for SIGTERM
----> Use another shell to kill -TERM 7546
Subprocess exiting gracefully

There is no traceback from the subprocess and the flow shows that the subprocess exits in a graceful manner. This is because the SIGTERM is caught by the subprocess signal handler which throws a normal Python exception which can be handled inside the process.

Allow process to finish rather than be interrupted when SIGTERM is used in Python 3

The answer is in the original question. I am just leaving this here for future Google searchers.

I never had an issue in the first place, my terminal was just having a problem printing 'ending' following the kill command.

---

Related Topics