When Did Hup Stop Getting Sent and What How to Do About It

Does linux kill background processes if we close the terminal from which it has started?

Who should kill jobs?

Normally, foreground and background jobs are killed by SIGHUP sent by kernel or shell in different circumstances.

When does kernel send SIGHUP?

Kernel sends SIGHUP to controlling process:

• for real (hardware) terminal: when disconnect is detected in a terminal driver, e.g. on hang-up on modem line;
• for pseudoterminal (pty): when last descriptor referencing master side of pty is closed, e.g. when you close terminal window.

Kernel sends SIGHUP to other process groups:

• to foreground process group, when controlling process terminates;
• to orphaned process group, when it becomes orphaned and it has stopped members.

Controlling process is the session leader that established the connection to the controlling terminal.

Typically, the controlling process is your shell. So, to sum up:

• kernel sends SIGHUP to the shell when real or pseudoterminal is disconnected/closed;
• kernel sends SIGHUP to foreground process group when the shell terminates;
• kernel sends SIGHUP to orphaned process group if it contains stopped processes.

Note that kernel does not send SIGHUP to background process group if it contains no stopped processes.

When does bash send SIGHUP?

Bash sends SIGHUP to all jobs (foreground and background):

• when it receives SIGHUP, and it is an interactive shell (and job control support is enabled at compile-time);
• when it exits, it is an interactive login shell, and huponexit option is set (and job control support is enabled at compile-time).

See more details here.

Notes:

• bash does not send SIGHUP to jobs removed from job list using disown;
• processes started using nohup ignore SIGHUP.

More details here.

What about other shells?

Usually, shells propagate SIGHUP. Generating SIGHUP at normal exit is less common.

Telnet or SSH

Under telnet or SSH, the following should happen when connection is closed (e.g. when you close telnet window on PC):

1. client is killed;
2. server detects that client connection is closed;
3. server closes master side of pty;
4. kernel detects that master pty is closed and sends SIGHUP to bash;
5. bash receives SIGHUP, sends SIGHUP to all jobs and terminates;
6. each job receives SIGHUP and terminates.

Problem

I can reproduce your issue using bash and telnetd from busybox or dropbear SSH server: sometimes, background job doesn't receive SIGHUP (and doesn't terminate) when client connection is closed.

It seems that a race condition occurs when server (telnetd or dropbear) closes master side of pty:

1. normally, bash receives SIGHUP and immediately kills background jobs (as expected) and terminates;
2. but sometimes, bash detects EOF on slave side of pty before handling SIGHUP.

When bash detects EOF, it by default terminates immediately without sending SIGHUP. And background job remains running!

Solution

It is possible to configure bash to send SIGHUP on normal exit (including EOF) too:

• Ensure that bash is started as login shell. The huponexit works only for login shells, AFAIK.

  Login shell is enabled by -l option or leading hyphen in argv[0]. You can configure telnetd to run /bin/bash -l or better /bin/login which invokes /bin/sh in login shell mode.

  E.g.:

  
  telnetd -l /bin/login
  

• Enable huponexit option.

  E.g.:

  
  shopt -s huponexit
  

  Type this in bash session every time or add it to .bashrc or /etc/profile.

Why does the race occur?

bash unblocks signals only when it's safe, and blocks them when some code section can't be safely interrupted by a signal handler.

Such critical sections invoke interruption points from time to time, and if signal is received when a critical section is executed, it's handler is delayed until next interruption point happens or critical section is exited.

You can start digging from quit.h in the source code.

Thus, it seems that in our case bash sometimes receives SIGHUP when it's in a critical section. SIGHUP handler execution is delayed, and bash reads EOF and terminates before exiting critical section or calling next interruption point.

Reference

• "Job Control" section in official Glibc manual.
• Chapter 34 "Process Groups, Sessions, and Job Control" of "The Linux Programming Interface" book.

How do I make a CGI::Fast based application kill -HUP aware?

You could use the block version of the eval function and the alarm function to add a timeout to Fast::CGI. In the code below I have set a five second timeout. If it times out it will go back to the start of the loop which will check to see if $continue has been set to zero yet. If it hasn't then it we start a new CGI::Fast object. This means that a maximum of five seconds will go by after you send a HUP before the code will start to stop if it was waiting for a new CGI::Fast object (the timeout has no affect on the rest of the loop).

my $continue = 1;
$SIG{HUP} = sub { $continue = 0 };
while ($continue) {
    my $cgi;
    eval {
        local $SIG{ALRM} = sub { die "timeout\n" };
        alarm 5; #set an alarm for five seconds
        $cgi = CGI::Fast->new;
        alarm 0; #turn alarm off
    };
    if ($@) {
        next if $@ eq "timeout\n"; #we will see the HUP signal's change now
        #died for a reason other than a timeout, so raise the error
        die $@;
    }
    last unless defined $cgi; #CGI::Fast has asked us to exit
    do_stuff($cgi);
}

#clean up

How to get the process ID to kill a nohup process?

When using nohup and you put the task in the background, the background operator (&) will give you the PID at the command prompt. If your plan is to manually manage the process, you can save that PID and use it later to kill the process if needed, via kill PID or kill -9 PID (if you need to force kill). Alternatively, you can find the PID later on by ps -ef | grep "command name" and locate the PID from there. Note that nohup keyword/command itself does not appear in the ps output for the command in question.

If you use a script, you could do something like this in the script:

nohup my_command > my.log 2>&1 &
echo $! > save_pid.txt

This will run my_command saving all output into my.log (in a script, $! represents the PID of the last process executed). The 2 is the file descriptor for standard error (stderr) and 2>&1 tells the shell to route standard error output to the standard output (file descriptor 1). It requires &1 so that the shell knows it's a file descriptor in that context instead of just a file named 1. The 2>&1 is needed to capture any error messages that normally are written to standard error into our my.log file (which is coming from standard output). See I/O Redirection for more details on handling I/O redirection with the shell.

If the command sends output on a regular basis, you can check the output occasionally with tail my.log, or if you want to follow it "live" you can use tail -f my.log. Finally, if you need to kill the process, you can do it via:

kill -9 `cat save_pid.txt`
rm save_pid.txt

Can we stop event hub capture for time being & re-enable?

Sure, you can disable the capture temporarily. However, make sure your retention period is long enough - 90 days for dedicated, that no data is purged w/o being captured.

See the capture description and the 'enabled' bool here > https://learn.microsoft.com/en-us/azure/templates/microsoft.eventhub/namespaces/eventhubs?tabs=bicep#capturedescription

See the overview of retention policy feature here > https://learn.microsoft.com/en-us/azure/event-hubs/event-hubs-features#event-retention

Related Topics