Does linux kill background processes if we close the terminal from which it has started?
Who should kill jobs?
Normally, foreground and background jobs are killed by SIGHUP
sent by kernel or shell in different circumstances.
When does kernel send SIGHUP
?
Kernel sends SIGHUP
to controlling process:
- for real (hardware) terminal: when disconnect is detected in a terminal driver, e.g. on hang-up on modem line;
- for pseudoterminal (pty): when last descriptor referencing master side of pty is closed, e.g. when you close terminal window.
Kernel sends SIGHUP
to other process groups:
- to foreground process group, when controlling process terminates;
- to orphaned process group, when it becomes orphaned and it has stopped members.
Controlling process is the session leader that established the connection to the controlling terminal.
Typically, the controlling process is your shell. So, to sum up:
- kernel sends
SIGHUP
to the shell when real or pseudoterminal is disconnected/closed; - kernel sends
SIGHUP
to foreground process group when the shell terminates; - kernel sends
SIGHUP
to orphaned process group if it contains stopped processes.
Note that kernel does not send SIGHUP
to background process group if it contains no stopped processes.
When does bash
send SIGHUP
?
Bash sends SIGHUP
to all jobs (foreground and background):
- when it receives
SIGHUP
, and it is an interactive shell (and job control support is enabled at compile-time); - when it exits, it is an interactive login shell, and
huponexit
option is set (and job control support is enabled at compile-time).
See more details here.
Notes:
bash
does not sendSIGHUP
to jobs removed from job list usingdisown
;- processes started using
nohup
ignoreSIGHUP
.
More details here.
What about other shells?
Usually, shells propagate SIGHUP
. Generating SIGHUP
at normal exit is less common.
Telnet or SSH
Under telnet or SSH, the following should happen when connection is closed (e.g. when you close telnet
window on PC):
- client is killed;
- server detects that client connection is closed;
- server closes master side of pty;
- kernel detects that master pty is closed and sends
SIGHUP
tobash
; bash
receivesSIGHUP
, sendsSIGHUP
to all jobs and terminates;- each job receives
SIGHUP
and terminates.
Problem
I can reproduce your issue using bash
and telnetd
from busybox
or dropbear
SSH server: sometimes, background job doesn't receive SIGHUP
(and doesn't terminate) when client connection is closed.
It seems that a race condition occurs when server (telnetd
or dropbear
) closes master side of pty:
- normally,
bash
receivesSIGHUP
and immediately kills background jobs (as expected) and terminates; - but sometimes,
bash
detectsEOF
on slave side of pty before handlingSIGHUP
.
When bash
detects EOF
, it by default terminates immediately without sending SIGHUP
. And background job remains running!
Solution
It is possible to configure bash
to send SIGHUP
on normal exit (including EOF
) too:
Ensure that
bash
is started as login shell. Thehuponexit
works only for login shells, AFAIK.Login shell is enabled by
-l
option or leading hyphen inargv[0]
. You can configuretelnetd
to run/bin/bash -l
or better/bin/login
which invokes/bin/sh
in login shell mode.E.g.:
telnetd -l /bin/loginEnable
huponexit
option.E.g.:
shopt -s huponexitType this in
bash
session every time or add it to.bashrc
or/etc/profile
.
Why does the race occur?
bash
unblocks signals only when it's safe, and blocks them when some code section can't be safely interrupted by a signal handler.
Such critical sections invoke interruption points from time to time, and if signal is received when a critical section is executed, it's handler is delayed until next interruption point happens or critical section is exited.
You can start digging from quit.h
in the source code.
Thus, it seems that in our case bash
sometimes receives SIGHUP
when it's in a critical section. SIGHUP
handler execution is delayed, and bash
reads EOF
and terminates before exiting critical section or calling next interruption point.
Reference
- "Job Control" section in official Glibc manual.
- Chapter 34 "Process Groups, Sessions, and Job Control" of "The Linux Programming Interface" book.
How do I kill background processes / jobs when my shell script exits?
To clean up some mess, trap
can be used. It can provide a list of stuff executed when a specific signal arrives:
trap "echo hello" SIGINT
but can also be used to execute something if the shell exits:
trap "killall background" EXIT
It's a builtin, so help trap
will give you information (works with bash). If you only want to kill background jobs, you can do
trap 'kill $(jobs -p)' EXIT
Watch out to use single '
, to prevent the shell from substituting the $()
immediately.
Start a process in background, do a task, then kill the process in the background
You can probably save the PID of the process in a variable, then use the kill command to kill it.
pushd .
nohup java -jar test/selenium-server.jar > /dev/null 2>&1 &
serverPID=$!
cd web/code/protected/tests/
phpunit functional/
kill $serverPID
popd
I haven't tested it myself, I'd like to write it on a comment, but not enough reputation yet :)
linux: kill background task
There's a special variable for this in bash:
kill $!
$! expands to the PID of the last process executed in the background.
How to prevent a background process from being stopped after closing SSH client in Linux
Check out the "nohup" program.
What killed my process and why?
If the user or sysadmin did not kill the program the kernel may have. The kernel would only kill a process under exceptional circumstances such as extreme resource starvation (think mem+swap exhaustion).
Related Topics
Create Iptables Rule Per Process/Service
What Context Does the Scheduler Code Run In
Installing Openssh on the Alpine Docker Container
Linux Makefile Structure and Documentation
Error: "Grep: Argument List Too Long"
How to Gzip All Files in All Sub-Directories into One Compressed File in Bash
How to Test for If Two Files Exist
What Is Active Memory and Inactive Memory
Linux Free Shows High Memory Usage But Top Does Not
Check If File Exists on Remote Host with Ssh
Run a Command at a Specific Time
How to Know the Interrupt/Gpio Number for a Specific Pin in Linux
Lsb_Release: Command Not Found in Latest Ubuntu Docker Container
Find and Replace in Shell Scripting
How to Reconfigure Oracle 10G Xe on Linux
How to Format My Grep Output to Show Line Numbers at the End of the Line, and Also the Hit Count
How to Calculate System Memory Usage from /Proc/Meminfo (Like Htop)