Job -l after nohup
You need to understand the difference between a process and a job. Jobs are managed by the shell, so when you end your terminal session and start a new one, you are now in a new instance of Bash with its own jobs table. You can't access jobs from the original shell but as the other answers have noted, you can still find and manipulate the processes that were started. For example:
$ nohup sleep 60 &
[1] 27767
# Our job is in the jobs table
$ jobs
[1]+ Running nohup sleep 60 &
# And this is the process we started
$ ps -p 27767
PID TTY TIME CMD
27767 pts/1 00:00:00 sleep
$ exit # and start a new session
# Now jobs returns nothing because the jobs table is empty
$ jobs
# But our process is still alive and kicking...
$ ps -p 27767
PID TTY TIME CMD
27767 pts/1 00:00:00 sleep
# Until we decide to kill it
$ kill 27767
# Now the process is gone
$ ps -p 27767
PID TTY TIME CMD
Nohup take no effect when to close the terminal end the process running in background?
nohup command
means "run command
and ignore HUP signals".
So before we can effectively use nohup
we need to ask: when and how is SIGHUP
sent? As the Bash manual says, "Before exiting, an interactive shell resends the SIGHUP to all jobs, running or stopped.". It goes on to say that the proper way to suppress this behavior is with disown
. I realize you are asking about nohup
, but it's worth calling out that disown
is the intended and easier way to accomplish what you want. Note that disown
is not equivalent to nohup
.
The reason nohup
is tricky to work with here is because it applies to a single process, whereas &
creates a background job of a whole command pipeline, which can consist of multiple processes. This means you need to nohup
each command in the pipeline in order to ensure that the individual commands don't receive a SIGHUP, e.g.:
$ nohup ls Music/*mp3 2>/dev/null | nohup xargs -d "\n" nohup mplayer &> /dev/null &
This should work, though I haven't tested it with these specific commands. If it doesn't, it's likely another process that you aren't starting directly is still receiving a SIGHUP. This is harder to address, which is exactly why we have disown
.
manishg's suggestion is also reasonable; by moving the pipeline into a separate process you can nohup
that process, which should in turn prevent a SIGHUP from reaching its children when your shell closes.
All that said, you don't need ls
and xargs
here in the first place; find
can be used to similar effect and will simplify reasoning about the command. Try:
$ nohup find Music -maxdepth 1 -name '*mp3' -exec mplayer {} + &> /dev/null &
Does linux kill background processes if we close the terminal from which it has started?
Who should kill jobs?
Normally, foreground and background jobs are killed by SIGHUP
sent by kernel or shell in different circumstances.
When does kernel send SIGHUP
?
Kernel sends SIGHUP
to controlling process:
- for real (hardware) terminal: when disconnect is detected in a terminal driver, e.g. on hang-up on modem line;
- for pseudoterminal (pty): when last descriptor referencing master side of pty is closed, e.g. when you close terminal window.
Kernel sends SIGHUP
to other process groups:
- to foreground process group, when controlling process terminates;
- to orphaned process group, when it becomes orphaned and it has stopped members.
Controlling process is the session leader that established the connection to the controlling terminal.
Typically, the controlling process is your shell. So, to sum up:
- kernel sends
SIGHUP
to the shell when real or pseudoterminal is disconnected/closed; - kernel sends
SIGHUP
to foreground process group when the shell terminates; - kernel sends
SIGHUP
to orphaned process group if it contains stopped processes.
Note that kernel does not send SIGHUP
to background process group if it contains no stopped processes.
When does bash
send SIGHUP
?
Bash sends SIGHUP
to all jobs (foreground and background):
- when it receives
SIGHUP
, and it is an interactive shell (and job control support is enabled at compile-time); - when it exits, it is an interactive login shell, and
huponexit
option is set (and job control support is enabled at compile-time).
See more details here.
Notes:
bash
does not sendSIGHUP
to jobs removed from job list usingdisown
;- processes started using
nohup
ignoreSIGHUP
.
More details here.
What about other shells?
Usually, shells propagate SIGHUP
. Generating SIGHUP
at normal exit is less common.
Telnet or SSH
Under telnet or SSH, the following should happen when connection is closed (e.g. when you close telnet
window on PC):
- client is killed;
- server detects that client connection is closed;
- server closes master side of pty;
- kernel detects that master pty is closed and sends
SIGHUP
tobash
; bash
receivesSIGHUP
, sendsSIGHUP
to all jobs and terminates;- each job receives
SIGHUP
and terminates.
Problem
I can reproduce your issue using bash
and telnetd
from busybox
or dropbear
SSH server: sometimes, background job doesn't receive SIGHUP
(and doesn't terminate) when client connection is closed.
It seems that a race condition occurs when server (telnetd
or dropbear
) closes master side of pty:
- normally,
bash
receivesSIGHUP
and immediately kills background jobs (as expected) and terminates; - but sometimes,
bash
detectsEOF
on slave side of pty before handlingSIGHUP
.
When bash
detects EOF
, it by default terminates immediately without sending SIGHUP
. And background job remains running!
Solution
It is possible to configure bash
to send SIGHUP
on normal exit (including EOF
) too:
Ensure that
bash
is started as login shell. Thehuponexit
works only for login shells, AFAIK.Login shell is enabled by
-l
option or leading hyphen inargv[0]
. You can configuretelnetd
to run/bin/bash -l
or better/bin/login
which invokes/bin/sh
in login shell mode.E.g.:
telnetd -l /bin/loginEnable
huponexit
option.E.g.:
shopt -s huponexitType this in
bash
session every time or add it to.bashrc
or/etc/profile
.
Why does the race occur?
bash
unblocks signals only when it's safe, and blocks them when some code section can't be safely interrupted by a signal handler.
Such critical sections invoke interruption points from time to time, and if signal is received when a critical section is executed, it's handler is delayed until next interruption point happens or critical section is exited.
You can start digging from quit.h
in the source code.
Thus, it seems that in our case bash
sometimes receives SIGHUP
when it's in a critical section. SIGHUP
handler execution is delayed, and bash
reads EOF
and terminates before exiting critical section or calling next interruption point.
Reference
- "Job Control" section in official Glibc manual.
- Chapter 34 "Process Groups, Sessions, and Job Control" of "The Linux Programming Interface" book.
When did HUP stop getting sent and what can I do about it?
I believe you're looking for the huponexit
shell option. You can set this easily with
$ shopt -s huponexit
Some details from the bash man page:
The shell exits by default upon receipt of a SIGHUP. Before
exiting, an interactive shell resends the SIGHUP to all jobs, running
or
stopped. Stopped jobs are sent SIGCONT to ensure that they receive the SIGHUP. To prevent the shell from sending the signal to
a par-
ticular job, it should be removed from the jobs table with the disown builtin (see SHELL BUILTIN COMMANDS below) or marked to not
receive
SIGHUP using disown -h.If the huponexit shell option has been set with shopt, bash sends a SIGHUP to all jobs when an interactive login shell exits.
Any way to exit bash script, but not quitting the terminal
The "problem" really is that you're sourcing and not executing the script. When you source a file, its contents will be executed in the current shell, instead of spawning a subshell. So everything, including exit, will affect the current shell.
Instead of using exit
, you will want to use return
.
How to prevent a background process from being stopped after closing SSH client in Linux
Check out the "nohup" program.
Related Topics
Dependency Walker Equivalent for Linux
Debugging Shared Libraries with Gdbserver
How to Find the Main Function's Entry Point of Elf Executable File Without Any Symbolic Information
Force Gnu Linker to Generate 32 Bit Elf Executables
Gdb Does Not Hit Any Breakpoints When I Run It from Inside Docker Container
How to Receive the Wrong Ethernet Frames and Disable the Crc/Fcs Calcul
Curl Simple File Upload - 417 Expectation Failed
How to Prevent a Linux User Space Pthread Yielding in Critical Code
Detecting Command Not Found in Bash Script
How to Execve a Process, Retaining Capabilities in Spite of Missing Filesystem-Based Capabilities
How to Use Local and Remote Variables Within a Heredoc or Command Over Ssh
How to Write Shell Command Within Pharo Smalltalk
How to Test If an Address Is Readable in Linux Userspace App