How Does Nohup Work

What's the difference between nohup and ampersand

nohup catches the hangup signal (see man 7 signal) while the ampersand doesn't (except the shell is confgured that way or doesn't send SIGHUP at all).

Normally, when running a command using & and exiting the shell afterwards, the shell will terminate the sub-command with the hangup signal (kill -SIGHUP <pid>). This can be prevented using nohup, as it catches the signal and ignores it so that it never reaches the actual application.

In case you're using bash, you can use the command shopt | grep hupon to find out whether
your shell sends SIGHUP to its child processes or not. If it is off, processes won't be
terminated, as it seems to be the case for you. More information on how bash terminates
applications can be found here.

There are cases where nohup does not work, for example when the process you start reconnects
the SIGHUP signal, as it is the case here.

How does nohup work?

Use the source, Luke!

(Extra characters added to keep the new answer length rules happy.)

Does nohup work across a pipe?

No, you need to add the nohup to the commands separately.

Something like this is recommended:

nohup sh -c "cmd1 | cmd2" &

Or alternatively:

nohup $SHELL <<EOF &
cmd1 | cmd2
EOF

How do I use the nohup command without getting nohup.out?

The nohup command only writes to nohup.out if the output would otherwise go to the terminal. If you have redirected the output of the command somewhere else - including /dev/null - that's where it goes instead.

 nohup command >/dev/null 2>&1   # doesn't create nohup.out

Note that the >/dev/null 2>&1 sequence can be abbreviated to just >&/dev/null in most (but not all) shells.

If you're using nohup, that probably means you want to run the command in the background by putting another & on the end of the whole thing:

 nohup command >/dev/null 2>&1 & # runs in background, still doesn't create nohup.out

On Linux, running a job with nohup automatically closes its input as well. On other systems, notably BSD and macOS, that is not the case, so when running in the background, you might want to close input manually. While closing input has no effect on the creation or not of nohup.out, it avoids another problem: if a background process tries to read anything from standard input, it will pause, waiting for you to bring it back to the foreground and type something. So the extra-safe version looks like this:

nohup command </dev/null >/dev/null 2>&1 & # completely detached from terminal

Note, however, that this does not prevent the command from accessing the terminal directly, nor does it remove it from your shell's process group. If you want to do the latter, and you are running bash, ksh, or zsh, you can do so by running disown with no argument as the next command. That will mean the background process is no longer associated with a shell "job" and will not have any signals forwarded to it from the shell. (A disowned process gets no signals forwarded to it automatically by its parent shell - but without nohup, it will still receive a HUP signal sent via other means, such as a manual kill command. A nohup'ed process ignores any and all HUP signals, no matter how they are sent.)

Explanation:

In Unixy systems, every source of input or target of output has a number associated with it called a "file descriptor", or "fd" for short. Every running program ("process") has its own set of these, and when a new process starts up it has three of them already open: "standard input", which is fd 0, is open for the process to read from, while "standard output" (fd 1) and "standard error" (fd 2) are open for it to write to. If you just run a command in a terminal window, then by default, anything you type goes to its standard input, while both its standard output and standard error get sent to that window.

But you can ask the shell to change where any or all of those file descriptors point before launching the command; that's what the redirection (<, <<, >, >>) and pipe (|) operators do.

The pipe is the simplest of these... command1 | command2 arranges for the standard output of command1 to feed directly into the standard input of command2. This is a very handy arrangement that has led to a particular design pattern in UNIX tools (and explains the existence of standard error, which allows a program to send messages to the user even though its output is going into the next program in the pipeline). But you can only pipe standard output to standard input; you can't send any other file descriptors to a pipe without some juggling.

The redirection operators are friendlier in that they let you specify which file descriptor to redirect. So 0<infile reads standard input from the file named infile, while 2>>logfile appends standard error to the end of the file named logfile. If you don't specify a number, then input redirection defaults to fd 0 (< is the same as 0<), while output redirection defaults to fd 1 (> is the same as 1>).

Also, you can combine file descriptors together: 2>&1 means "send standard error wherever standard output is going". That means that you get a single stream of output that includes both standard out and standard error intermixed with no way to separate them anymore, but it also means that you can include standard error in a pipe.

So the sequence >/dev/null 2>&1 means "send standard output to /dev/null" (which is a special device that just throws away whatever you write to it) "and then send standard error to wherever standard output is going" (which we just made sure was /dev/null). Basically, "throw away whatever this command writes to either file descriptor".

When nohup detects that neither its standard error nor output is attached to a terminal, it doesn't bother to create nohup.out, but assumes that the output is already redirected where the user wants it to go.

The /dev/null device works for input, too; if you run a command with </dev/null, then any attempt by that command to read from standard input will instantly encounter end-of-file. Note that the merge syntax won't have the same effect here; it only works to point a file descriptor to another one that's open in the same direction (input or output). The shell will let you do >/dev/null <&1, but that winds up creating a process with an input file descriptor open on an output stream, so instead of just hitting end-of-file, any read attempt will trigger a fatal "invalid file descriptor" error.

Nohup vs Nohup &

It would be better if you would understand what's going on.

nohup set's the process to ignore HUP signal. That's all. Nothing more nothing less. When does a process receive a HUP signal? Usually when a terminal logouts. And the default action on HUP signal is to terminate.

hitting "CTRL+C" just sends INT signal to the process. The default action (and you can trap "echo something" INT override it too) is to terminate the process.

nohup sh script.sh upon receiving INT signal will terminate (assuming script.sh didn't specially handle the INT signal) as it didn't set up a custom action on receiving a INT signal and it will ignore HUP signal.

The & placed after a command runs it in the background. As a separate process. So sh script.sh & runs sh in the background. The process will still terminate if you send it INT signal, just CTRL+C doesn't send it to that process, but to process that is in foreground. You can send it still using kill command. And the command will still terminate when the terminal exits, when the process receives the HUP signal.

So running nohup sh script.sh & will run the process in the background and ignore the signal that is send when the terminal exits. But still it will terminate on receiving INT signal. Just pressing CTRL+C in terminal will not send it to this process, as shell sends the term signal to the foreground process, not the background one.

Difference between nohup and ctrl+z command in Linux

Running your process as a job constrains your job to the current session. So if you run

$ start_running_job &
$ exit

The job will stop when you exit.

Pressing Control-Z has the same effect as the lines above.

Running nohup places the job so that it survives the end of the current session. So if you run

$ nohup start_running_job &
$ exit

The job will continue running.

Does nohup work as long as the server is running?

Yes. As long as the server is running, you can close out of your SSH connection and your program will continue to run with the nohup command. If your server reboots, for whatever reason, you should consider using respawn.

What are the differences between nohup and bg?

Processes backgrounded with 'bg' are still tied to the shell that launched them and will close when the shell closes. Processes detached with 'nohup' will run even after you log out.