Does Nohup Work Across a Pipe

Does nohup work across a pipe?

No, you need to add the nohup to the commands separately.

Something like this is recommended:

nohup sh -c "cmd1 | cmd2" &

Or alternatively:

nohup $SHELL <<EOF &
cmd1 | cmd2
EOF

How do I use the nohup command without getting nohup.out?

The nohup command only writes to nohup.out if the output would otherwise go to the terminal. If you have redirected the output of the command somewhere else - including /dev/null - that's where it goes instead.

 nohup command >/dev/null 2>&1   # doesn't create nohup.out

Note that the >/dev/null 2>&1 sequence can be abbreviated to just >&/dev/null in most (but not all) shells.

If you're using nohup, that probably means you want to run the command in the background by putting another & on the end of the whole thing:

 nohup command >/dev/null 2>&1 & # runs in background, still doesn't create nohup.out

On Linux, running a job with nohup automatically closes its input as well. On other systems, notably BSD and macOS, that is not the case, so when running in the background, you might want to close input manually. While closing input has no effect on the creation or not of nohup.out, it avoids another problem: if a background process tries to read anything from standard input, it will pause, waiting for you to bring it back to the foreground and type something. So the extra-safe version looks like this:

nohup command </dev/null >/dev/null 2>&1 & # completely detached from terminal 

Note, however, that this does not prevent the command from accessing the terminal directly, nor does it remove it from your shell's process group. If you want to do the latter, and you are running bash, ksh, or zsh, you can do so by running disown with no argument as the next command. That will mean the background process is no longer associated with a shell "job" and will not have any signals forwarded to it from the shell. (A disowned process gets no signals forwarded to it automatically by its parent shell - but without nohup, it will still receive a HUP signal sent via other means, such as a manual kill command. A nohup'ed process ignores any and all HUP signals, no matter how they are sent.)

Explanation:

In Unixy systems, every source of input or target of output has a number associated with it called a "file descriptor", or "fd" for short. Every running program ("process") has its own set of these, and when a new process starts up it has three of them already open: "standard input", which is fd 0, is open for the process to read from, while "standard output" (fd 1) and "standard error" (fd 2) are open for it to write to. If you just run a command in a terminal window, then by default, anything you type goes to its standard input, while both its standard output and standard error get sent to that window.

But you can ask the shell to change where any or all of those file descriptors point before launching the command; that's what the redirection (<, <<, >, >>) and pipe (|) operators do.

The pipe is the simplest of these... command1 | command2 arranges for the standard output of command1 to feed directly into the standard input of command2. This is a very handy arrangement that has led to a particular design pattern in UNIX tools (and explains the existence of standard error, which allows a program to send messages to the user even though its output is going into the next program in the pipeline). But you can only pipe standard output to standard input; you can't send any other file descriptors to a pipe without some juggling.

The redirection operators are friendlier in that they let you specify which file descriptor to redirect. So 0<infile reads standard input from the file named infile, while 2>>logfile appends standard error to the end of the file named logfile. If you don't specify a number, then input redirection defaults to fd 0 (< is the same as 0<), while output redirection defaults to fd 1 (> is the same as 1>).

Also, you can combine file descriptors together: 2>&1 means "send standard error wherever standard output is going". That means that you get a single stream of output that includes both standard out and standard error intermixed with no way to separate them anymore, but it also means that you can include standard error in a pipe.

So the sequence >/dev/null 2>&1 means "send standard output to /dev/null" (which is a special device that just throws away whatever you write to it) "and then send standard error to wherever standard output is going" (which we just made sure was /dev/null). Basically, "throw away whatever this command writes to either file descriptor".

When nohup detects that neither its standard error nor output is attached to a terminal, it doesn't bother to create nohup.out, but assumes that the output is already redirected where the user wants it to go.

The /dev/null device works for input, too; if you run a command with </dev/null, then any attempt by that command to read from standard input will instantly encounter end-of-file. Note that the merge syntax won't have the same effect here; it only works to point a file descriptor to another one that's open in the same direction (input or output). The shell will let you do >/dev/null <&1, but that winds up creating a process with an input file descriptor open on an output stream, so instead of just hitting end-of-file, any read attempt will trigger a fatal "invalid file descriptor" error.

Nohup vs Nohup &

It would be better if you would understand what's going on.

nohup set's the process to ignore HUP signal. That's all. Nothing more nothing less. When does a process receive a HUP signal? Usually when a terminal logouts. And the default action on HUP signal is to terminate.

hitting "CTRL+C" just sends INT signal to the process. The default action (and you can trap "echo something" INT override it too) is to terminate the process.

nohup sh script.sh upon receiving INT signal will terminate (assuming script.sh didn't specially handle the INT signal) as it didn't set up a custom action on receiving a INT signal and it will ignore HUP signal.

The & placed after a command runs it in the background. As a separate process. So sh script.sh & runs sh in the background. The process will still terminate if you send it INT signal, just CTRL+C doesn't send it to that process, but to process that is in foreground. You can send it still using kill command. And the command will still terminate when the terminal exits, when the process receives the HUP signal.

So running nohup sh script.sh & will run the process in the background and ignore the signal that is send when the terminal exits. But still it will terminate on receiving INT signal. Just pressing CTRL+C in terminal will not send it to this process, as shell sends the term signal to the foreground process, not the background one.

Nohup take no effect when to close the terminal end the process running in background?

nohup command means "run command and ignore HUP signals".

So before we can effectively use nohup we need to ask: when and how is SIGHUP sent? As the Bash manual says, "Before exiting, an interactive shell resends the SIGHUP to all jobs, running or stopped.". It goes on to say that the proper way to suppress this behavior is with disown. I realize you are asking about nohup, but it's worth calling out that disown is the intended and easier way to accomplish what you want. Note that disown is not equivalent to nohup.

The reason nohup is tricky to work with here is because it applies to a single process, whereas & creates a background job of a whole command pipeline, which can consist of multiple processes. This means you need to nohup each command in the pipeline in order to ensure that the individual commands don't receive a SIGHUP, e.g.:

$ nohup ls Music/*mp3 2>/dev/null | nohup xargs -d "\n" nohup mplayer &> /dev/null &

This should work, though I haven't tested it with these specific commands. If it doesn't, it's likely another process that you aren't starting directly is still receiving a SIGHUP. This is harder to address, which is exactly why we have disown.

manishg's suggestion is also reasonable; by moving the pipeline into a separate process you can nohup that process, which should in turn prevent a SIGHUP from reaching its children when your shell closes.

---

All that said, you don't need ls and xargs here in the first place; find can be used to similar effect and will simplify reasoning about the command. Try:

$ nohup find Music -maxdepth 1 -name '*mp3' -exec mplayer {} + &> /dev/null &

Why can't I use Unix Nohup with Bash For-loop?

Because 'nohup' expects a single-word command and its arguments - not a shell loop construct. You'd have to use:

nohup sh -c 'for i in mydir/*.fasta; do ./myscript.sh "$i"; done >output.txt' &

Nohup does not work with bash for loop properly

IMHO sh is not an alias of bash on your system.
If you start it like this:

nohup bash -c 'for i in {0..9}; do spark-submit --class some.Code /some/Jar.jar --input_path /some/path/part-001"$i"* > test_log.log; done' &

it will get working.
As far as I can remember, the used syntax is the bash's own.

Ps: but I'm not sure, if the " marks around $i are needed. I prefer ${i} form.

Edit: if on your system sh is an alias of dash, then you could use this:

nohup sh -c 'for i in `seq 0 9`; do spark-submit --class some.Code /some/Jar.jar --input_path /some/path/part-001"$i"* > test_log.log; done &'

Running a nohup command through SSH with Python's Popen then logging out

From the docs: Python Docs

Warning Use communicate() rather than .stdin.write, .stdout.read or .stderr.read to avoid deadlocks due to any of the other OS pipe buffers filling up and blocking the child process.

Alright pretty sure i got it now:

About communicate(): [...]Wait for process to terminate
So i guess your earlier Solution was better. But if you call it at the end if shouldn't be a problem or just don't call it at all if you don't need stdin or stderr as output

However according to this:StackOverflow Comment If you set preexec_fn=os.setpgrp in your Popen call it should work.

How do I ask nohup not to redirect stderr to stdout if it is detached from terminal?

That error redirection actually applies to the gzip command, not /.a.out. What you're looking for is ./a.out 2> log.txt | gzip -c > result.txt.gz.

Pass the argument to if condition used in pipe

What you need is to convert the if statement into an argument to bash. Doing a simplistic transform, assuming that the code in the question has a chance of working, you get:

tail -f nohup.out |
xargs -I '{}' bash -c "if [[ {} == *"denied"* ]]; then dig -x $(cut -d '-' -f 6 {} | cut -d ':' -f 1) & fi"

This is exactly the same basic treatment as was needed for a for loop being executed by nohup — you need a shell to run the built-in command. See Why can't I use Unix nohup with Bash for loop? for an exactly analogous situation.

However, on further reflection, you want to cut the string which is the IP address, not the file with that as a name, so the command needs to echo the string into the cut commands. You also have to tear your hair getting the sub-commands executed correctly; you need a backslash before the $ of $(…), or before each of the back-ticks if you insist on using `…` notation, as well as using backslash-double-quote to protect the angle-brackets in the string.

tail -f nohup.out |
xargs -I '{}' bash -c "if [[ '{}' != *denied* ]]; then echo dig -x \"\$(echo '{}' | cut -d '-' -f 6 | cut -d ':' -f 1)\" & fi"

Now we need to debate the use of the condition and two cut commands (and the general hair loss). You could use:

tail -f nohup.out |
grep -v denied |
xargs -I '{}' bash -c "echo dig -x \$(echo '{}' | cut -d '-' -f 6 | cut -d ':' -f 1) &"

or, more sensibly:

tail -f nohup.out |
awk -F '[-:]' '/denied/ { next } { print "dig -x " $7 " &" }' |
sh -x

or any of a myriad other ways to do it.

---

Related Topics