Doesn't Sh Support Process Substitution <(...)

Doesn't sh support process substitution (...)?

The syntax <(...) is only supported by BASH.

For any POSIX shell, use this approach:

sh -c 'tail -n 1000 -F catalina.out | if grep -qP --line-buffered ".+" ; then ...'

i.e. move the stdin redirection in front of the if with a pipe. The if will pass stdin on to the grep.

if tail ...| grep won't work since the if won't be able to see it's then/fi because the pipe separates processes.

Why does process substitution not work in a shell script?

It isn't entirely clear yet, but the chances are very high that you either have an incorrect shebang line at the top of the script:

#!/bin/sh

or you are using sh script.sh instead of bash script.sh while testing it, or you have SHELL=/bin/sh or something similar set in the environment. Your failure is on the process substitution code. When Bash is run as sh (in POSIX mode), then process substitution is not available:

30. Process substitution is not available.

You need to write:

#!/bin/bash

temp=$(comm -12 <(sort -u /home/xyz/a.csv1) <(sort -u /home/abc/tempfile) | wc -l)
echo $temp

or even simply:

#!/bin/bash

comm -12 <(sort -u /home/xyz/a.csv1) <(sort -u /home/abc/tempfile) | wc -l

which will achieve the same effect as the capture followed by the echo. When testing, use bash -x script.sh or bash script.sh.

---

Deciphering the indecipherable comment

In an indecipherable comment, the information appears to include:

BASH=/bin/sh

BASHOPTS=cmdhist:extquote:force_fignore:hostcomplete:interactive_comments:progco‌mp:promptvars:sourcepath

BASH_ALIASES=()

BASH_ARGC=()

BASH_ARGV=()

BASH_CMDS=()

BASH_LINENO=([0]="0")

BASH_SOURCE=([0]="a.sh")

BASH_VERSINFO=([0]="4" [1]="1" [2]="2" [3]="1" [4]="release" [5]="x86_64-redhat-linux-gnu")

BASH_VERSION='4.1.2(1)-release'

CVS_RSH=ssh

SHELL=/bin/bash

SHELLOPTS=braceexpand:hashall:interactive-comments:posix

SHLVL=2

Note that BASH=/bin/sh and SHELLOPTS=braceexpand:hashall:interactive-comments:posix. Either or both of these might be a major part of the problem.

why process substitution does not always work with while loop in bash?

Note: <filename is not process substitution. It's a redirection. Process substitution has the format <(command).

Process substitution substitutes the name of a process for the <(...). Despite the use of the < symbol, it is not a redirect.

So when you say cat <(echo foo), bash creates a subprocess to run the echo command, and substitutes the name of a pseudo-file which can be read to get the output of that command. The result of the substitution will be something like this:

cat /dev/fd/63

Note the absence of a redirect. (You can see this in action by typing echo <(echo foo).)

Like many utilities, cat can be invoked with or without a command-line argument; if no file is specified, then it reads from stdin. So cat file.txt and cat < file.txt are very similar.

But the while command does not accept additional arguments. So

while read -r line; do echo "$line"; done < file.txt 

is valid, but

while read -r line; do echo "$line"; done file.txt

is a syntax error.

Process substitution doesn't change that. So

while read -r line; do echo "$line"; done /dev/fd/63

is a syntax error, and consequently so is

while read -r line; do echo "$line"; done <(echo foo)

To specify the redirect from the process substitution, you need a redirect:

while read -r line; do echo "$line"; done < <(echo foo)

Note that there must be a space between the two < symbols to avoid confusion with the "here-doc" syntax, <<word.

Syntax error in shell script with process substitution

The syntax you've used is a bash extension to the basic shell syntax, so you must take care to run your script with bash. (Ksh also has >(…) process substitution but doesn't support it after a redirection. Zsh would be fine.)

Given the error message you're getting, you are running this script in bash, but in its POSIX compatibility mode, not in full bash mode. Take care to invoke your script with an explicit #!/bin/bash line. #!/bin/sh won't do, even if /bin/sh is a symbolic link to bash, because bash runs in POSIX mode if it's invoked under the name sh. Always invoke bash by name if you use bash features.

Also take care not to set the environment variable POSIXLY_CORRECT or to pass the --posix option on the command line if you want to use bash features.

Alternatively, don't use this bash-specific syntax; use a portable construct such as the one proposed by Stephane Rouberol.

In bash, is it generally better to use process substitution or pipelines

tl;dr: Use pipes, unless you have a convincing reason not to.

Piping and redirecting stdin from a process substitution is essentially the same thing: both will result in two processes connected by an anonymous pipe.

There are three practical differences:

1. Bash defaults to creating a fork for every stage in a pipeline.

Which is why you started looking into this in the first place:

#!/bin/bash
cat "$1" | while IFS= read -r last; do true; done
echo "Last line of $1 is $last"

This script won't work by default with a pipelines, because unlike ksh and zsh, bash will fork a subshell for each stage.

If you set shopt -s lastpipe in bash 4.2+, bash mimics the ksh and zsh behavior and works just fine.

2. Bash does not wait for process substitutions to finish.

POSIX only requires a shell to wait for the last process in a pipeline, but most shells including bash will wait for all of them.

This makes a notable difference when you have a slow producer, like in a /dev/random password generator:

tr -cd 'a-zA-Z0-9' < /dev/random | head -c 10     # Slow?
head -c 10 < <(tr -cd 'a-zA-Z0-9' < /dev/random)  # Fast?

The first example will not benchmark favorably. Once head is satisfied and exits, tr will wait around for its next write() call to discover that the pipe is broken.

Since bash waits for both head and tr to finish, it will appear seem slower.

In the procsub version, bash only waits for head, and lets tr finish in the background.

3. Bash does not currently optimize away forks for single simple commands in process substitutions.

If you invoke an external command like sleep 1, then the Unix process model requires that bash forks and executes the command.

Since forks are expensive, bash optimizes the cases that it can. For example, the command:

bash -c 'sleep 1'

Would naively incur two forks: one to run bash, and one to run sleep. However, bash can optimize it because there's no need for bash to stay around after sleep finishes, so it can instead just replace itself with sleep (execve with no fork). This is very similar to tail call optimization.

( sleep 1 ) is similarly optimized, but <( sleep 1 ) is not. The source code does not offer a particular reason why, so it may just not have come up.

$ strace -f bash -c '/bin/true | /bin/true'     2>&1 | grep -c clone
2
$ strace -f bash -c '/bin/true < <(/bin/true)'  2>&1 | grep -c clone
3

---

Given the above you can create a benchmark favoring whichever position you want, but since the number of forks is generally much more relevant, pipes would be the best default.

And obviously, it doesn't hurt that pipes are the POSIX standard, canonical way of connecting stdin/stdout of two processes, and works equally well on all platforms.

POSIX shell equivalent to ()

mkfifo foo.fifo

## if your "commands" is multiple commands
# { commands ...; } >foo.fifo &

# otherwise, if it's just one
commands ... >foo.fifo &

something_else foo.fifo

is the closest available equivalent to

something_else <( commands ... )

sh -c and process substitution

sh is often dash not bash (see man sh).
dash doesn't do process substitution, only POSIX stuff.

You'll need to do:

  bash -c 'cat <(echo "hello")'

ksh & zsh can do process substitution too.

With your example, you can simply do:

watch -n 1  'iptables -L INPUT;  iptables -L FORWARD'

no need for an advanced shell or process subtitution.

shell module: Bash (process substitution) with ansible

EDIT: though this answers this the question I think mgsk's answer is a better answer since I agree that it's not the right way to go about it with Ansible.

This should fix your issue:

- name: install pip
  shell: "python <(curl https://bootstrap.pypa.io/get-pip.py)" executable=/bin/bash

If you are wondering the difference between these two commands:

python <(curl https://bootstrap.pypa.io/get-pip.py)
python <$(curl https://bootstrap.pypa.io/get-pip.py)

The first one uses process substitution which is a bash feature which is why you cannot use it with /bin/sh as your shell. What it's doing is taking the output of the curl command (which is a python script), writing it to a temporary file and using that file as an argument to python which takes a python script as its first argument.

The second one is an ambiguous redirect because the python script that is generated from the curl is not a file

cat (echo 'hello') can run, but can't run in shell script mode

It looks like you are using Bash-specific syntax in your script, so you should change the shebang line to something like:

#!/bin/bash

and if you want to invoke the program like you were doing you should run:

bash -x test.sh

syntax error near unexpected token `'

You get the error because process substitution (the <(some command) part) is not a standard feature (defined in POSIX) in sh, which means it may work on some OS but may not in others or in the same OS with different configuration.

You clarified that you have #!/bin/bash at the top of your script, but I guess you still run the script via sh foo.sh, as such, #!/bin/bash will be ignored and the script is interpreted by sh.

I assume your default shell is bash (run echo $SHELL), so all problems are gone if you paste the script in terminal and execute.

==== UPDATE ====

Possible solution if my assumption is correct:

Leave #!/bin/bash as it is, make your script an executable by chmod +x foo.sh. Then run it directly by ./foo.sh

---

Related Topics