Running Shell Script in Parallel

Running shell script in parallel

Check out bash subshells, these can be used to run parts of a script in parallel.

I haven't tested this, but this could be a start:

#!/bin/bash
for i in $(seq 1 1000)
do
   ( Generating random numbers here , sorting  and outputting to file$i.txt ) &
   if (( $i % 10 == 0 )); then wait; fi # Limit to 10 concurrent subshells.
done
wait

How do you run multiple programs in parallel from a bash script?

To run multiple programs in parallel:

prog1 &
prog2 &

If you need your script to wait for the programs to finish, you can add:

wait

at the point where you want the script to wait for them.

Running Multiple Bash Scripts parallel

You could use xargs:

echo "~/path/test1.sh $1 ~/path/test2.sh $1" | xargs -P0 -n2 /bin/bash

-P0 says "run all in parallel"

-n2 passes two arguments to /bin/bash, in this case the script and the parameter

How to use parallel execution in a shell script?

Convert this into a Makefile with proper dependencies. Then you can use make -j to have Make run everything possible in parallel.

Note that all the indents in a Makefile must be TABs. TAB shows Make where the commands to run are.

Also note that this Makefile is now using GNU Make extensions (the wildcard and subst functions).

It might look like this:

export PATH := .:${PATH}

FILES=$(wildcard file*)
RFILES=$(subst file,r,${FILES})

final: combine ${RFILES}
    combine ${RFILES} final
    rm ${RFILES}

ex: example.c

combine: combine.c

r%: file% ex
    ex $< $@

Run shell script in parallel in bash/linux

You can just do

< tables.txt xargs -I% -n1 -P10 echo sqoop job --exec %

the -P10 will run 10 processes in parallel. And you don't even need the helper script.

As @CharlesDuffy commented, you don't need the -I, e.g. even simpler:

< tables.txt xargs -n1 -P10 echo sqoop job --exec

Run a script in parallel until another one finishes

You can use this loop:

#!/bin/bash

python A.py &

while [[ $(jobs -pr) ]]; do
    python B.py
done

jobs -pr lists the process IDs (-p) of running jobs (-r). If the output is empty, the backgrounds command has finished.

Run two shell script in parallel and capture their output

Running both scripts in parallel can look like this:

#!/bin/bash
#CONFIGURE SOME STUFF
$path/instance2_commands.sh >instance2.out 2>&1 &
$path/instance1_commands.sh >instance1.out 2>&1 &
wait

Notes:

• wait pauses until the children, instance1 and instance2, finish
• 2>&1 on each line redirects error messages to the relevant output file
• & at the end of a line causes the main script to continue running after forking, thereby producing a child that is executing that line of the script concurrently with the rest of the main script
• each script should send its output to a separate file. Sending both to the same file will be visually messy and impossible to sort out when the instances generate similar output messages.
• you may attempt to read the output files while the scripts are running with any reader, e.g. less instance1.out however output may be stuck in a buffer and not up-to-date. To fix that, the programs would have to open stdout in line buffered or unbuffered mode. It is also up to you to use -f or > to refresh the display.

Example D from an article on Apache Spark and parallel processing on my blog provides a similar shell script for calculating sums of a series for Pi on all cores, given a C program for calculating the sum on one core. This is a bit beyond the scope of the question, but I mention it in case you'd like to see a deeper example.

---

Related Topics