Bash Script to Run a Constant Number of Jobs in the Background

bash script to run a constant number of jobs in the background

With GNU xargs:

printf '%s\0' j{1..6} | xargs -0 -n1 -P3 sh -c './"$1"' _

---

With bash (4.x) builtins:

max_jobs=3; cur_jobs=0
for ((i=0; i<6; i++)); do
  # If true, wait until the next background job finishes to continue.
  ((cur_jobs >= max_jobs)) && wait -n
  # Increment the current number of jobs running.
  ./j"$i" & ((++cur_jobs))
done
wait

Note that the approach relying on builtins has some corner cases -- if you have multiple jobs exiting at the exact same time, a single wait -n can reap several of them, thus effectively consuming multiple slots. If we wanted to be more robust, we might end up with something like the following:

max_jobs=3
declare -A cur_jobs=( ) # build an associative array w/ PIDs of jobs we started
for ((i=0; i<6; i++)); do
  if (( ${#cur_jobs[@]} >= max_jobs )); then
    wait -n # wait for at least one job to exit
    # ...and then remove any jobs that aren't running from the table
    for pid in "${!cur_jobs[@]}"; do
      kill -0 "$pid" 2>/dev/null && unset cur_jobs[$pid]
    done
  fi
  ./j"$i" & cur_jobs[$!]=1
done
wait

...which is obviously a lot of work, and still has a minor race. Consider using xargs -P instead. :)

How to get the correct number of background jobs running, f.ex. $(jobs | wc -l | xargs) returns 1 instead of 0

This fixes it:

function waitForUploadFinish() {
  runningJobs=$(jobs | wc -l | xargs)
  echo "Waiting for ${runningJobs} background upload jobs to finish"

  while [ `jobs -r | wc -l | tr -d " "` != 0 ]; do 
    jobs -r | wc -l | tr -d " "
    echo -n "."     # no trailing newline
    sleep 1
  done
  echo ""
}

Note: you will only count the background processes that are started by this bash script, you will not see the background processes from the starting shell.

As the gniourf_gniourf commented: if you only need to wait and don't need to output then a simple wait after the sleeps is much simpler.

for i in {1..3}; do
  sleep $i &
done

wait

Wait for bash background jobs in script to be finished

There's a bash builtin command for that.

wait [n ...]
      Wait for each specified process and return its termination  sta‐
      tus.   Each  n  may be a process ID or a job specification; if a
      job spec is given, all processes  in  that  job’s  pipeline  are
      waited  for.  If n is not given, all currently active child pro‐
      cesses are waited for, and the return  status  is  zero.   If  n
      specifies  a  non-existent  process or job, the return status is
      127.  Otherwise, the return status is the  exit  status  of  the
      last process or job waited for.

Bash: limit the number of concurrent jobs?

If you have GNU Parallel http://www.gnu.org/software/parallel/ installed you can do this:

parallel gzip ::: *.log

which will run one gzip per CPU core until all logfiles are gzipped.

If it is part of a larger loop you can use sem instead:

for i in *.log ; do
    echo $i Do more stuff here
    sem -j+0 gzip $i ";" echo done
done
sem --wait

It will do the same, but give you a chance to do more stuff for each file.

If GNU Parallel is not packaged for your distribution you can install GNU Parallel simply by:

$ (wget -O - pi.dk/3 || lynx -source pi.dk/3 || curl pi.dk/3/ || \
   fetch -o - http://pi.dk/3 ) > install.sh
$ sha1sum install.sh | grep 883c667e01eed62f975ad28b6d50e22a
12345678 883c667e 01eed62f 975ad28b 6d50e22a
$ md5sum install.sh | grep cc21b4c943fd03e93ae1ae49e28573c0
cc21b4c9 43fd03e9 3ae1ae49 e28573c0
$ sha512sum install.sh | grep da012ec113b49a54e705f86d51e784ebced224fdf
79945d9d 250b42a4 2067bb00 99da012e c113b49a 54e705f8 6d51e784 ebced224
fdff3f52 ca588d64 e75f6033 61bd543f d631f592 2f87ceb2 ab034149 6df84a35
$ bash install.sh

It will download, check signature, and do a personal installation if it cannot install globally.

Watch the intro videos for GNU Parallel to learn more:
https://www.youtube.com/playlist?list=PL284C9FF2488BC6D1

automatically running a bash script in the background

$ ./script &
           ^--- run script in background from the get-go

Loop background job

Remove the ; after sleep

for i in $(seq 3); do echo $i ; sleep 2 & done

BTW, such loops are better written on separate lines with proper indentation (if you are writing this in a shell script file).

for i in $(seq 3)
do
   echo $i
   sleep 2 &
done

Running multiple background processes through shell scripting

The && in your script seem a bit confused. For the record:

• Put a single & after a command to run it in the background
• && is for chaining multiple commands if successful, for example cmd1 && cmd2 will execute cmd1 and only if it exits with success, it will execute cmd2. Both commands will run in the foreground, there is no backgrounding here at all.

Maybe you want to do something like this:

echo "Launching server1"
java StartServer1.jar >server1.log 2>server1.err
sleep 5  # give some time for the server to come up

serverCommand1
serverCommand2

echo "Launching server2"
java StartServer2.jar >server2.log 2>server2.err
sleep 5  # give some time for the server to come up

echo "Running script"
python scriptRun.py

Actually, rather than sleeping for a fixed amount of time, it's better if you can detect that the server is ready and react on that. For example in the logs, maybe there is a message indicating that the server is ready, let's say a message that says "READY". Then you can do something like this:

echo "Launching server1"
java StartServer1.jar >server1.log 2>server1.err
while :; do sleep 5; grep -q READY server1.log && break; done

That's an infinite loop there, in every sleep it sleeps for 5 seconds and checks if the log contains the text "READY". If it does it ends the loop. You can come up with a variation of this that suits your needs.

---

Related Topics