Preventing to Bash Script from Running in Parallel or Overlap Using Cron

Preventing to Bash script from running in parallel or overlap using cron

Just putting some flesh on top of the suggestions already there in the comments.

Put in the beginning of java_prog1.sh:

[ -f /var/run/java_prog1.sh.pid ] && exit
echo "$$" > /var/run/java_prog1.sh.pid

... everything else our script does ...

rm -f /var/run/java_prog1.sh.pid

Then in the beginning of java_prog2.sh:

[ -f /var/run/java_prog1.sh.pid ] && exit

... the rest of your java_prog2.sh ...

Or:

if [ -f /var/run/java_prog1.sh.pid ]; then
   kill -0 `cat /var/run/java_prog1.sh.pid` > /dev/null 2>&1 && exit
fi

... the rest of your java_prog2.sh ...

The first one will just exit immediately, if the a file (possibly and probably containing a PID from the first script) exists. The second one will exit only if the file exists and a process with a PID in in that file is in the process table (kill -0 will return 0 in case it does).

As far as the actual scripting goes, you probably have to experiment a bit. This is just to give you a rought idea how it might go.

How to order the Cronjobs to avoid overlap or conflict

Though you're running the same script at the same time with two different entries in crontab, those are treated as two different cron tasks. Not sure what OS you're using. But the typical order for Ubuntu is top-down (in parallel), i.e. let's say you've three tasks defined in crontab like this:

* * * * * T1
* * * * * T2
* * * * * T3

T1 starts first, then T2 without waiting for T1 to complete, then T3 without waiting for T2 or T1 to complete. Ubuntu inherits this order from Debian. But in general this behavior may vary by Linux distributions or versions and cron implementation. And it really doesn't make sense to depend on it to be the same. For instance, in FreeBSD, the order is bottom-up. If the scripts depend on each other, best to call them in sequence, one from the other, or from a common wrapper script, which is the only one cron actually executes. The overlapping logs that you have seen might be due to the first change directory command before the actual invocation of your script in both tasks. You might want to try specifying the absolute path of your script.

*/5 * * * * /usr/bin/python3 /path/to/script/comand.py -pgw y
*/5 * * * * /usr/bin/python3 /path/to/script/comand.py -px y

Another option is to create a sample wrapper script something like this and schedule it in crontab. You're free to modify.

#!/bin/bash

/usr/bin/python3 /path/to/script/comand.py -pgw y > /path/to/task1.log 2>&1 &
TASK1_PID=$!
echo "Task1 PID: $TASK1_PID Started"
while ps -p $TASK1_PID; do sleep 1; done; echo "Task1 PID: $TASK1_PID Completed"

/usr/bin/python3 /path/to/script/comand.py -pgw y > /path/to/task2.log 2>&1 &
TASK2_PID=$!
echo "Task2 PID: $TASK2_PID Started"
while ps -p $TASK2_PID; do sleep 1; done; echo "Task2 PID: $TASK2_PID: Completed" 

Running two commands sequentially in a cron job?

If the first command is required to be completed first, you should separate them with the && operator as you would in the shell. If the first fails, the second will not run.

How to get a unix script to run every 15 seconds?

I would use cron to run a script every minute, and make that script run your script four times with a 15-second sleep between runs.

(That assumes your script is quick to run - you could adjust the sleep times if not.)

That way, you get all the benefits of cron as well as your 15 second run period.

Edit: See also @bmb's comment below.

Does a Cron job overwrite itself

You need a locking mechanism to identify that the script is already running.

There are several ways of doing this but you need to be careful to use an atomic method.

I use lock directories as creating a directory is guaranteed to be atomic -

LOCKDIR=/tmp/myproc.lock

if ! mkdir $LOCKDIR >/dev/null 2>&1
then
  print -u2 "Processing already running - terminating"
  exit 1
fi

trap "rm -rf $LOCKDIR" EXIT

Will Cron start a new job if the current job is not complete?

They'll run at the same time. The standard practice around this is to create a file lock (commonly referred to as a flock), and if the lock isn't available, don't run.

The advantages to this over Zdenek's approach is that it doesn't require a database, and when the process ends, the flock is automatically released. This includes cases where the the process is killed, server rebooted, etc.

While you don't mention what your cron job is written in, flock is standard in most languages. I'd suggest googling for locks and the language you're using, as this will be the more robust solution, and not relying upon random timeouts. Here are some examples from SO:

• Shell script
• Python
• PHP
• Perl

Running only one Perl script instance by cron

I have always had good luck using File::NFSLock to get an exclusive lock on the script itself.

use Fcntl qw(LOCK_EX LOCK_NB);
use File::NFSLock;

# Try to get an exclusive lock on myself.
my $lock = File::NFSLock->new($0, LOCK_EX|LOCK_NB);
die "$0 is already running!\n" unless $lock;

This is sort of the same as the other lock file suggestions, except I don't have to do anything except attempt to get the lock.

Limit the number of cron jobs running a PHP script

I used a dirty way of tracking the number of the scripts being executed via a table in a database. A launched script inserts a row with an id and the field started_time. On exit it removes his row. Too old rows are considered as "failed\dead scripts".

The worker scripts aren't launched by cron directly. Cron launches "the launcher script" every 1 second or so, which checks the table for the number of active workers and spawns more workers if needed.

Such a scheme is working online for me for 2.5 years already. Just grabbing some constantly updated content.

---

Related Topics