Linux Crond Resource Limits

Linux CROND resource limits

Well, I found a solution during writing this question.

The main pointer to the problem was that I once saw in journalctl message

kernel: cgroup: fork rejected by pids controller in /system.slice/crond.service

So I checked the cron.service and found a parameter TasksMax.

# systemctl show crond.service
Type=simple
Restart=no
...
TasksMax=512
EnvironmentFile=/etc/sysconfig/crond (ignore_errors=no)
UMask=0022
LimitCPU=18446744073709551615
LimitCPUSoft=18446744073709551615

Solution

Add parameter TasksMax to the service configuration in /usr/lib/systemd/system/crond.service, e.g.:

Note: As Mark Plotnick wrote, better way is copy this service to /etc/systemd/system/ folder and modify this file to avoid rewriting service in /usr/ during upgrade.

# cat /usr/lib/systemd/system/crond.service
[Unit]
Description=Command Scheduler
After=auditd.service nss-user-lookup.target systemd-user-sessions.service time-sync.target ypbind.service

[Service]
EnvironmentFile=/etc/sysconfig/crond
ExecStart=/usr/sbin/crond -n $CRONDARGS
ExecReload=/bin/kill -HUP $MAINPID
KillMode=process
TasksMax=100000

[Install]
WantedBy=multi-user.target

Then reload systemd daemons

# systemctl daemon-reload

General solution

If you want avoid this problem with any systemd service you can change default value in /etc/systemd/system.conf, e.g.:

sed -i 's/#DefaultTasksMax=512/DefaultTasksMax=10000/' /etc/systemd/system.conf

And reload systemd daemons to apply the changes

# systemctl daemon-reload

But I don't know the exact consequences of this solution, so I can not recommend it.

Cron job limitations

If you want to break up the script, you can leave the weekly cron but instead of sending mails you can queue it into a database table. Then, using a second cron that run every 5 or 10 minutes, you can read the database mail queue (searching for max 50 or 100 rows) and if you find something, you send a chunk of emails...

In general, this strategy (huge queue loading, smaller queue processing in chunks) allow you to split execution of large processes.

Too Many Cron Jobs giving me bash: fork: Resource temporarily unavailable

The only sane thing you could do is using lockfiles to guarantee that there is only one instance running for every particular cronjob. The simplest way to do this is by using lockfiles from within the cron-scripts. ("cooperative locking"):

• On startup, the (cron) job tests if the lockfile exists
• if the lockfile happens to exist, the job performs a kill -0 <pid> on the other process (#1)
• if the errorcode from the kill is zero, the process actually exists and is from the same userid. The new job should exit. (#2)
• if the errorcode from the kill is not zero, either the process does not exist anymore (good) or belongs to another (unrelated) process for a different uid
• if the process does not exist, the new job can continue by creating the lockfile, and writing its pid into it (#3)
• Now the actual payload can be executed
• finally the lockfile can be be removed.

#1: kill -0 is a no-op; it only checks the validity of the pid

#2: there is a small chance that the pid belongs to an unrelated process for our pid. We can refine the search by inspecting the output of ps, and checking if the pid actually belongs to an older instance of our cron job.

#3: this is not race-free, but for a cronjob that runs once a minute it is probably good enough.

Maximum number of children processes on Linux

The number of child processes can be limited with setrlimit(2) using RLIMIT_NPROC. Notice that fork(2) can fail for several reasons. You could use bash builtin ulimit to set that limit.

You can use getrlimit (or parse /proc/self/limits, see proc(5)) to get that information.

System-wide, you might use /proc/sys/kernel/threads-max since:

This file specifies the system-wide limit on the number of threads
(tasks) that can be created on the system.

There is also /proc/sys/kernel/pid_max

This file specifies the value at which PIDs wrap around (i.e., the
value in this file is one greater than the maximum PID). PIDs
greater than this value are not allocated; thus, the value in this
file also acts as a system-wide limit on the total number of
processes and threads. The default value for this file, 32768,
results in the same range of PIDs as on earlier kernels. On 32-bit
platforms, 32768 is the maximum value for pid_max. On 64-bit
systems, pid_max can be set to any value up to 2^22 (PID_MAX_LIMIT,
approximately 4 million).

However, there could be other limitations (notably swap space).

A task for the kernel is either a single-threaded process or some thread inside some process - e.g. created by low-level syscall clone(2) (or some kernel thread like kworker, ksoftirqd etc...).

BTW, the practical number of processes is much more limited by available resources. A typical Linux desktop has only a few hundreds of them (right now, my Debian/x86-64 desktop with 32Gb RAM & i5-4690S has 227 processes). So a process is a quite expensive resource (it needs RAM, it needs CPU...). If you have too many of them you'll experience thrashing. And in practice, you don't want to have too many runnable processes or schedulable tasks (probably only a few dozens of them at most, perhaps no more than a few per core).

Is there a time limit to Cron jobs in Google Apps?

Cron jobs are subject to a 10 minute deadline, not 30 seconds.

See App Engine version 1.4 release page:

No more 30-second limit for background work - With this release, we’ve
significantly raised this limit for offline requests from Task Queue
and Cron: you can now run for up to 10 minutes without interruption.

@Alex Martelli's answer was correct at the time he wrote it, but is now out of date.

Some programs not started by bash script when script ran as a crontab job but all programs can be started by manually run the bash script

Finally, this problem solved.
The key is TasksMax's setting for cron.service limits the scripts to create more processes.

Resolution for Ubuntu Server 18.04

Edit the system.conf file in /etc/systemd

sudo nano /etc/systemd/system.conf

add the following line to the end of system.conf

DefaultTasksMax=100000

reboot the server

How to check

command "systemctl status cron" can show below infomation

before modify system.conf

xxsc@Strategy2:/etc/systemd$ sudo systemctl service cron
Unknown operation service.
xxsc@Strategy2:/etc/systemd$ sudo systemctl status cron
¡ñ cron.service - Regular background program processing daemon
   Loaded: loaded (/lib/systemd/system/cron.service; enabled; vendor preset: enabled)
   Active: active (running) since Sat 2020-08-29 15:26:17 CST; 3 days ago
     Docs: man:cron(8)
 Main PID: 966 (cron)
    Tasks: 4743 (limit: 4915)
   CGroup: /system.slice/cron.service
           ©À©€  966 /usr/sbin/cron -f
           ©À©€ 4322 ./prodTickRecorderv057.15
           ©À©€ 4355 ./Comodity_AP_NTime_Del00_v0533_0212
           ©À©€ 4356 ./Comodity_AP_NTime_Del00_v0533_0306
           ...

notice the line Tasks: 4743 (limit: 4915), Tasks is close to limit.
after modify the system.conf file

xxsc@Strategy2:~$ systemctl status cron
¡ñ cron.service - Regular background program processing daemon
   Loaded: loaded (/lib/systemd/system/cron.service; enabled; vendor preset: enabled)
   Active: active (running) since Wed 2020-09-02 15:47:46 CST; 2min 26s ago
     Docs: man:cron(8)
 Main PID: 1069 (cron)
    Tasks: 1 (limit: 100000)
   CGroup: /system.slice/cron.service
           ©ž©€1069 /usr/sbin/cron -f
           ...

Cron jobs and random times, within given hours

If I understand what you're looking for, you'll need to do something a bit messy, like having a cron job that runs a bash script that randomizes the run times... Something like this:

crontab:

0 9 * * * /path/to/bashscript

and in /path/to/bashscript:

#!/bin/bash

maxdelay=$((14*60))  # 14 hours from 9am to 11pm, converted to minutes
for ((i=1; i<=20; i++)); do
    delay=$(($RANDOM%maxdelay)) # pick an independent random delay for each of the 20 runs
    (sleep $((delay*60)); /path/to/phpscript.php) & # background a subshell to wait, then run the php script
done

A few notes: this approach it a little wasteful of resources, as it fires off 20 background processes at 9am, each of which waits around for a random number of minutes (up to 14 hours, i.e. 11pm), then launches the php script and exits. Also, since it uses a random number of minutes (not seconds), the start times aren't quite as random as they could be. But $RANDOM only goes up to 32,767, and there are 50,400 seconds between 9am and 11pm, it'd be a little more complicated to randomize the seconds as well. Finally, since the start times are random and independent of each other, it's possible (but not very likely) that two or more instances of the script will be started simultaneously.

---

Related Topics