Cron job not running? How to find out when it silently stops

Cron does not tell anyone when a job fails or never starts. Here is how to find out why it stopped, and how to hear the next time it does.

The problem

Cron is quiet by design. It starts a command at the time you gave it, and if the command fails, or never starts, cron tells nobody. At best it mails the output to a local mailbox on the server that nobody reads.

So a backup job stops on a Tuesday because the disk filled up or a password changed, and the first sign comes weeks later, on the day you need the backup and the newest one is a month old. Invoice runs, report emails and certificate renewals fail the same way.

There are two jobs here. First, find out why the job is not running now. Then make sure you hear the next time it stops, without having to remember to look.

First, find out why

Is cron running?

On Debian and Ubuntu, systemctl status cron. On Fedora and the other Red Hat family systems the service is called crond. In a container, cron is often not running at all, because the container starts only its main process.

Did cron start the job?

Cron logs every command it starts. Try grep CRON /var/log/syslog on Debian and Ubuntu, or journalctl -u cron --since today (-u crond on Red Hat systems). A line with your command means cron started it and the job failed afterwards. No line means cron never ran it.

Is the line right?

Five fields, then the command. This one runs at 02:30 every day:

# minute  hour  day-of-month  month  day-of-week  command
  30      2     *             *      *            /usr/local/bin/backup-nightly

The usual mistakes: * 2 * * * runs every minute from 02:00 to 02:59, not once. When both day of month and day of week are set, cron runs on either, so 0 2 1 * 1 runs on the first of the month and on every Monday. A % in the command means a new line to cron and has to be written \%, which breaks the familiar date +%F in backup file names. And the last line of a crontab needs a line break after it, or some versions of cron ignore it.

Does it work in cron's environment?

This is the most common cause. Your shell has a long PATH and your profile. Cron gives the command a minimal environment, often a PATH of only /usr/bin:/bin and /bin/sh as the shell. A command that works when you type it fails under cron because node, php or pg_dump lives somewhere else. Use full paths, or set PATH=/usr/local/bin:/usr/bin:/bin at the top of the crontab. To see what cron sees, add this line for a minute and compare the file with env in your own shell:

* * * * *  env > /tmp/cron-env.txt

Relative paths

Cron starts the command in the user's home directory, not where the script lives. A script that reads ./config.yml or writes to backups/ works from its own folder and fails from cron. Change to the right directory at the top of the script, or use absolute paths throughout.

The user and permissions

A crontab you edit with crontab -e runs as you. The system crontab and the files in /etc/cron.d/ have an extra field for the user, between the schedule and the command, and a line without it fails. The script has to be executable (chmod +x), and the user has to be allowed to read what it reads and write where it writes. Scripts in /etc/cron.daily/ are started by run-parts, which on Debian and Ubuntu skips any file name with a dot in it, so a backup.sh there never runs.

The timezone

Cron uses the server's timezone, and most servers run on UTC. A job meant for two in the morning your time can run hours earlier or later. Check with timedatectl. On a server that follows local time, the change to and from summer time can skip or repeat a job scheduled between 01:00 and 03:00.

The machine was off

Cron does not catch up. If the server was rebooting, asleep or scaled down at 02:30, that night's run simply does not happen. anacron, or a systemd timer with Persistent=true, runs a missed job once the machine is back.

The output goes nowhere

Cron mails a job's output to its user through the server's local mail, which is often not set up, so the error that explains everything is thrown away. Set [email protected] at the top of the crontab if the server can send mail, or send the output to a log you can read:

30 2 * * *  /usr/local/bin/backup-nightly >> /var/log/backup-nightly.log 2>&1

The 2>&1 matters. Errors go to standard error, and without it they are still lost.

Then watch it

Fixing today's failure says nothing about the next one, and a log only helps if someone reads it. Turn the silence into a signal instead: the job reports when it finishes, and something outside the server expects that report on a schedule and tells you when it does not arrive. This is the heartbeat pattern. The report is one HTTP request to an address that belongs to the job:

30 2 * * *  /usr/local/bin/backup-nightly && curl -fsS -m 10 --retry 3 https://ping.example.com/nightly-backup > /dev/null

&& or ;

With &&, curl runs only when the job exits with 0, so a failed backup sends nothing and the silence is noticed. With ;, curl runs whatever happened, and the monitor hears that all is well from a job that failed. Make sure the job exits with something other than 0 when it fails: set -e at the top of a shell script does most of that.

The curl flags

-f makes curl fail on an HTTP error instead of treating an error page as success. -sS keeps it quiet but still prints an error. -m 10 stops it waiting more than ten seconds, so a slow network cannot hold the job open. --retry 3 tries again when the network drops a request, so a lost ping does not become a false alarm of its own.

Report a failure as a failure

Silence only tells you once the grace window has passed. The job usually knows sooner. If your monitor has a separate address for failures, call it, and keep the job's own exit code so cron and your logs still see the failure. A small wrapper does both:

#!/bin/sh
# run-and-report: run the job, then say how it went.
PING=https://ping.example.com/nightly-backup

/usr/local/bin/backup-nightly
code=$?
if [ "$code" -eq 0 ]; then
  curl -fsS -m 10 --retry 3 "$PING" > /dev/null
else
  curl -fsS -m 10 --retry 3 "$PING/fail" > /dev/null
fi
exit "$code"

Grace time

A heartbeat needs two numbers: how often the ping is expected, and how late it may be before anyone is told. The period is the schedule, not the run time, so a daily job is expected every 24 hours. Grace covers the variation, such as the run that takes forty minutes instead of fifteen because the database grew. So a daily job needs a window longer than a day: 24 hours plus grace, and grace is the longest run you have seen plus a margin. Too short, and a slow night wakes you. Too long, and you hear late.

Worked examples

A nightly database backup

The script stops at the first error, and pipefail makes a failed dump fail the whole pipe. Without it, a dump that failed piped into a gzip that worked exits with 0, and you keep an empty file every night. The % is safe here because it is in the script, not in the crontab line.

#!/bin/bash
set -euo pipefail
cd /var/backups/db
pg_dump shop | gzip > "shop-$(date +%F).sql.gz"
find . -name 'shop-*.sql.gz' -mtime +14 -delete

PATH=/usr/local/bin:/usr/bin:/bin
30 2 * * *  /usr/local/bin/backup-nightly >> /var/log/backup-nightly.log 2>&1 && curl -fsS -m 10 --retry 3 https://ping.example.com/nightly-backup > /dev/null

The ping is expected every 24 hours. If the backup takes up to twenty minutes, an hour of grace is plenty.

A queue worker that should ping while healthy

A worker that runs all the time has no end to ping from. Ping on a schedule while it is healthy instead, and expect a ping every five minutes with ten minutes of grace:

*/5 * * * *  curl -fsS -m 5 http://127.0.0.1:9000/health > /dev/null && curl -fsS -m 10 --retry 3 https://ping.example.com/queue-worker > /dev/null

The health endpoint should check something real, such as that the worker finished a job recently. If the worker crashes, hangs or the machine goes away, the pings stop. Better still, ping from inside the worker's loop after each batch.

One monitor per job

Give every job its own address. If three jobs share one, a ping from any of them hides the silence of the other two. Name each after the job and the server, such as nightly-backup on db-1.

How Upglory does it

An Upglory heartbeat is the address a job pings and the promise it keeps: a period, and 30 minutes of grace unless you change it. Its address looks like https://ping.upglory.io/abc123. Join the ping with && as above. Ping /start when the job begins and the finish ping records how long it ran, so a long run is not mistaken for a missed one. Ping /fail to report a failure at once instead of waiting for the grace window.

Nothing alerts until the first ping arrives, so a job you have not deployed yet raises no alarm. When a ping is late past the grace window, the alert says which job and when its last ping came, and when the next run pings, the heartbeat recovers on its own.

The crontab guide in the docs has the lines to copy, and the heartbeats page shows the rest.