Monitor a Kubernetes CronJob.

One curl at the end of the container's command. Six minutes, start to finish.

Before you start

You need a heartbeat in Upglory with its period and grace set to match the CronJob, and its ping address in a Secret called upglory under the key ping-url. The heartbeat's page shows the address.

Settings that match this schedule

• Period24 hoursThe schedule on the CronJob.
• Grace45 minutesThe run time times the retries the backoffLimit allows, plus the starting deadline.
• Channelsemail, SlackInherited from the workspace default. Change it on this heartbeat if it matters less than the others.

The line

No sidecar and no extra container. Put the ping at the end of the command the job already runs, so it goes only when the process exits zero.

spec:
  schedule: "0 2 * * *"
  startingDeadlineSeconds: 300
  jobTemplate:
    spec:
      backoffLimit: 2
      template:
        spec:
          containers:
            - name: backup
              image: registry.example/backup:1.4
              command: ["/bin/sh", "-c"]
              args:
                - /app/backup.sh && curl -fsS -m 10 --retry 3 "$UPGLORY_PING_URL"
              env:
                - name: UPGLORY_PING_URL
                  valueFrom:
                    secretKeyRef: { name: upglory, key: ping-url }

backoffLimit is what decides the grace window: with two retries the job can take three times its own run time before it gives up, and only then is the silence real. startingDeadlineSeconds keeps a job that could not start on time from starting hours late and pinging as if nothing happened.

Measure the run time

Ping /start as the container begins. The ping at the end then records how long the run took, retries included.

- curl -fsS -m 10 "$UPGLORY_PING_URL/start"; /app/backup.sh && curl -fsS -m 10 --retry 3 "$UPGLORY_PING_URL"

Report a failure

The exit 1 keeps the pod failed, so Kubernetes still retries up to the backoffLimit, and each failed try is reported at once instead of when the grace window runs out.

- /app/backup.sh && curl -fsS -m 10 "$UPGLORY_PING_URL" || { curl -fsS -m 10 "$UPGLORY_PING_URL/fail"; exit 1; }

Test it

Use Send a test ping on the heartbeat's page, or start one run now with kubectl create job --from=cronjob/backup backup-test. Nothing alerts until the first ping arrives, so a job you have not deployed yet never raises an alarm.

What you will see if it stops

Forty-five minutes after 02:00 the heartbeat turns down. The usual causes, in the order they happen: the image tag no longer exists, the node had no room to schedule the pod, or the job hit its backoffLimit and Kubernetes gave up quietly.

The other guides