Monitor a Go job.

One request after the work, with a deadline. Five minutes, start to finish.

Before you start

You need a heartbeat in Upglory with its period and grace set to match the schedule that starts the program. Its page shows the ping address; read it from the environment rather than a constant, so the same binary can run in staging without telling us the production job ran.

Settings that match this schedule

• Period1 hourThe schedule that starts the program, whether that is cron, a timer or a scheduler.
• Grace10 minutesThe longest run you have seen, plus a little.
• Channelsemail, TelegramInherited from the workspace default. Change it on this heartbeat if it matters less than the others.

The line

One client with a timeout, one call after the work returned nil. The deadline is not optional: a job that hangs on its own ping is a job that never finishes.

// after the work returned nil
var pinger = &http.Client{Timeout: 10 * time.Second}

func ping(ctx context.Context, path string) {
	req, err := http.NewRequestWithContext(ctx, http.MethodGet, pingURL+path, nil)
	if err != nil {
		return
	}
	res, err := pinger.Do(req)
	if err == nil {
		_ = res.Body.Close()
	}
}

if err := rebuildIndex(ctx); err != nil {
	return err
}
ping(ctx, "")

The ping's own errors are swallowed on purpose: a monitoring call that fails must never fail the job it is reporting on.

Measure the run time

Ping /start before the work. The ping after it then records how long the job ran, and a slow run is not mistaken for a missed one.

ping(ctx, "/start")
if err := rebuildIndex(ctx); err != nil {
	return err
}
ping(ctx, "")

Report a failure

Ping /fail before returning the error: the alert arrives at once and says the job reported a failure, instead of waiting for the grace window to run out.

if err := rebuildIndex(ctx); err != nil {
	ping(ctx, "/fail")
	return err
}
ping(ctx, "")

Test it

Use Send a test ping on the heartbeat's page, or run the program once by hand. Nothing alerts until the first ping arrives, so a job you have not deployed yet never raises an alarm.

What you will see if it stops

Ten minutes past the hour, the heartbeat turns down and the alert names the last ping and how late this one is. A /fail ping arrives at once instead, and says so.

The other guides