Why uptime monitors send false alarms, and how to stop them

The alert says the site is down. You open it and it loads. Here is what usually happened, and how to stop it happening again.

The problem

The alert says your site is down. You open it on your phone and it loads. You try your laptop and it loads there too. By the time you have looked, the monitor says the site is back.

Once is a shrug. The third time in a week, alerts start to read as noise: you mute the channel, or raise the threshold until it stays quiet. That is when a real outage gets through, because the alert you trained yourself to ignore is the one that mattered. A monitor that cries wolf is worse than none, since it still gives you the feeling of being covered.

Most false alarms are not random. The checker did see a failure. It was a failure only the checker saw, or one that lasted a few seconds. Each kind has a tell, and a fix.

The usual causes

A firewall or bot protection blocks the checker

How to recognise it: the check fails with a 403, a 429 or a challenge page, often from one location, while the site works in your browser. It usually starts the day someone turned on a stricter rule, a rate limit or a country block.

The fix: let the checker through. Monitoring services publish the addresses their checks come from and send a fixed user agent. Allow those in your firewall or bot protection, or exclude the one address you monitor from the challenge. Do not loosen the rule for the whole site.

A 503 from a maintenance or caching plugin

How to recognise it: a 503 at the same time every night or during every deploy, sometimes with a page saying the site is briefly unavailable for maintenance. Content management systems put that page up while they update, and some caching layers answer 503 while they rebuild.

The fix: plan the work inside a maintenance window, so the monitor expects it, and keep updates short. If the 503 comes from a cache warming up, monitor an address the cache does not hold, such as a health endpoint the application answers itself.

Slow, not down: the timeout is too short

How to recognise it: the failures say timeout, and the response time graph climbs just before each one. Under load, during a backup or after a cold start, a page that usually answers in 400 milliseconds takes six seconds, and a five second timeout calls that an outage.

The fix: set the timeout to what a visitor would give up on, not to the usual response time. For a web page that is around ten seconds. Treat slowness as its own signal, watched on the response time graph, instead of letting a slow answer count as down.

One network route fails

How to recognise it: one location fails while the others pass, with a connection reset, a timeout while connecting or a host that cannot be reached. Traffic between that location and your server crossed a broken link or an overloaded exchange for a few minutes. Visitors on other routes never noticed.

The fix: check from more than one location, and call it down only when more than one agrees. A single location is good at telling you something somewhere is wrong. It is bad at telling you your site is down.

DNS answers differently by location

How to recognise it: one location cannot resolve the name, or connects to an old address, often in the hours after a DNS record changed. Resolvers keep the old answer until its TTL runs out, and different resolvers let it go at different times. A nameserver that was never updated keeps handing out the old address for good.

The fix: lower the TTL a day before you move a record, and keep the old server answering until the old TTL has passed. If one location keeps getting the wrong answer, ask every nameserver of the domain and compare:

for ns in $(dig +short NS example.com); do
  echo "$ns: $(dig +short example.com @"$ns")"
done

A certificate problem only one client sees

How to recognise it: a TLS error from the checker while browsers show the padlock. The usual cause is a missing intermediate certificate: a browser fills the gap from its own cache, a stricter client does not. An expired certificate on one of several servers behind a load balancer does the same, and fails only the checks that land on that server.

The fix: serve the full chain, renew every server at once, and test the way a strict client connects. A result of 0 means the chain is complete:

openssl s_client -connect example.com:443 -servername example.com < /dev/null | grep "Verify return code"

What works with any tool

• Check from more than one location, and require two to agree. This removes the route and DNS cases almost entirely, and most of the firewall ones.
• Retry before alerting. One failed check followed by a pass is a blip. Two failures in a row from the same place is a pattern. Waiting for the second costs one check interval and saves most false alarms.
• Set timeouts to what a person would wait, and treat slowness as a separate signal from down.
• Check for a keyword, not only the status code. A proxy or a cache can answer 200 with an error page. A word you expect in the body, such as your site's name in the footer, proves the page rendered. Pick a word that is there for a signed out visitor.
• Use maintenance windows for planned work, so a deploy or an update at three in the morning is expected and not an incident.
• Monitor what a visitor uses: the home page, or a health endpoint that touches the database, rather than a static file that answers even when the application is broken.

One failure, two outcomes

Say a site is checked every minute from two locations, A and B, and it counts as down only when both agree. At 10:00:00 location A gets a timeout. B passed twenty seconds earlier. Nothing is sent yet: the monitor shows a warning and waits for the next check.

A blip

A real outage

The outage reached you twenty seconds later than an alert on the first failure would have, and you know it is real for more than one network. Without the second location, the blip is an alert at 10:00 and a recovery at 10:01: the kind that teaches people to ignore alerts.

How Upglory does it

We check from 3 locations. Each monitor is checked every 30 seconds, 1 minute or 5 minutes, depending on the plan, and a paid plan checks it from up to 2 of them, chosen by you. One failing location marks the monitor degraded and wakes nobody. When a location fails, we ask the other one to check at once instead of waiting for its turn. The incident opens when both have failed twice in a row, and the alert carries what each check saw. Free is the exception: a free workspace alerts from its one location, after two failures in a row there.

A check waits 10 seconds for an answer unless you change it, up to 30 seconds. A slow answer is shown as slow and never sent as an alert. A check expects 200 or any status you choose, and can require a keyword in the body. Maintenance windows cover the monitors you pick, and nothing they cover alerts while the work runs.

The uptime checks page shows what a check keeps when it fails, and the docs cover setting one up.