How to monitor an API: what a 200 does not tell you
An API that answers 200 can still be broken for everyone calling it. Here is what a useful check looks at, and how to build an endpoint worth checking.
The problem
The simplest API monitor asks for a URL and checks that the answer is 200. It catches the server that is off and the load balancer with nothing behind it. It misses most of the ways an API actually breaks.
An API can answer 200 with {"error": "database unavailable"} in the body, because the framework wraps every error in a successful response. It can answer 200 with yesterday's data from a cache, while the service behind it has been down for an hour. It can answer 200 with an empty list, because a query that times out returns nothing instead of failing. It can answer 200 with the sign-in page, because the request was redirected there. Each of these is an outage for the people calling it, and each is a green tick on a monitor that only reads the status.
A useful API check asks what a client needs, not only whether something answered.
What to check
The exact status, and that a redirect is not success
Expect the one status a working endpoint returns, usually 200, not any 2xx. Treat a redirect as a question: an API endpoint that suddenly answers 301 or 302 has usually been moved, put behind a sign-in, or caught by a rule meant for the website. If your monitor follows redirects, make sure the page it lands on has to pass on its own, so a redirect to a sign-in page that answers 200 is not counted as up.
Something in the body
A keyword or a field you only get when the answer is real: "status":"ok" from a health endpoint, a field name from a real resource, or the opposite, the absence of error. Pick something that appears only when the work was done, not something every error page carries too.
The response time
Record how long each answer takes, and treat slow as its own signal. An API that answers in three seconds instead of three hundred milliseconds is up by the monitor's rules and broken for the mobile app that gives up after two.
A timeout that fits
Set it to what your slowest client would wait. For most APIs that is 5 to 10 seconds. A timeout longer than any client waits only delays the alert.
The method and the headers
Some endpoints answer GET differently from the POST your clients send, need an Accept header to return JSON, or need a header that names the tenant. If the endpoint you watch needs those, the check has to send them, or it is testing something your clients never call. Often the cleaner answer is a health endpoint that needs none of them.
Authentication, with a key that can do nothing
If the check has to sign in, give it its own key: read-only, limited to the one resource it reads, named after the monitor so it is easy to find, and never a person's credentials. A key that can only read one product list is harmless if a monitoring setting ever leaks. Rotate it like any other key, and expect the check to fail the day you forget.
Designing a health endpoint
The best thing to monitor is an endpoint built for it, which answers the question "can this service do its job right now?" in one request. A good one touches what the service depends on: a quick query against the database, a ping to the queue, a lookup in the search index. When everything works it answers 200:
HTTP/1.1 200 OK
Content-Type: application/json
Cache-Control: no-store
{
"status": "ok",
"checks": {
"database": "ok",
"queue": "ok",
"search": "ok"
}
}When something it depends on fails, it says which part and answers 503, so a check that only reads the status still sees the failure:
HTTP/1.1 503 Service Unavailable
Content-Type: application/json
Cache-Control: no-store
{
"status": "fail",
"checks": {
"database": "ok",
"queue": "fail",
"search": "ok"
}
}What it must not do:
- Expose anything. No version numbers, host names, connection strings, stack traces or error messages from the database. The answer is public to anyone who finds the URL, so it should say which part failed and nothing about why.
- Be expensive. It runs every minute from several places. Give each dependency a short timeout of its own, a second or two, so one slow part answers fail instead of holding the whole request. If a check is costly, cache its result for a few seconds.
- Need a person's credentials. It should answer without signing in. If it must be protected, use a dedicated token as above, not a user account.
- Be cached by anything in front. Send
Cache-Control: no-store, or a proxy will keep answering ok after the service behind it is gone.
Keep two endpoints apart if your platform restarts unhealthy processes. A liveness endpoint says only that the process answers, and a restart fixes it when it fails. The readiness endpoint above says whether it can serve, and a restart does not fix a database that is down. Restart on the first, alert on the second.
What a check sees, with curl
Before you trust a monitor, look at the endpoint the way it does. The status and the total time, in seconds:
curl -sS -o /dev/null -w '%{http_code} %{time_total}\n' https://api.example.com/health200 0.184321
A keyword check, the same thing a monitor's keyword setting does:
curl -sS --max-time 10 https://api.example.com/health | grep -q '"status":"ok"' && echo up || echo down
A real resource with a read-only key, failing when the list comes back empty. jq -e exits with an error when the expression is false:
curl -sS --max-time 10 -H "Authorization: Bearer $MONITOR_KEY" \ 'https://api.example.com/v1/products?limit=1' | jq -e '.items | length > 0' > /dev/null && echo up || echo down
And whether an endpoint redirects, and where to:
curl -sS -o /dev/null -w '%{http_code} %{redirect_url}\n' https://api.example.com/v1/orders302 https://example.com/login
That last answer is the one to worry about: the orders endpoint now sends clients to a sign-in page.
Timeouts and retries, with the arithmetic
Retries stop false alarms, but they add up. Say a check waits 30 seconds for an answer and tries 3 times before it calls a failure. When the API hangs, that is 90 seconds before the first failure is even recorded. If the alert then waits for a second failed check, that one takes as long again, and the slowest kind of outage, the hang, is the one you hear about last: three minutes or more after it started.
Now take a 10 second timeout with one quick retry 3 seconds later: at most 23 seconds to a recorded failure, and a blip that clears in under 3 seconds never counts. Short timeouts and few, quick retries find hangs fast. Long timeouts and many retries mostly hide them.
A good starting point: a timeout of about three times the slowest answer you see on a normal day, never more than your clients wait, and one retry a few seconds later.
How Upglory does it
An Upglory monitor sends a GET request with our own user agent unless you choose another request, and passes when the last answer has the status you expect, 200 unless you choose another. It follows up to 5 redirects, never from https to plain http, and compares the status of the answer it ends on. A keyword you set must appear in the first 1 MB of that answer. A check waits 10 seconds for an answer unless you change it, up to 30 seconds, and a failed check is tried again from the same location 3 seconds later before it counts.
Under Request, below the address, a monitor can send GET, HEAD, POST, PUT, PATCH, DELETE or OPTIONS, up to 10 request headers, and with POST, PUT, PATCH or DELETE a body of up to 4 KB as JSON, a form, plain text or XML. Every plan can. A header value is never shown again once it is saved: the monitor shows the header's name and that its value is set, and a new value replaces it. The headers and the body go only to the address's own host, so a redirect to another host gets neither.
A monitor that sends a header or a body carries secrets: we keep only the status, the timings, DNS and TLS of its answers, never the body or the headers, and its alerts leave out the address. If any other answer can carry anything private, mark that monitor as carrying secrets the same way. A health endpoint that answers without a key, as above, is still the simplest thing to watch.
For slow answers, pick a threshold of 500 ms, 800 ms or 2 seconds. An answer slower than that twice in a row shows the monitor as degraded, and degraded never sends an alert.
The API reference shows how to pause a monitor or plan a maintenance window from a deploy script, webhooks send alerts to an endpoint of your own, and the uptime checks page shows what a failed check keeps.