Is it DNS or the server? A five-step check with dig and curl
When the site is down, any of four layers can be the cause, and each one needs a different person. Five commands tell them apart.
The problem
"The site is down" is four different problems wearing one sentence. The name may not resolve, so nobody can find the server. The name may resolve, but nothing answers at that address. Something may answer, but the secure connection fails. Or the connection works and the application returns an error.
Each layer belongs to a different person. DNS is whoever manages the domain and its records. The address is whoever runs the server, the network or the hosting. The certificate is whoever renews it. The error page is the developers. Working out which layer failed, before anyone starts fixing, saves the half hour where three people restart things that were never broken.
Five commands answer it, in order, from the bottom layer up. Run them from your own machine, and if you can, from a second network too, such as a phone hotspot.
Five steps
1. Does the name resolve?
dig +short example.com
One or more addresses means DNS answered. Nothing at all means no address came back, and the next command says why:
dig example.com | grep status
;; ->>HEADER<<- opcode: QUERY, status: NXDOMAIN, id: 40211
NOERROR with no address means the name exists but has no record of that type. NXDOMAIN means the name does not exist at all: a deleted record, a typo in a new record, or a domain whose registration ended. SERVFAIL means the resolver could not get a usable answer, often a broken DNSSEC signature or nameservers that do not answer. If it works on another network but not on yours, your resolver may be holding an old answer.
2. What do the domain's own nameservers say?
Your resolver repeats what it was told and keeps it for the record's TTL. The nameservers of the domain are the source. Find them, then ask each one directly:
dig +short NS example.com
ns1.example.net. ns2.example.net.
dig +short example.com @ns1.example.net dig +short example.com @ns2.example.net
If the nameservers give the right address and your resolver does not, the record changed and caches are still catching up: wait out the TTL. If the nameservers disagree with each other, one was not updated. If they do not answer, or the NS list itself is wrong, the problem is at the DNS host or the registrar. To see the whole chain, from the root down to the answer, follow it with:
dig +trace example.com
3. Does the address answer?
Now that you know the address, try to connect to it:
curl -sv --connect-timeout 5 -o /dev/null https://example.com 2>&1 | grep -E 'Trying|Connected|connect to'
* Trying 203.0.113.10:443... * Connected to example.com (203.0.113.10) port 443
Connected means the server accepted the connection, so the network and the address are fine. Connection refused means the machine is there but nothing listens on that port: the web server or the proxy is stopped. A timeout means nothing answered at all: the machine is down, a firewall drops the traffic, or the network in between is broken.
To rule DNS out completely, connect to a known good address with the name, whatever DNS says. If this works while the plain request fails, the problem is DNS, not the server:
curl -sv --connect-timeout 5 -o /dev/null --resolve example.com:443:203.0.113.10 https://example.com
4. Does the secure connection work?
echo | openssl s_client -connect example.com:443 -servername example.com 2>/dev/null | openssl x509 -noout -subject -issuer -enddate
You should see the subject with your name, an issuer and an end date in the future. An empty answer or an error means the handshake failed: a certificate that ended, one for another name, a missing intermediate certificate, or a server that offers only old protocol versions. That is the certificate owner's problem, not DNS and not the application.
5. What does the application answer?
The last layer: the status code, and where the time went.
curl -sS -o /dev/null --max-time 10 -w 'status %{http_code} dns %{time_namelookup} connect %{time_connect} tls %{time_appconnect} first byte %{time_starttransfer} total %{time_total}\n' https://example.comstatus 502 dns 0.012 connect 0.031 tls 0.074 first byte 10.081 total 10.082
Read it left to right. Here DNS took 12 milliseconds, the connection and TLS were quick, and then the server took ten seconds to send its first byte before answering 502. The network is fine. A proxy in front gave up waiting for the application behind it, so this one is for the developers. A 5xx is the application or what sits in front of it. A 4xx such as 403 or 429 is usually a rule that blocks the request, which matters when only a monitor gets it.
Symptom, layer, who to call
| What you see | Likely layer | Who to call |
|---|---|---|
NXDOMAIN | DNS record or domain registration | Whoever manages the domain |
SERVFAIL, or nameservers not answering | DNS hosting or DNSSEC | The DNS host |
| Wrong or old address on some networks | A recent record change and caches | Nobody, if the nameservers agree: wait out the TTL |
| Connection refused | Web server or proxy stopped | Whoever runs the server |
| Connection timed out | Machine down, firewall or network | Hosting or network |
| Certificate error | TLS | Whoever renews certificates |
| 5xx, or a slow first byte | The application or its proxy | The developers |
| 403 or 429 for a checker only | A firewall or rate limit rule | Whoever owns the rules |
Why one location is not enough
Every command above runs from one place, through one resolver and one network route. That is enough to find the layer. It is not enough to know whether the problem is yours or everyone's.
A name that fails from one network and resolves from another points at caches after a change, or at one resolver. A server that times out from one place and answers from another points at a network route, not at the server. The same failure from several places, at the same moment, is the real outage. So when step 1 or step 3 fails, run it again from a second network before you call it, and when you set up monitoring, check from more than one location for the same reason.
How Upglory does it
We check from 3 locations, and a paid plan checks each monitor from up to 2 of them. A monitor checked from two locations is down only when both have failed twice in a row. Free is the exception: a free workspace alerts from its one location, after two failures in a row there.
A failed check keeps what you would have run by hand: the addresses DNS answered with, how long the lookup, the connection, the TLS handshake and the wait for the first byte each took, and whether it connected, finished the handshake and received headers. It keeps the status code, the few headers that explain an outage, the first 2 kB of the answer, and the certificate's subject and issuer, for 30 days.
The alert names the failure in words, such as these:
- the name does not exist
- the connection was refused
- the check did not finish in time
- the certificate or TLS failed
- the wrong status code
When a lookup fails, we ask a second resolver before we call it. A lookup that gets no answer at all says nothing about your site, so it is not counted against it.
The uptime checks page shows a failed check's evidence, and the docs cover setting up a monitor and its locations.