HTTP 503 from a load balancer during origin drain

Why a load balancer says 503

A load balancer (cloud LB, Cloudflare Load Balancing pool, or an origin pool in front of the site) answers 503 when it has no healthy origin to forward traffic to. Its job is to send you to one; if all configured backends fail health checks, it replies 503 with a connection-refused or "no healthy upstream". The same mechanism serves the site down page a moment before failover, and it hides the origin now, so the health check is the only probe that matters.

The three health-check phases

Drain vs remove

A load balancer that keeps an origin in "draining" stops new connections to it but waits for in-flight requests to finish. The Cloudflare origin pool article has the exact pool semantics. The 503 during a drain is different from the 503 of a dead origin: drain is deliberate and temporary, and the LB answers 503 when every pool member is draining or down.

What to do in each case:

• Warm the new origin first. Batch of config, DNS, and dependency; run a manual health probe before the LB pulls it into the pool.
• Remove the draining origin from the pool, not just from DNS. A pool with one member marked draining keeps answering 503 to every request even though the DNS is fine.
• Failover: configure a fallback origin pool (or a fallback A record) so a drained pool fails over to the standby, not to 503.

Verify the health check is honest

The 503 is only as truthful as the health check. A path that returns 200 for the whole site's load path ("/healthz" that hits only the app and not the DB) makes the pool healthy until the first bad request, then the mix of edge 503 and app errors. Make the health check depend on the true dependencies of serving a page.

• Probe URL: /healthz; verify it exercises PHP and the database connection, not just a static file.
• Interval and threshold: the site survives short restarts without flaps (e.g. 3 consecutive passes before marked healthy, 3 failures before down).
• Port and TLS: a check on port 80 against an HTTPS-only origin keeps the origin "healthy" when it is effectively dead.

The uptime monitoring article covers external monitors; a health probe sits closer to the host and runs before any external monitor fires.

Retry semantics for clients

A client that retries a 503 can help or hurt:

• A "retry on 503 without limit" floods the failing origin exactly when it is weakest. Use exponential backoff and the Retry-After header (the 429 retry article explains the header contract).
• A "retry immediately" loop during a cross-origin drain keeps the page 503 until the LB sees the origin healthy again.
• Browsers and search crawlers get the 503 as a temporary signal; the 503 reference article covers why a planned maintenance keeps a 503 rather than a 504, and what to tell search engines during a planned drain.

Verify the drain

1. The LB console shows the origin toggling to draining/down.
2. The health-probe log at the origin stops receiving probes.
3. The error log shows connection attempts to the removed origin (or its engine) during the drain window, then silence.
4. External checks show the site up once the fallback serves.

Prevention

• Run at least two origins per pool where the budget allows, in different availability zones.
• Keep health checks aligned with the app's real dependencies.
• Script the drain: drain, verify, remove, unset; never remove a live origin by hand for a planned window while the pool still has no fallback.
• Alert on both "all origins down" (unserved 503) and "pool at one member" (risk of 503) so a sliding single-member pool never quietly starves.

Where the site uses a single origin with no LB, the 503 is the web server's own response and the 503 reference article is the fix list. Where the load balancer gives up and times out instead, the 504 gateway article is the adjacent symptom.