A five-year-old bug, gone.

Our search workers had been restarting every few minutes for five years, and each restart took about five minutes. In the day before the fix they spent 68% of their time starting up, and pods were killed when both workers were down at once.

I traced it to two causes. My fix for the first shipped to production on 7 October, in the same release that made starts fast.

  • Python
  • Gunicorn
  • Kubernetes
  • Production debugging
One hour of one worker starting up serving requests
Beforeprod, Oct 6–7Afterprod, Oct 7–80 min10 min20 min30 min40 min50 min60 min

Restarts were frequent

Cause.
Workers restart after a request limit, and health checks counted toward it. About 95% of requests were health checks.
Fix.
A pre_request hook gives each health check back, so it nets to zero. Seven lines and a test against a real SyncWorker.
def pre_request(worker, req):
    if req.path == HEALTH_CHECK_PATH:
        worker.nr -= 1  # probes don't age the worker

Restarts were slow

Cause.
Each worker start read a Git repo with one git show per file: about 3,300 processes.
Fix.
One git ls-tree and one git cat-file --batch per branch cut the read from 278.7 s to 9.5 s on staging. The release that shipped bakes the repo into the image instead, so a worker now starts in about 5 seconds.
# before: 3,300 × git show <branch>:<file>
git ls-tree -r <branch>
git cat-file --batch  # every blob, one process

Results

Worker restarts
3,9013,901
prod, 26 h each
Average worker life
7.9 min5.7 h
prod, 26 h each
Liveness kills
5353
prod, 26 h each
Failed health probes
1,5971,597
prod, 26 h each
Worker startup, p50
322 s322.0 s
prod, same release
Time spent starting up
~68%<0.1%
prod, 26 h each

Why there are still restarts

The 89 that remain are the limit doing its job. Workers now serve about 320 real requests each before they restart, which matches the limit of 200 plus up to 200 jitter. Restarts follow customer traffic now, peaking in US business hours.

What I'm watching

Workers live much longer, so they hold more memory: peak use rose from 8.3 to 9.7 GiB of an 11 GiB limit. It follows traffic rather than climbing, but one day can't rule out a slow leak. If peaks keep rising, the fix is to lower the request limit, which now counts real requests only.

How it was measured

Two equal windows of 26 h 20 m on either side of the release, 10 pods × 2 workers in both. Every Gunicorn lifecycle log line was counted once, with workers keyed by pod and pid together. Kills and probe failures come from Kubernetes events. The after-window starts once the last pod of the rollout was serving, so rollout boots don't count.

More case studies

← All case studiesHomeGet in touch