From c28a9b7b49e303b168242963a1c05c14047b15cb Mon Sep 17 00:00:00 2001 From: Niklas Ye Date: Tue, 13 Oct 2026 11:49:05 +0200 Subject: [PATCH] Let the bootstrap hook wait for a database that is slow to start The server exits when the database does not answer within about ten seconds, so while the database is down the pod crash-loops with a restart backoff that grows to five minutes, and /healthz only answers after it has connected. The hook gave up after a fixed two minutes per attempt, and Flux's default five-minute timeout came first anyway. A failed hook fails the release and nothing runs it again until the next upgrade, so a database that was merely slow left the install without an administrator. Each attempt now waits bootstrap.waitSeconds (default 600), logging progress, and the Job gets an activeDeadlineSeconds derived from it so a database that never comes up still ends the hook. No change to the binary, API or schema. Claude-Session: https://claude.ai/code/session_016mBLURvJoMuUEr9cB2RpUN --- .../templates/bootstrap-job.yaml | 32 +++++++++++++------ charts/terdut-server/values.yaml | 6 ++++ docs/deployment.md | 1 + 3 files changed, 29 insertions(+), 10 deletions(-) diff --git a/charts/terdut-server/templates/bootstrap-job.yaml b/charts/terdut-server/templates/bootstrap-job.yaml index a64c6b0..3d8078f 100644 --- a/charts/terdut-server/templates/bootstrap-job.yaml +++ b/charts/terdut-server/templates/bootstrap-job.yaml @@ -13,6 +13,10 @@ metadata: helm.sh/hook-delete-policy: before-hook-creation,hook-succeeded spec: backoffLimit: 3 + # Bounds the whole Job, retries included, so a database that never comes up ends the + # hook instead of leaving it pending. Each attempt waits up to bootstrap.waitSeconds + # for the server, and the allowance covers every attempt plus the pod backoff between. + activeDeadlineSeconds: {{ add (mul (int .Values.bootstrap.waitSeconds) 4) 300 }} template: metadata: labels: @@ -66,18 +70,26 @@ spec: BODY=$(echo "$_resp" | sed '$d') } - echo "Waiting for terdut-server to be ready..." - RETRIES=60 - while [ "$RETRIES" -gt 0 ]; do - curl -sf "$SERVICE_URL/healthz" > /dev/null 2>&1 && break - RETRIES=$((RETRIES - 1)) + # The server exits when the database does not answer within ~10 seconds, so + # while the database is down the pod crash-loops, with a restart backoff that + # grows to five minutes, and /healthz only answers once it has connected. Wait + # for that for as long as bootstrap.waitSeconds allows, not a fixed two minutes: + # a failed hook fails the release and nothing runs it again until the next + # upgrade, so a database that was merely slow would leave no administrator. + WAIT={{ int .Values.bootstrap.waitSeconds }} + echo "Waiting up to ${WAIT}s for terdut-server to be ready (it needs the database)..." + WAITED=0 + while ! curl -sf "$SERVICE_URL/healthz" > /dev/null 2>&1; do + if [ "$WAITED" -ge "$WAIT" ]; then + echo "Timed out after ${WAITED}s waiting for the server to be ready." + echo "Check the database and the terdut-server pod's logs, then run helm upgrade again." + exit 1 + fi + [ $((WAITED % 30)) -eq 0 ] && [ "$WAITED" -gt 0 ] && echo "Still waiting (${WAITED}s)..." sleep 2 + WAITED=$((WAITED + 2)) done - if [ "$RETRIES" -eq 0 ]; then - echo "Timed out waiting for server to be ready." - exit 1 - fi - echo "Server is ready." + echo "Server is ready after ${WAITED}s." # The Secret is the source of truth for the administrator's password: reuse # the one it holds, so recreating the database brings the same account back. diff --git a/charts/terdut-server/values.yaml b/charts/terdut-server/values.yaml index 4268e69..eee7949 100644 --- a/charts/terdut-server/values.yaml +++ b/charts/terdut-server/values.yaml @@ -167,6 +167,12 @@ bootstrap: enabled: true username: admin email: admin@example.com + # How long each attempt waits for the server to answer /healthz, which it only does once + # the database is reachable. Raise it when the database starts after the release (a fresh + # cluster, a database operator) -- and keep Flux's HelmRelease spec.timeout / helm + # --timeout above it, because Helm waits for this hook and fails the release when its own + # timeout comes first (the Job still carries on and can finish the bootstrap). + waitSeconds: 600 # secretName overrides the default of -admin-key. It holds username, password and # api-key. The password is generated once and then reused: delete the Secret and the # database to start over. diff --git a/docs/deployment.md b/docs/deployment.md index 3a30f0e..38ef4fd 100644 --- a/docs/deployment.md +++ b/docs/deployment.md @@ -36,6 +36,7 @@ than an `Ingress`. TLS is terminated at the gateway, so the server itself never | `networking.listener` | `""` | Gateway listener (`sectionName`) to bind to. Empty attaches to every matching listener, **including plaintext HTTP** — set it to the HTTPS listener's name to serve TLS only | | `networking.servicePort` | `8080` | Port the route forwards to; keep in sync with `service.port` | | `bootstrap.enabled` | `true` | Runs a post-install/upgrade hook that creates the first administrator with a generated password and stores `username`, `password` and `api-key` in the `-admin-key` Secret. The password is generated once and reused, so recreating the database brings the same account back. Already-bootstrapped servers are left alone. Set `false` to create the first user yourself with `POST /api/bootstrap` | +| `bootstrap.waitSeconds` | `600` | How long each attempt of the hook waits for the server to answer `/healthz`, which it does only once the database is reachable. Keep Helm's `--timeout` (Flux `spec.timeout`) above it; if the timeout comes first the release is marked failed, but the Job keeps running and can still create the administrator | | `bootstrap.username` / `bootstrap.email` | `admin` / `admin@example.com` | Account the hook creates | | `bootstrap.secretName` | `""` | Secret to keep the credentials in; empty means `-admin-key` | | `database.dsn` | `""` | **Required.** Postgres DSN, with no password in it. The chart provisions no database |