Let the bootstrap hook wait for a database that is slow to start

The server exits when the database does not answer within about ten
seconds, so while the database is down the pod crash-loops with a restart
backoff that grows to five minutes, and /healthz only answers after it has
connected. The hook gave up after a fixed two minutes per attempt, and Flux's
default five-minute timeout came first anyway. A failed hook fails the
release and nothing runs it again until the next upgrade, so a database that
was merely slow left the install without an administrator.

Each attempt now waits bootstrap.waitSeconds (default 600), logging progress,
and the Job gets an activeDeadlineSeconds derived from it so a database that
never comes up still ends the hook. No change to the binary, API or schema.

Claude-Session: https://claude.ai/code/session_016mBLURvJoMuUEr9cB2RpUN

charts/terdut-server/templates/bootstrap-job.yaml

@@ -13,6 +13,10 @@ metadata:
     helm.sh/hook-delete-policy: before-hook-creation,hook-succeeded
 spec:
   backoffLimit: 3
+  # Bounds the whole Job, retries included, so a database that never comes up ends the
+  # hook instead of leaving it pending. Each attempt waits up to bootstrap.waitSeconds
+  # for the server, and the allowance covers every attempt plus the pod backoff between.
+  activeDeadlineSeconds: {{ add (mul (int .Values.bootstrap.waitSeconds) 4) 300 }}
   template:
     metadata:
       labels:
@@ -66,18 +70,26 @@ spec:
                 BODY=$(echo "$_resp" | sed '$d')
               }
 
-              echo "Waiting for terdut-server to be ready..."
-              RETRIES=60
-              while [ "$RETRIES" -gt 0 ]; do
-                curl -sf "$SERVICE_URL/healthz" > /dev/null 2>&1 && break
-                RETRIES=$((RETRIES - 1))
+              # The server exits when the database does not answer within ~10 seconds, so
+              # while the database is down the pod crash-loops, with a restart backoff that
+              # grows to five minutes, and /healthz only answers once it has connected. Wait
+              # for that for as long as bootstrap.waitSeconds allows, not a fixed two minutes:
+              # a failed hook fails the release and nothing runs it again until the next
+              # upgrade, so a database that was merely slow would leave no administrator.
+              WAIT={{ int .Values.bootstrap.waitSeconds }}
+              echo "Waiting up to ${WAIT}s for terdut-server to be ready (it needs the database)..."
+              WAITED=0
+              while ! curl -sf "$SERVICE_URL/healthz" > /dev/null 2>&1; do
+                if [ "$WAITED" -ge "$WAIT" ]; then
+                  echo "Timed out after ${WAITED}s waiting for the server to be ready."
+                  echo "Check the database and the terdut-server pod's logs, then run helm upgrade again."
+                  exit 1
+                fi
+                [ $((WAITED % 30)) -eq 0 ] && [ "$WAITED" -gt 0 ] && echo "Still waiting (${WAITED}s)..."
                 sleep 2
+                WAITED=$((WAITED + 2))
               done
-              if [ "$RETRIES" -eq 0 ]; then
-                echo "Timed out waiting for server to be ready."
-                exit 1
-              fi
-              echo "Server is ready."
+              echo "Server is ready after ${WAITED}s."
 
               # The Secret is the source of truth for the administrator's password: reuse
               # the one it holds, so recreating the database brings the same account back.

charts/terdut-server/values.yaml

@@ -167,6 +167,12 @@ bootstrap:
   enabled: true
   username: admin
   email: admin@example.com
+  # How long each attempt waits for the server to answer /healthz, which it only does once
+  # the database is reachable. Raise it when the database starts after the release (a fresh
+  # cluster, a database operator) -- and keep Flux's HelmRelease spec.timeout / helm
+  # --timeout above it, because Helm waits for this hook and fails the release when its own
+  # timeout comes first (the Job still carries on and can finish the bootstrap).
+  waitSeconds: 600
   # secretName overrides the default of <fullname>-admin-key. It holds username, password and
   # api-key. The password is generated once and then reused: delete the Secret and the
   # database to start over.