Set the chart's placeholder version to 0.5.0

Cosmetic: make helm-package passes --version and --app-version from the
tag, so these two fields decide nothing about what gets published. Still
done, as with d502485 (0.4.0) and 88172ad (0.3.0) before it, because a
tree heading for v0.5.0 that still says 0.4.0 tells its reader
something false.

Co-authored-by: Claude <noreply@anthropic.com>

DESIGN.md

@@ -169,6 +169,8 @@ spec:
     matchers: "alertname=Watchdog"
     timeout: 15m
     severity: critical
+  credentials:
+    deletionPolicy: Retain         # Retain (default) | Delete -- what deleting this CR does to the instance credential Secret; see §6 point 2
   notify:
     ntfyURL: "http://ntfy.ntfy.svc.cluster.local"
     fallbackTopic: ""
@@ -529,6 +531,18 @@ when nothing ever crosses into a tenant namespace in the first place.
    the same rigor §5's general idempotent-create rule already applies
    elsewhere:
    - `status.credentialsSecretRef` already set: done, nothing to do.
+   - Otherwise, look for the instance credential Secret itself
+     (`<namespace>.<name>-instance-credentials`, point 2 below): a
+     `TerdutServer` deleted and recreated under the same name leaves it
+     behind by default (`spec.credentials.deletionPolicy: Retain`), and the
+     server it logs in to has not changed, because deleting a
+     `TerdutServer` never touches its database. If it exists, ask the server
+     for the operator's own service account with that token
+     (`GET /api/service-accounts?name=terdut-operator`). Accepted: adopt it,
+     set `status.credentialsSecretRef`, and skip everything below. Rejected
+     with a `401`/`403`: it is stale (a database reset since), so ignore it
+     and carry on; the steps below replace it. Any other failure is a retry,
+     not a guess.
    - Otherwise, check for an intermediate
      `<namespace>.<name>-bootstrap-admin` Secret in
      the operator's own namespace first. If it exists, its key is a still-
@@ -538,15 +552,21 @@ when nothing ever crosses into a tenant namespace in the first place.
      on `201`, immediately checkpoint its response's raw admin key
      (`{"user": ..., "api_key": {"key": "<raw>", ...}}`) into that Secret
      before doing anything else with it. A `403` with neither
-     `status.credentialsSecretRef` nor this checkpoint Secret present is
+     `status.credentialsSecretRef`, nor an instance credential the server
-     the one genuinely pathological case left (the checkpoint deleted out
+     accepts, nor this checkpoint Secret present is the one genuinely
-     from under a reconcile already past this point) — handled the same
+     pathological case left: the server's database is already bootstrapped
-     way the design already handles unrecoverable server-issued material
+     and no credential for it survives here (the checkpoint deleted out from
-     elsewhere (§5's webhook-Secret-loss rule): fail closed,
+     under a reconcile already past this point, `deletionPolicy: Delete`, or
-     `Ready: False, reason: BootstrapStateLost`, with the same recovery as
+     the Secret removed by hand). Fail closed, `Ready: False, reason:
-     that case, delete and recreate the `TerdutServer` (its finalizer tears
+     BootstrapStateLost`, with a message that names the instance credential
-     down the Deployment/database-backing and server-side rows; a fresh
+     Secret to restore. Deleting and recreating the `TerdutServer` does **not**
-     create starts clean) — not a workaround peculiar to this one path.
+     recover from it: the finalizer removes Secrets only, and the database
+     (the one thing that still says "already bootstrapped") is not the
+     operator's to reset. Restore the Secret from a copy, or reset the
+     server's database and then recreate the `TerdutServer`, which then
+     bootstraps like a first install. (This section used to say a fresh
+     create "starts clean"; that was only ever true when the database went
+     with it.)
    - With an admin key in hand (fresh or checkpointed): `POST
      /api/service-accounts {name: "terdut-operator", scope: "instance"}`.
      A `409` here means a prior attempt got this far before being
@@ -563,10 +583,15 @@ when nothing ever crosses into a tenant namespace in the first place.
    under a fixed data key, `token`), referenced back from
    `TerdutServer.status.credentialsSecretRef: {name, key}` (§4.1). No
    `OwnerReference` (those can't cross namespaces, and this Secret doesn't
-   share a namespace with the `TerdutServer` that caused it); the `TerdutServer`'s
+   share a namespace with the `TerdutServer` that caused it). What the
-   finalizer deletes this Secret directly as part of its own teardown,
+   `TerdutServer`'s finalizer does with it is `spec.credentials.deletionPolicy`:
-   the same way it already has to clean up the server-side resources it
+   `Retain` (the default) leaves it for a recreated `TerdutServer` to adopt
-   created (§5's general finalizer rule extends naturally to this Secret).
+   (point 1), `Delete` removes it directly as part of the teardown. There are
+   no server-side resources to undo either way: the bootstrap user and
+   service account have no delete verb in terdut-server's API. The bootstrap
+   checkpoint Secret is always removed. A retained Secret of a `TerdutServer`
+   that is gone for good is an orphan to delete by hand, and it is inert:
+   adoption asks the server to accept the token first.
 3. When a `TerdutTeam` first becomes `Ready` (its `serverRef` resolved,
    `allowedTeams` satisfied if cross-namespace), its controller uses the
    `TerdutServer`'s instance-scoped credential (read from the operator's own

api/v1alpha1/terdutserver_types.go

@@ -23,6 +23,37 @@ type SecretKeyRef struct {
 	Key string `json:"key"`
 }
 
+// CredentialsDeletionPolicy is what deleting a TerdutServer does to the
+// instance credential Secret the operator generated for it.
+// +kubebuilder:validation:Enum=Retain;Delete
+type CredentialsDeletionPolicy string
+
+const (
+	// CredentialsRetain keeps the Secret when the TerdutServer is deleted, so
+	// a TerdutServer recreated with the same name and namespace against the
+	// same database adopts it again instead of finding a server it cannot
+	// log in to. Deleting a TerdutServer never touches its database, so the
+	// operator's service account is still there to be reused. The default.
+	CredentialsRetain CredentialsDeletionPolicy = "Retain"
+	// CredentialsDelete removes the Secret with the TerdutServer. Choose it
+	// when the database goes too, or when the credential must not outlive the
+	// object.
+	CredentialsDelete CredentialsDeletionPolicy = "Delete"
+)
+
+// CredentialsSpec configures the lifecycle of the generated instance
+// credential.
+type CredentialsSpec struct {
+	// deletionPolicy: whether the instance credential Secret is kept
+	// (Retain, the default) or removed (Delete) when this TerdutServer is
+	// deleted. A kept Secret is only ever adopted after the server accepts
+	// its token, so one left over from a database that has since been reset
+	// is ignored and replaced.
+	// +kubebuilder:default=Retain
+	// +optional
+	DeletionPolicy CredentialsDeletionPolicy `json:"deletionPolicy,omitempty"`
+}
+
 // ImageSpec is the terdut-server image to run.
 type ImageSpec struct {
 	// +kubebuilder:validation:MinLength=1
@@ -324,6 +355,11 @@ type TerdutServerSpec struct {
 	// +optional
 	Deadman DeadmanSpec `json:"deadman,omitempty"`
 
+	// credentials: what happens to the instance credential this operator
+	// generates for the server.
+	// +optional
+	Credentials CredentialsSpec `json:"credentials,omitempty"`
+
 	// +optional
 	Notify NotifySpec `json:"notify,omitempty"`
 
@@ -382,9 +418,12 @@ const (
 	// ReasonBootstrapStateLost: a checkpointed admin credential
 	// (DESIGN.md §6) was lost after being used but before the lasting
 	// credential it was for could be persisted -- the one genuinely
-	// pathological case in the self-registration flow. Fail-closed, same
+	// pathological case in the self-registration flow -- or the server's
-	// recovery as DESIGN.md §5's webhook-Secret-loss rule: delete and
+	// database is already bootstrapped and no credential for it survives
-	// recreate this TerdutServer.
+	// (spec.credentials.deletionPolicy: Delete, or the Secret removed by
+	// hand). Fail-closed: the operator cannot mint a credential, and
+	// deleting and recreating the TerdutServer does not clear the database.
+	// Restore the Secret, or reset the server's database.
 	ReasonBootstrapStateLost = "BootstrapStateLost"
 	// ReasonAdopted: the happy path. A working credential is in hand, the
 	// Deployment has a ready replica, and the database (if postgresClusterRef)

api/v1alpha1/zz_generated.deepcopy.go

@@ -47,6 +47,21 @@ func (in *AllowedTeamsNamespaces) DeepCopy() *AllowedTeamsNamespaces {
 	return out
 }
 
+// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil.
+func (in *CredentialsSpec) DeepCopyInto(out *CredentialsSpec) {
+	*out = *in
+}
+
+// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new CredentialsSpec.
+func (in *CredentialsSpec) DeepCopy() *CredentialsSpec {
+	if in == nil {
+		return nil
+	}
+	out := new(CredentialsSpec)
+	in.DeepCopyInto(out)
+	return out
+}
+
 // DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil.
 func (in *DatabaseSpec) DeepCopyInto(out *DatabaseSpec) {
 	*out = *in
@@ -764,6 +779,7 @@ func (in *TerdutServerSpec) DeepCopyInto(out *TerdutServerSpec) {
 	in.Database.DeepCopyInto(&out.Database)
 	out.Sweeper = in.Sweeper
 	out.Deadman = in.Deadman
+	out.Credentials = in.Credentials
 	in.Notify.DeepCopyInto(&out.Notify)
 	in.OIDC.DeepCopyInto(&out.OIDC)
 	in.AllowedTeams.DeepCopyInto(&out.AllowedTeams)

charts/terdut-operator/Chart.yaml

@@ -6,8 +6,8 @@ type: application
 # These fields decide nothing: `make helm-package` passes --version and
 # --app-version from the release tag (same reasoning as terdut-server's own
 # chart). They're for whoever reads the tree before a tag exists.
-version: 0.4.0
+version: 0.5.0
-appVersion: "v0.4.0"
+appVersion: "v0.5.0"
 
 keywords:
   - kubernetes

charts/terdut-operator/templates/crd/terdutservers.terdut.ryuvia.com.yaml

@@ -128,6 +128,24 @@ spec:
                         x-kubernetes-map-type: atomic
                     type: object
                 type: object
+              credentials:
+                description: |-
+                  credentials: what happens to the instance credential this operator
+                  generates for the server.
+                properties:
+                  deletionPolicy:
+                    default: Retain
+                    description: |-
+                      deletionPolicy: whether the instance credential Secret is kept
+                      (Retain, the default) or removed (Delete) when this TerdutServer is
+                      deleted. A kept Secret is only ever adopted after the server accepts
+                      its token, so one left over from a database that has since been reset
+                      is ignored and replaced.
+                    enum:
+                    - Retain
+                    - Delete
+                    type: string
+                type: object
               database:
                 description: |-
                   DatabaseSpec is the Postgres connection this TerdutServer uses. Exactly

charts/terdut-operator/templates/terdutserver/terdutserver.yaml

@@ -38,6 +38,10 @@ spec:
   deadman:
     {{- toYaml . | nindent 4 }}
   {{- end }}
+  {{- with .Values.terdutServer.credentials }}
+  credentials:
+    {{- toYaml . | nindent 4 }}
+  {{- end }}
   {{- with .Values.terdutServer.notify }}
   notify:
     {{- toYaml . | nindent 4 }}

charts/terdut-operator/values.yaml

@@ -270,6 +270,13 @@ terdutServer:
   #   matchers: "alertname=Watchdog"
   #   timeout: 15m
   #   severity: critical
+  # What happens to the instance credential the operator generates for this
+  # server when the TerdutServer is deleted. Retain (the default) keeps it, so
+  # a TerdutServer recreated with the same name against the same database
+  # adopts it again; Delete removes it with the TerdutServer. A kept Secret
+  # is only adopted if the server accepts its token.
+  # credentials:
+  #   deletionPolicy: Retain
   # notify: {}
   # oidc: {}
 

config/crd/bases/terdut.ryuvia.com_terdutservers.yaml

@@ -125,6 +125,24 @@ spec:
                         x-kubernetes-map-type: atomic
                     type: object
                 type: object
+              credentials:
+                description: |-
+                  credentials: what happens to the instance credential this operator
+                  generates for the server.
+                properties:
+                  deletionPolicy:
+                    default: Retain
+                    description: |-
+                      deletionPolicy: whether the instance credential Secret is kept
+                      (Retain, the default) or removed (Delete) when this TerdutServer is
+                      deleted. A kept Secret is only ever adopted after the server accepts
+                      its token, so one left over from a database that has since been reset
+                      is ignored and replaced.
+                    enum:
+                    - Retain
+                    - Delete
+                    type: string
+                type: object
               database:
                 description: |-
                   DatabaseSpec is the Postgres connection this TerdutServer uses. Exactly

examples/demo/01-server.yaml

@@ -14,13 +14,22 @@ metadata:
 spec:
   image:
     repository: git.ryuvia.com/niklas/terdut-server
-    # v0.34.0: fixes callerMayManageServiceAccount so an instance-scoped
+    # v0.36.0 is the floor now that replicas below is 2 (this demo pins
-    # service account can adopt/rotate a key on a team-scoped account it
+    # the current release, v0.43.0, so it shows the current web UI too): that
-    # didn't just create in the same call -- without this, terdutteam-*
+    # release put the sweeper, the notifier and the migration runner each
-    # can wedge permanently on exactly the crash-window race this demo
+    # behind a Postgres advisory lock, and gave incident creation its own
-    # hit live (niklas/terdut-operator#3).
+    # conflict resolution, which is what makes a second replica safe
-    tag: v0.34.0
+    # instead of racing the first. (Still carries v0.34.0's fix too --
-  replicas: 1
+    # callerMayManageServiceAccount, so an instance-scoped service account
+    # can adopt/rotate a key on a team-scoped account it didn't just create
+    # in the same call -- without which terdutteam-* can wedge permanently
+    # on the crash-window race this demo hit live, niklas/terdut-operator#3.)
+    tag: v0.43.0
+  # Matches this CRD's own spec.replicas default (v0.4.0) -- stated
+  # explicitly, like every other field in this file, rather than left to
+  # the default. RollingUpdate follows automatically; this operator does
+  # not expose Strategy as a spec field.
+  replicas: 2
   networking:
     hostname: terdut-operator-demo.example
     servicePort: 8080

examples/demo/README.md

@@ -133,6 +133,20 @@ Each `(team, scenario)` pair is one stable fingerprint, so firing the same
 one twice updates the same alert (a real re-fire) and `resolve` closes
 exactly that one.
 
+Set `CLUSTER` to send the alert as if it came from one of several clusters:
+
+```sh
+CLUSTER=prod-eu ./fire-alerts.sh platform high-cpu
+CLUSTER=prod-us ./fire-alerts.sh platform high-cpu   # a second incident, not a join
+CLUSTER=prod-eu ./fire-alerts.sh platform high-cpu resolve
+```
+
+It stands in for a Prometheus external label plus `cluster` in Alertmanager's
+`group_by` (terdut-server's README, "Several clusters, one team"): the web UI
+then shows the cluster chip on each incident and a cluster filter in the
+queue. `CLUSTER` is part of the fingerprint, so resolve with the same value you
+fired with. `./run-demo.sh` fires its alerts across `prod-eu` and `prod-us`.
+
 ### Dead man's switches
 
 `06-deadman-platform.yaml` / `07-deadman-payments.yaml` expect a heartbeat

examples/demo/fire-alerts.sh

@@ -16,7 +16,15 @@
 # is the one part of that URL still usable here.
 #
 # Usage:
-#   ./fire-alerts.sh <platform|payments> <high-cpu|disk-full|pod-crash|heartbeat> [resolve]
+#   [CLUSTER=prod-eu] ./fire-alerts.sh <platform|payments> <high-cpu|disk-full|pod-crash|heartbeat> [resolve]
+#
+# CLUSTER stands in for a Prometheus externalLabel plus `cluster` in
+# Alertmanager's group_by (terdut-server's README, "Several clusters, one
+# team"): it is put on the alert's labels and on groupLabels, so the incident
+# carries it and the web UI shows the cluster chip and the queue's cluster
+# filter. It is also part of the group key and the fingerprint, which is what
+# keeps the same alert in two clusters from joining one incident. Unset, the
+# alert is sent exactly as before.
 #
 # Prerequisites: kubectl context pointed at the demo namespace, jq, curl,
 # and (in another terminal) a running:
@@ -24,6 +32,7 @@
 set -euo pipefail
 
 NAMESPACE="${NAMESPACE:-}"
+CLUSTER="${CLUSTER:-}"
 BASE_URL="${BASE_URL:-http://localhost:8080}"
 
 usage() {
@@ -45,6 +54,8 @@ scenarios:
 env vars:
   NAMESPACE  kubectl -n for reading the webhook Secret (required)
   BASE_URL   where the port-forwarded terdut-server is (default http://localhost:8080)
+  CLUSTER    optional cluster name, e.g. prod-eu: sent as a `cluster` label and
+             group label, so the UI shows where the incident came from
 EOF
   exit 1
 }
@@ -90,7 +101,7 @@ key="$(kubectl -n "$NAMESPACE" get secret "$secret_name" -o jsonpath='{.data.key
 # created: terdut-server correlates on (team_id, fingerprint), not on
 # anything else in the payload. Real Alertmanager computes this from the
 # alert's label set; a fixed string plays the same role here.
-fingerprint="demo-${team}-${scenario}"
+fingerprint="demo-${team}-${scenario}${CLUSTER:+-$CLUSTER}"
 
 now="$(date -u +%Y-%m-%dT%H:%M:%SZ)"
 if [ "$status" = firing ]; then
@@ -101,7 +112,8 @@ fi
 
 payload="$(jq -n \
   --arg status "$status" \
-  --arg groupKey "demo:${team}:${scenario}" \
+  --arg groupKey "demo:${team}:${scenario}${CLUSTER:+:$CLUSTER}" \
+  --arg cluster "$CLUSTER" \
   --arg alertname "$alertname" \
   --arg team "$team" \
   --arg severity "$severity" \
@@ -113,10 +125,10 @@ payload="$(jq -n \
     version: "4",
     status: $status,
     groupKey: $groupKey,
-    groupLabels: { alertname: $alertname, team: $team },
+    groupLabels: ({ alertname: $alertname, team: $team } + (if $cluster != "" then { cluster: $cluster } else {} end)),
     alerts: [{
       status: $status,
-      labels: { alertname: $alertname, severity: $severity, team: $team, instance: "demo" },
+      labels: ({ alertname: $alertname, severity: $severity, team: $team, instance: "demo" } + (if $cluster != "" then { cluster: $cluster } else {} end)),
       annotations: { summary: $summary },
       startsAt: $startsAt,
       endsAt: $endsAt,
@@ -126,7 +138,7 @@ payload="$(jq -n \
   }')"
 
 url="${BASE_URL}/api/integrations/${key}/alertmanager"
-echo "POST $url  (team=$team scenario=$scenario status=$status)" >&2
+echo "POST $url  (team=$team scenario=$scenario status=$status${CLUSTER:+ cluster=$CLUSTER})" >&2
 code="$(curl -sS -o /tmp/fire-alerts-response.json -w '%{http_code}' \
   -X POST "$url" -H 'Content-Type: application/json' -d "$payload")"
 echo "-> HTTP $code" >&2

examples/demo/run-demo.sh

@@ -107,26 +107,41 @@ apply_demo() {
   kubectl apply -n "$NAMESPACE" -k "$SCRIPT_DIR" >/dev/null
 }
 
-wait_for_ready() {
+wait_for_objects() {
-  local objects=(
-    "terdutserver/terdut-operator-demo"
-    "terdutteam/terdutteam-platform"
-    "terdutteam/terdutteam-payments"
-    "terdutescalationrule/terdutescalationrule-platform"
-    "terdutescalationrule/terdutescalationrule-payments"
-    "terdutdeadmanswitch/terdutdeadmanswitch-platform"
-    "terdutdeadmanswitch/terdutdeadmanswitch-payments"
-    "terdutalertsource/terdutalertsource-platform"
-    "terdutalertsource/terdutalertsource-payments"
-  )
   local obj
-  for obj in "${objects[@]}"; do
+  for obj in "$@"; do
     log "waiting for $obj to become Ready"
     kubectl wait --for=condition=Ready --timeout "$WAIT_TIMEOUT" -n "$NAMESPACE" "$obj" >/dev/null \
       || die "timed out waiting for $obj -- try: kubectl describe -n $NAMESPACE $obj"
   done
 }
 
+# Just the server and the two teams -- everything redeem_platform_invite and
+# join_payments_team need. Deliberately NOT the escalation rules here: this
+# demo kit's own terdutescalationrule-platform names alice as a level-1
+# target, and that CR cannot reach Ready until alice actually exists
+# (terdut-server resolves every named username at reconcile time, not just
+# at escalation time) -- a real dependency this script has to satisfy by
+# creating her first, not something kubectl wait can be told to ignore.
+wait_for_teams_ready() {
+  wait_for_objects \
+    "terdutserver/terdut-operator-demo" \
+    "terdutteam/terdutteam-platform" \
+    "terdutteam/terdutteam-payments"
+}
+
+# Everything that was waiting on alice (or just on the teams above, now
+# already satisfied) to exist.
+wait_for_remaining_ready() {
+  wait_for_objects \
+    "terdutescalationrule/terdutescalationrule-platform" \
+    "terdutescalationrule/terdutescalationrule-payments" \
+    "terdutdeadmanswitch/terdutdeadmanswitch-platform" \
+    "terdutdeadmanswitch/terdutdeadmanswitch-payments" \
+    "terdutalertsource/terdutalertsource-platform" \
+    "terdutalertsource/terdutalertsource-payments"
+}
+
 start_port_forward() {
   # A stale pidfile from an earlier run would otherwise collide with us on
   # $LOCAL_PORT -- if that pid is still alive, stop it first.
@@ -181,6 +196,18 @@ redeem_platform_invite() {
   invite_token="${invite_url##*invite=}"
   [ -n "$invite_token" ] || die "could not parse an invite token out of $invite_url"
 
+  # A re-run: the invite was spent by the first run, and the server answers a
+  # spent invite with 403 before it ever looks at the username, so the 409
+  # handled below never arrives. If alice can already sign in, she exists.
+  local login_code
+  login_code="$(curl -sS -o /dev/null -w '%{http_code}' \
+    -X POST "${BASE_URL}/api/login" -H 'Content-Type: application/json' \
+    -d "$(jq -n --arg u "$ALICE_USERNAME" --arg p "$DEMO_PASSWORD" '{username: $u, password: $p}')")"
+  if [ "$login_code" = "200" ]; then
+    log "account ${ALICE_USERNAME} already exists and can sign in, skipping signup (re-run detected)"
+    return 0
+  fi
+
   log "signing up ${ALICE_USERNAME} via Platform's invite"
   local body resp_file code
   body="$(jq -n \
@@ -235,9 +262,13 @@ join_payments_team() {
 
 fire_demo_alerts() {
   log "firing representative demo alerts"
-  NAMESPACE="$NAMESPACE" BASE_URL="$BASE_URL" "$SCRIPT_DIR/fire-alerts.sh" platform high-cpu
+  # Two clusters, so the queue shows the cluster chip and offers its filter.
-  NAMESPACE="$NAMESPACE" BASE_URL="$BASE_URL" "$SCRIPT_DIR/fire-alerts.sh" platform disk-full
+  # high-cpu fires in both: the same alert in two clusters is two incidents.
-  NAMESPACE="$NAMESPACE" BASE_URL="$BASE_URL" "$SCRIPT_DIR/fire-alerts.sh" payments pod-crash
+  local fire="$SCRIPT_DIR/fire-alerts.sh"
+  CLUSTER=prod-eu NAMESPACE="$NAMESPACE" BASE_URL="$BASE_URL" "$fire" platform high-cpu
+  CLUSTER=prod-us NAMESPACE="$NAMESPACE" BASE_URL="$BASE_URL" "$fire" platform high-cpu
+  CLUSTER=prod-us NAMESPACE="$NAMESPACE" BASE_URL="$BASE_URL" "$fire" platform disk-full
+  CLUSTER=prod-eu NAMESPACE="$NAMESPACE" BASE_URL="$BASE_URL" "$fire" payments pod-crash
 }
 
 print_summary() {
@@ -254,8 +285,8 @@ terdut demo is up.
 
   Fire more alerts:
     export NAMESPACE=${NAMESPACE} BASE_URL=${BASE_URL}
-    ./fire-alerts.sh platform high-cpu
+    CLUSTER=prod-eu ./fire-alerts.sh platform high-cpu
-    ./fire-alerts.sh platform high-cpu resolve
+    CLUSTER=prod-eu ./fire-alerts.sh platform high-cpu resolve
     ./fire-alerts.sh payments heartbeat   # send repeatedly (e.g. every minute)
                                           # to keep a dead man's switch alive;
                                           # stop sending it and, 15 minutes
@@ -319,10 +350,11 @@ main() {
   ensure_kind_cluster
   install_operator
   apply_demo
-  wait_for_ready
+  wait_for_teams_ready
   start_port_forward
   redeem_platform_invite
   join_payments_team
+  wait_for_remaining_ready
   fire_demo_alerts
   print_summary
 

internal/controller/terdutserver_bootstrap.go

@@ -16,18 +16,24 @@ import (
 )
 
 // bootstrapStateLostError is DESIGN.md §6's one genuinely pathological
-// case: a checkpointed admin credential was used and then lost before the
+// case: the server's database is already bootstrapped, and no credential for
-// lasting credential it was for could be persisted. Distinct from a plain
+// it survives here -- a checkpointed admin key was used and then lost before
-// error so Reconcile can route it to a Ready: False condition (the
+// the lasting credential could be persisted, or the instance credential
-// documented recovery is delete-and-recreate, not an automatic retry) rather
+// Secret was deleted (spec.credentials.deletionPolicy: Delete, or by hand).
-// than treating it as a transient reconcile failure.
+// Distinct from a plain error so Reconcile can route it to a Ready: False
-type bootstrapStateLostError struct{ detail string }
+// condition rather than treating it as a transient reconcile failure: no
+// retry can fix it.
+type bootstrapStateLostError struct {
+	detail     string
+	secretName string // the instance credential Secret that would have fixed it
+}
 
 func (e *bootstrapStateLostError) Error() string {
 	return fmt.Sprintf(
-		"server reports already bootstrapped, but neither status.credentialsSecretRef nor a "+
+		"server reports already bootstrapped, but there is no credential for it here: %s. "+
-			"checkpointed admin credential exist here: %s. This TerdutServer cannot recover a "+
+			"The operator cannot mint one on its own, and deleting and recreating this TerdutServer does "+
-			"credential on its own; delete and recreate it", e.detail)
+			"not clear the database. Restore Secret %q in the operator's namespace if you have a copy, "+
+			"or reset the server's database and recreate this TerdutServer", e.detail, e.secretName)
 }
 
 // reconcileBootstrap implements DESIGN.md §6 point 1's self-registration
@@ -36,6 +42,17 @@ func (e *bootstrapStateLostError) Error() string {
 // srv.Status.CredentialsSecretRef is nil and the Deployment has a ready
 // replica.
 func (r *TerdutServerReconciler) reconcileBootstrap(ctx context.Context, srv *terdutv1alpha1.TerdutServer) error {
+	// A TerdutServer recreated against a database that is already
+	// bootstrapped: its instance credential may still be here (the default
+	// deletion policy keeps it), and then there is nothing to bootstrap.
+	adopted, err := r.adoptRetainedCredentials(ctx, srv)
+	if err != nil {
+		return err
+	}
+	if adopted {
+		return nil
+	}
+
 	adminKey, err := r.getOrCreateCheckpointedAdminKey(ctx, srv)
 	if err != nil {
 		return err
@@ -64,6 +81,51 @@ func (r *TerdutServerReconciler) reconcileBootstrap(ctx context.Context, srv *te
 	return nil
 }
 
+// adoptRetainedCredentials looks for the instance credential Secret an
+// earlier TerdutServer of the same name and namespace left behind
+// (spec.credentials.deletionPolicy: Retain), and adopts it if the server
+// still accepts its token. Reports whether it did.
+//
+// The token is tried before it is trusted: a Secret that outlived a database
+// reset holds a key the server has never heard of, and adopting that would
+// make every later call 401. A rejected token (401/403) is not an error here;
+// it just means there is nothing to adopt, and bootstrap proceeds as it would
+// for a first install -- which succeeds against a freshly reset database and
+// replaces the Secret. Anything else (the server unreachable, a 5xx) is
+// returned for a retry rather than guessed at.
+func (r *TerdutServerReconciler) adoptRetainedCredentials(ctx context.Context, srv *terdutv1alpha1.TerdutServer) (bool, error) {
+	name := credentialsSecretName(srv)
+	var sec corev1.Secret
+	if err := r.Get(ctx, client.ObjectKey{Namespace: r.OperatorNamespace, Name: name}, &sec); err != nil {
+		if apierrors.IsNotFound(err) {
+			return false, nil
+		}
+		return false, err
+	}
+	token := string(sec.Data[credentialsSecretDataKey])
+	if token == "" {
+		return false, nil
+	}
+
+	// The operator's own service account is the one thing this credential
+	// is for, so asking the server for it by name checks the token and that
+	// it belongs to that account in the same call.
+	sa, err := r.NewClient(serviceURL(srv)).WithToken(token).GetServiceAccountByName(ctx, serviceAccountName)
+	if err != nil {
+		if statusErr, ok := errors.AsType[*tdclient.StatusError](err); ok &&
+			(statusErr.Code == http.StatusUnauthorized || statusErr.Code == http.StatusForbidden) {
+			return false, nil
+		}
+		return false, fmt.Errorf("checking the retained credential in Secret %q: %w", name, err)
+	}
+	if sa == nil {
+		return false, nil
+	}
+
+	srv.Status.CredentialsSecretRef = &terdutv1alpha1.SecretKeyRef{Name: name, Key: credentialsSecretDataKey}
+	return true, nil
+}
+
 // getOrCreateCheckpointedAdminKey returns a usable admin key: from the
 // checkpoint Secret if an earlier, interrupted attempt already got one, or
 // freshly from /api/bootstrap, immediately checkpointed before it's used
@@ -89,7 +151,10 @@ func (r *TerdutServerReconciler) getOrCreateCheckpointedAdminKey(ctx context.Con
 			// status.credentialsSecretRef) means a prior reconcile already
 			// won this exact race and its checkpoint was lost afterward --
 			// the one case §6 doesn't try to paper over.
-			return "", &bootstrapStateLostError{detail: "/api/bootstrap returned 403"}
+			return "", &bootstrapStateLostError{
+				detail:     "/api/bootstrap returned 403",
+				secretName: credentialsSecretName(srv),
+			}
 		}
 		return "", fmt.Errorf("POST /api/bootstrap: %w", err)
 	}

internal/controller/terdutserver_controller.go

@@ -39,10 +39,10 @@ const resyncInterval = 5 * time.Minute
 // than "someone edited something out of band."
 const waitInterval = 15 * time.Second
 
-// finalizerName cleans up the credentials Secret(s) this controller
+// finalizerName cleans up the Secret(s) this controller generates in the
-// generates in the operator's own namespace on delete — the Deployment and
+// operator's own namespace on delete (which of them, spec.credentials.
-// Service are owned (OwnerReference, DESIGN.md §7) and need no finalizer of
+// deletionPolicy decides) — the Deployment and Service are owned
-// their own.
+// (OwnerReference, DESIGN.md §7) and need no finalizer of their own.
 const finalizerName = "terdut.ryuvia.com/terdutserver"
 
 // serviceAccountName is the name the operator registers itself under
@@ -215,18 +215,31 @@ func (r *TerdutServerReconciler) setNotReady(
 	return ctrl.Result{RequeueAfter: d}, nil
 }
 
-// reconcileDelete cleans up the credentials Secret(s) this controller
+// reconcileDelete cleans up the Secrets this controller generated in the
-// generated in the operator's own namespace. The Deployment and Service are
+// operator's own namespace. The Deployment and Service are owned
-// owned (OwnerReference, DESIGN.md §7) and need no attention here — normal
+// (OwnerReference, DESIGN.md §7) and need no attention here — normal GC
-// GC handles them. There is no server-side "delete this install" call to
+// handles them. There is no server-side "delete this install" call to
 // make: bootstrap created a user and a service account, and terdut-server's
 // API has no way to delete either (only to revoke individual keys), so
-// there is nothing meaningful to undo there either.
+// there is nothing meaningful to undo there either, and the database is
+// never touched.
+//
+// That is why the instance credential is kept by default
+// (spec.credentials.deletionPolicy: Retain). Everything it logs in to
+// outlives the TerdutServer, so a recreated one finds a bootstrapped server
+// it has no key for -- unless the key is still here to be adopted (see
+// adoptRetainedCredentials). The bootstrap checkpoint is always removed: it
+// is a short-lived admin key, and the instance credential is all that is
+// needed afterwards.
 func (r *TerdutServerReconciler) reconcileDelete(ctx context.Context, srv *terdutv1alpha1.TerdutServer) (ctrl.Result, error) {
 	if !controllerutil.ContainsFinalizer(srv, finalizerName) {
 		return ctrl.Result{}, nil
 	}
-	for _, name := range []string{checkpointSecretName(srv), credentialsSecretName(srv)} {
+	names := []string{checkpointSecretName(srv)}
+	if srv.Spec.Credentials.DeletionPolicy == terdutv1alpha1.CredentialsDelete {
+		names = append(names, credentialsSecretName(srv))
+	}
+	for _, name := range names {
 		sec := &corev1.Secret{ObjectMeta: metav1.ObjectMeta{Name: name, Namespace: r.OperatorNamespace}}
 		if err := r.Delete(ctx, sec); err != nil && !apierrors.IsNotFound(err) {
 			return ctrl.Result{}, err

internal/controller/terdutserver_controller_test.go

@@ -54,9 +54,13 @@ type fakeTerdutServer struct {
 	mu           sync.Mutex
 	bootstrapped bool
 	bootstrap403 bool // force every /api/bootstrap call to 403, even the first
-	nextID       int64
+	// rejectedTokens: bearer tokens the fake answers 401 on GET
-	accounts     map[string]int64 // name -> id
+	// /api/service-accounts, the way a server that never issued the key
-	keyMints     map[int64]int    // id -> number of keys minted so far
+	// (a database reset since) would.
+	rejectedTokens map[string]bool
+	nextID         int64
+	accounts       map[string]int64 // name -> id
+	keyMints       map[int64]int    // id -> number of keys minted so far
 
 	nextTeamID int64
 	teams      map[string]int64 // name -> id
@@ -170,6 +174,10 @@ func (f *fakeTerdutServer) ServeHTTP(w http.ResponseWriter, r *http.Request) {
 		})
 
 	case r.URL.Path == "/api/service-accounts" && r.Method == http.MethodGet:
+		if f.rejectedTokens[strings.TrimPrefix(r.Header.Get("Authorization"), "Bearer ")] {
+			writeJSON(w, http.StatusUnauthorized, map[string]string{errJSONKey: "invalid or expired API key"})
+			return
+		}
 		name := r.URL.Query().Get("name")
 		id, exists := f.accounts[name]
 		if !exists {
@@ -875,13 +883,14 @@ var _ = Describe("TerdutServer Controller", func() {
 	})
 
 	Describe("deletion", func() {
-		It("removes the credentials and checkpoint Secrets and the finalizer", func(ctx SpecContext) {
+		// runToReady brings a server to Ready against a fresh fake and
-			fake, fakeSrv := newFakeTerdutServer()
+		// returns the name of the instance credential Secret it minted.
-			_ = fake
+		runToReady := func(ctx SpecContext, spec terdutv1alpha1.TerdutServerSpec) string {
+			_, fakeSrv := newFakeTerdutServer()
 			DeferCleanup(fakeSrv.Close)
 			reconciler.NewClient = func(string) *tdclient.Client { return tdclient.New(fakeSrv.URL) }
 
-			createServer(ctx, dsnSpec())
+			createServer(ctx, spec)
 			reconcileOnce(ctx)
 			reconcileOnce(ctx)
 			markDeploymentReady(ctx)
@@ -889,17 +898,114 @@ var _ = Describe("TerdutServer Controller", func() {
 
 			srv := &terdutv1alpha1.TerdutServer{}
 			Expect(k8sClient.Get(ctx, objKey, srv)).To(Succeed())
-			credsName := srv.Status.CredentialsSecretRef.Name
+			return srv.Status.CredentialsSecretRef.Name
-
+		}
+		deleteAndFinalize := func(ctx SpecContext) {
+			srv := &terdutv1alpha1.TerdutServer{}
+			Expect(k8sClient.Get(ctx, objKey, srv)).To(Succeed())
 			Expect(k8sClient.Delete(ctx, srv)).To(Succeed())
 			reconcileOnce(ctx) // runs the finalizer
+			Expect(k8sClient.Get(ctx, objKey, srv)).NotTo(Succeed(),
+				"the TerdutServer itself should be gone once the finalizer clears")
+		}
+		secretExists := func(ctx SpecContext, secretName string) bool {
+			var s corev1.Secret
+			err := k8sClient.Get(ctx, types.NamespacedName{Name: secretName, Namespace: operatorNamespace}, &s)
+			return err == nil
+		}
 
-			err := k8sClient.Get(ctx, objKey, srv)
+		It("keeps the instance credential by default and removes the checkpoint", func(ctx SpecContext) {
-			Expect(err).To(HaveOccurred(), "the TerdutServer itself should be gone once the finalizer clears")
+			credsName := runToReady(ctx, dsnSpec())
+			// A checkpoint left behind, as if the best-effort delete had failed.
+			Expect(k8sClient.Create(ctx, &corev1.Secret{
+				ObjectMeta: metav1.ObjectMeta{Name: checkpointSecretNameFor(name), Namespace: operatorNamespace},
+				Data:       map[string][]byte{credentialsSecretDataKey: []byte("admin-key-raw")},
+			})).To(Succeed())
 
-			var leftover corev1.Secret
+			deleteAndFinalize(ctx)
-			err = k8sClient.Get(ctx, types.NamespacedName{Name: credsName, Namespace: operatorNamespace}, &leftover)
+
-			Expect(err).To(HaveOccurred(), "the credentials Secret should have been cleaned up by the finalizer")
+			Expect(secretExists(ctx, credsName)).To(BeTrue(),
+				"the credential is kept so a recreated TerdutServer can adopt it")
+			Expect(secretExists(ctx, checkpointSecretNameFor(name))).To(BeFalse(),
+				"the checkpoint is a short-lived admin key and is always removed")
+		})
+
+		It("removes the credentials and checkpoint Secrets and the finalizer when the policy is Delete", func(ctx SpecContext) {
+			spec := dsnSpec()
+			spec.Credentials.DeletionPolicy = terdutv1alpha1.CredentialsDelete
+			credsName := runToReady(ctx, spec)
+
+			deleteAndFinalize(ctx)
+
+			Expect(secretExists(ctx, credsName)).To(BeFalse(),
+				"the credentials Secret should have been cleaned up by the finalizer")
+		})
+	})
+
+	Describe("recreating a TerdutServer against an already-bootstrapped server", func() {
+		// retainedSecret stands in for what the previous TerdutServer left.
+		retainedSecret := func(ctx SpecContext, token string) {
+			Expect(k8sClient.Create(ctx, &corev1.Secret{
+				ObjectMeta: metav1.ObjectMeta{Name: credentialsSecretNameFor(name), Namespace: operatorNamespace},
+				Data:       map[string][]byte{credentialsSecretDataKey: []byte(token)},
+			})).To(Succeed())
+		}
+		bringUp := func(ctx SpecContext, fakeSrv *httptest.Server) {
+			DeferCleanup(fakeSrv.Close)
+			reconciler.NewClient = func(string) *tdclient.Client { return tdclient.New(fakeSrv.URL) }
+			createServer(ctx, dsnSpec())
+			reconcileOnce(ctx) // finalizer
+			reconcileOnce(ctx) // Deployment/Service
+			markDeploymentReady(ctx)
+			reconcileOnce(ctx)
+		}
+		credsToken := func(ctx SpecContext) string {
+			var s corev1.Secret
+			Expect(k8sClient.Get(ctx, types.NamespacedName{Name: credentialsSecretNameFor(name), Namespace: operatorNamespace}, &s)).To(Succeed())
+			return string(s.Data[credentialsSecretDataKey])
+		}
+
+		It("adopts the retained credential when the server still accepts it, without bootstrapping", func(ctx SpecContext) {
+			fake, fakeSrv := newFakeTerdutServer()
+			fake.bootstrapped = true // every /api/bootstrap call 403s
+			fake.accounts[serviceAccountName] = 7
+			retainedSecret(ctx, "retained-key")
+
+			bringUp(ctx, fakeSrv)
+
+			Expect(readyCondition(ctx).Status).To(Equal(metav1.ConditionTrue))
+			srv := &terdutv1alpha1.TerdutServer{}
+			Expect(k8sClient.Get(ctx, objKey, srv)).To(Succeed())
+			Expect(srv.Status.CredentialsSecretRef).NotTo(BeNil())
+			Expect(srv.Status.CredentialsSecretRef.Name).To(Equal(credentialsSecretNameFor(name)))
+			Expect(credsToken(ctx)).To(Equal("retained-key"), "the retained key is reused, not replaced")
+		})
+
+		It("ignores a retained credential the server rejects and bootstraps afresh after a database reset", func(ctx SpecContext) {
+			fake, fakeSrv := newFakeTerdutServer() // a fresh database: bootstrap succeeds
+			fake.rejectedTokens = map[string]bool{"stale-key": true}
+			retainedSecret(ctx, "stale-key")
+
+			bringUp(ctx, fakeSrv)
+
+			Expect(readyCondition(ctx).Status).To(Equal(metav1.ConditionTrue))
+			Expect(credsToken(ctx)).NotTo(Equal("stale-key"), "the stale credential is replaced by a freshly minted one")
+		})
+
+		It("fails closed, naming the Secret to restore, when the server is bootstrapped and the retained credential is rejected", func(ctx SpecContext) {
+			fake, fakeSrv := newFakeTerdutServer()
+			fake.bootstrapped = true
+			fake.rejectedTokens = map[string]bool{"stale-key": true}
+			retainedSecret(ctx, "stale-key")
+
+			bringUp(ctx, fakeSrv)
+
+			cond := readyCondition(ctx)
+			Expect(cond.Status).To(Equal(metav1.ConditionFalse))
+			Expect(cond.Reason).To(Equal(terdutv1alpha1.ReasonBootstrapStateLost))
+			Expect(cond.Message).To(ContainSubstring(credentialsSecretNameFor(name)),
+				"the message names the Secret that would restore it")
+			Expect(cond.Message).To(ContainSubstring("does not clear the database"))
 		})
 	})