Keep the instance credential across a TerdutServer delete, and adopt it on recreate

Deleting a TerdutServer removed the credential Secrets but never touched the
database, so a recreated one found a server that was already bootstrapped and
no key for it: /api/bootstrap answered 403 and the operator stopped at
BootstrapStateLost, whose message and DESIGN.md both said "delete and
recreate". That is how the terdut-demo install on the cluster got stuck on
2026-10-03: Helm's cleanupOnFail deleted its TerdutServer after a failed
upgrade, the recreate found the bootstrapped database, and it sat at Ready:
False for five days until the database was reset by hand. Recreating cannot
fix it, because the finalizer clears Secrets and the database is not its to
reset, so "a fresh create starts clean" was only ever true when the database
went with it.

spec.credentials.deletionPolicy is Retain by default: the finalizer keeps the
instance credential Secret (Delete removes it, as before). The bootstrap
checkpoint is always removed. Before calling /api/bootstrap, reconcile now
looks for the retained Secret and asks the server for the operator's own
service account with its token. Accepted: adopt it and skip bootstrap.
Rejected with 401/403: the Secret outlived a database reset, so ignore it and
bootstrap like a first install, which replaces it. Any other error retries.
terdut-server's own tests already call that endpoint with an instance-scoped
key, so the permission is not new.

BootstrapStateLost is still the answer when the server is bootstrapped and no
credential it accepts survives, but its message now names the Secret to
restore and says that recreating does not clear the database. DESIGN.md §6
says the same, and the chart passes the setting through as
terdutServer.credentials.deletionPolicy.

A retained Secret of a TerdutServer that is gone for good is an orphan to
delete by hand. It is inert: nothing adopts it unless the server accepts the
token.

Checked on the kind demo with a locally built image against the real
terdut-server v0.43.0: deleting the TerdutServer kept the Secret, recreating it
reached Ready with the same credential (identical hash) and both TerdutTeams
came back Ready with their original ids. The controller specs cover adoption,
a rejected token after a reset, the bootstrapped-and-rejected failure, and
both deletion policies.

Co-authored-by: Claude <noreply@anthropic.com>

internal/controller/terdutserver_bootstrap.go

@@ -16,18 +16,24 @@ import (
 )
 
 // bootstrapStateLostError is DESIGN.md §6's one genuinely pathological
-// case: a checkpointed admin credential was used and then lost before the
-// lasting credential it was for could be persisted. Distinct from a plain
-// error so Reconcile can route it to a Ready: False condition (the
-// documented recovery is delete-and-recreate, not an automatic retry) rather
-// than treating it as a transient reconcile failure.
-type bootstrapStateLostError struct{ detail string }
+// case: the server's database is already bootstrapped, and no credential for
+// it survives here -- a checkpointed admin key was used and then lost before
+// the lasting credential could be persisted, or the instance credential
+// Secret was deleted (spec.credentials.deletionPolicy: Delete, or by hand).
+// Distinct from a plain error so Reconcile can route it to a Ready: False
+// condition rather than treating it as a transient reconcile failure: no
+// retry can fix it.
+type bootstrapStateLostError struct {
+	detail     string
+	secretName string // the instance credential Secret that would have fixed it
+}
 
 func (e *bootstrapStateLostError) Error() string {
 	return fmt.Sprintf(
-		"server reports already bootstrapped, but neither status.credentialsSecretRef nor a "+
-			"checkpointed admin credential exist here: %s. This TerdutServer cannot recover a "+
-			"credential on its own; delete and recreate it", e.detail)
+		"server reports already bootstrapped, but there is no credential for it here: %s. "+
+			"The operator cannot mint one on its own, and deleting and recreating this TerdutServer does "+
+			"not clear the database. Restore Secret %q in the operator's namespace if you have a copy, "+
+			"or reset the server's database and recreate this TerdutServer", e.detail, e.secretName)
 }
 
 // reconcileBootstrap implements DESIGN.md §6 point 1's self-registration
@@ -36,6 +42,17 @@ func (e *bootstrapStateLostError) Error() string {
 // srv.Status.CredentialsSecretRef is nil and the Deployment has a ready
 // replica.
 func (r *TerdutServerReconciler) reconcileBootstrap(ctx context.Context, srv *terdutv1alpha1.TerdutServer) error {
+	// A TerdutServer recreated against a database that is already
+	// bootstrapped: its instance credential may still be here (the default
+	// deletion policy keeps it), and then there is nothing to bootstrap.
+	adopted, err := r.adoptRetainedCredentials(ctx, srv)
+	if err != nil {
+		return err
+	}
+	if adopted {
+		return nil
+	}
+
 	adminKey, err := r.getOrCreateCheckpointedAdminKey(ctx, srv)
 	if err != nil {
 		return err
@@ -64,6 +81,51 @@ func (r *TerdutServerReconciler) reconcileBootstrap(ctx context.Context, srv *te
 	return nil
 }
 
+// adoptRetainedCredentials looks for the instance credential Secret an
+// earlier TerdutServer of the same name and namespace left behind
+// (spec.credentials.deletionPolicy: Retain), and adopts it if the server
+// still accepts its token. Reports whether it did.
+//
+// The token is tried before it is trusted: a Secret that outlived a database
+// reset holds a key the server has never heard of, and adopting that would
+// make every later call 401. A rejected token (401/403) is not an error here;
+// it just means there is nothing to adopt, and bootstrap proceeds as it would
+// for a first install -- which succeeds against a freshly reset database and
+// replaces the Secret. Anything else (the server unreachable, a 5xx) is
+// returned for a retry rather than guessed at.
+func (r *TerdutServerReconciler) adoptRetainedCredentials(ctx context.Context, srv *terdutv1alpha1.TerdutServer) (bool, error) {
+	name := credentialsSecretName(srv)
+	var sec corev1.Secret
+	if err := r.Get(ctx, client.ObjectKey{Namespace: r.OperatorNamespace, Name: name}, &sec); err != nil {
+		if apierrors.IsNotFound(err) {
+			return false, nil
+		}
+		return false, err
+	}
+	token := string(sec.Data[credentialsSecretDataKey])
+	if token == "" {
+		return false, nil
+	}
+
+	// The operator's own service account is the one thing this credential
+	// is for, so asking the server for it by name checks the token and that
+	// it belongs to that account in the same call.
+	sa, err := r.NewClient(serviceURL(srv)).WithToken(token).GetServiceAccountByName(ctx, serviceAccountName)
+	if err != nil {
+		if statusErr, ok := errors.AsType[*tdclient.StatusError](err); ok &&
+			(statusErr.Code == http.StatusUnauthorized || statusErr.Code == http.StatusForbidden) {
+			return false, nil
+		}
+		return false, fmt.Errorf("checking the retained credential in Secret %q: %w", name, err)
+	}
+	if sa == nil {
+		return false, nil
+	}
+
+	srv.Status.CredentialsSecretRef = &terdutv1alpha1.SecretKeyRef{Name: name, Key: credentialsSecretDataKey}
+	return true, nil
+}
+
 // getOrCreateCheckpointedAdminKey returns a usable admin key: from the
 // checkpoint Secret if an earlier, interrupted attempt already got one, or
 // freshly from /api/bootstrap, immediately checkpointed before it's used
@@ -89,7 +151,10 @@ func (r *TerdutServerReconciler) getOrCreateCheckpointedAdminKey(ctx context.Con
 			// status.credentialsSecretRef) means a prior reconcile already
 			// won this exact race and its checkpoint was lost afterward --
 			// the one case §6 doesn't try to paper over.
-			return "", &bootstrapStateLostError{detail: "/api/bootstrap returned 403"}
+			return "", &bootstrapStateLostError{
+				detail:     "/api/bootstrap returned 403",
+				secretName: credentialsSecretName(srv),
+			}
 		}
 		return "", fmt.Errorf("POST /api/bootstrap: %w", err)
 	}