Keep the instance credential across a TerdutServer delete, and adopt it on recreate

Deleting a TerdutServer removed the credential Secrets but never touched the
database, so a recreated one found a server that was already bootstrapped and
no key for it: /api/bootstrap answered 403 and the operator stopped at
BootstrapStateLost, whose message and DESIGN.md both said "delete and
recreate". That is how the terdut-demo install on the cluster got stuck on
2026-10-03: Helm's cleanupOnFail deleted its TerdutServer after a failed
upgrade, the recreate found the bootstrapped database, and it sat at Ready:
False for five days until the database was reset by hand. Recreating cannot
fix it, because the finalizer clears Secrets and the database is not its to
reset, so "a fresh create starts clean" was only ever true when the database
went with it.

spec.credentials.deletionPolicy is Retain by default: the finalizer keeps the
instance credential Secret (Delete removes it, as before). The bootstrap
checkpoint is always removed. Before calling /api/bootstrap, reconcile now
looks for the retained Secret and asks the server for the operator's own
service account with its token. Accepted: adopt it and skip bootstrap.
Rejected with 401/403: the Secret outlived a database reset, so ignore it and
bootstrap like a first install, which replaces it. Any other error retries.
terdut-server's own tests already call that endpoint with an instance-scoped
key, so the permission is not new.

BootstrapStateLost is still the answer when the server is bootstrapped and no
credential it accepts survives, but its message now names the Secret to
restore and says that recreating does not clear the database. DESIGN.md §6
says the same, and the chart passes the setting through as
terdutServer.credentials.deletionPolicy.

A retained Secret of a TerdutServer that is gone for good is an orphan to
delete by hand. It is inert: nothing adopts it unless the server accepts the
token.

Checked on the kind demo with a locally built image against the real
terdut-server v0.43.0: deleting the TerdutServer kept the Secret, recreating it
reached Ready with the same credential (identical hash) and both TerdutTeams
came back Ready with their original ids. The controller specs cover adoption,
a rejected token after a reset, the bootstrapped-and-rejected failure, and
both deletion policies.

Co-authored-by: Claude <noreply@anthropic.com>

internal/controller/terdutserver_controller.go

@@ -39,10 +39,10 @@ const resyncInterval = 5 * time.Minute
 // than "someone edited something out of band."
 const waitInterval = 15 * time.Second
 
-// finalizerName cleans up the credentials Secret(s) this controller
-// generates in the operator's own namespace on delete — the Deployment and
-// Service are owned (OwnerReference, DESIGN.md §7) and need no finalizer of
-// their own.
+// finalizerName cleans up the Secret(s) this controller generates in the
+// operator's own namespace on delete (which of them, spec.credentials.
+// deletionPolicy decides) — the Deployment and Service are owned
+// (OwnerReference, DESIGN.md §7) and need no finalizer of their own.
 const finalizerName = "terdut.ryuvia.com/terdutserver"
 
 // serviceAccountName is the name the operator registers itself under
@@ -215,18 +215,31 @@ func (r *TerdutServerReconciler) setNotReady(
 	return ctrl.Result{RequeueAfter: d}, nil
 }
 
-// reconcileDelete cleans up the credentials Secret(s) this controller
-// generated in the operator's own namespace. The Deployment and Service are
-// owned (OwnerReference, DESIGN.md §7) and need no attention here — normal
-// GC handles them. There is no server-side "delete this install" call to
+// reconcileDelete cleans up the Secrets this controller generated in the
+// operator's own namespace. The Deployment and Service are owned
+// (OwnerReference, DESIGN.md §7) and need no attention here — normal GC
+// handles them. There is no server-side "delete this install" call to
 // make: bootstrap created a user and a service account, and terdut-server's
 // API has no way to delete either (only to revoke individual keys), so
-// there is nothing meaningful to undo there either.
+// there is nothing meaningful to undo there either, and the database is
+// never touched.
+//
+// That is why the instance credential is kept by default
+// (spec.credentials.deletionPolicy: Retain). Everything it logs in to
+// outlives the TerdutServer, so a recreated one finds a bootstrapped server
+// it has no key for -- unless the key is still here to be adopted (see
+// adoptRetainedCredentials). The bootstrap checkpoint is always removed: it
+// is a short-lived admin key, and the instance credential is all that is
+// needed afterwards.
 func (r *TerdutServerReconciler) reconcileDelete(ctx context.Context, srv *terdutv1alpha1.TerdutServer) (ctrl.Result, error) {
 	if !controllerutil.ContainsFinalizer(srv, finalizerName) {
 		return ctrl.Result{}, nil
 	}
-	for _, name := range []string{checkpointSecretName(srv), credentialsSecretName(srv)} {
+	names := []string{checkpointSecretName(srv)}
+	if srv.Spec.Credentials.DeletionPolicy == terdutv1alpha1.CredentialsDelete {
+		names = append(names, credentialsSecretName(srv))
+	}
+	for _, name := range names {
 		sec := &corev1.Secret{ObjectMeta: metav1.ObjectMeta{Name: name, Namespace: r.OperatorNamespace}}
 		if err := r.Delete(ctx, sec); err != nil && !apierrors.IsNotFound(err) {
 			return ctrl.Result{}, err