Keep the instance credential across a TerdutServer delete, and adopt it on recreate

Deleting a TerdutServer removed the credential Secrets but never touched the
database, so a recreated one found a server that was already bootstrapped and
no key for it: /api/bootstrap answered 403 and the operator stopped at
BootstrapStateLost, whose message and DESIGN.md both said "delete and
recreate". That is how the terdut-demo install on the cluster got stuck on
2026-10-03: Helm's cleanupOnFail deleted its TerdutServer after a failed
upgrade, the recreate found the bootstrapped database, and it sat at Ready:
False for five days until the database was reset by hand. Recreating cannot
fix it, because the finalizer clears Secrets and the database is not its to
reset, so "a fresh create starts clean" was only ever true when the database
went with it.

spec.credentials.deletionPolicy is Retain by default: the finalizer keeps the
instance credential Secret (Delete removes it, as before). The bootstrap
checkpoint is always removed. Before calling /api/bootstrap, reconcile now
looks for the retained Secret and asks the server for the operator's own
service account with its token. Accepted: adopt it and skip bootstrap.
Rejected with 401/403: the Secret outlived a database reset, so ignore it and
bootstrap like a first install, which replaces it. Any other error retries.
terdut-server's own tests already call that endpoint with an instance-scoped
key, so the permission is not new.

BootstrapStateLost is still the answer when the server is bootstrapped and no
credential it accepts survives, but its message now names the Secret to
restore and says that recreating does not clear the database. DESIGN.md §6
says the same, and the chart passes the setting through as
terdutServer.credentials.deletionPolicy.

A retained Secret of a TerdutServer that is gone for good is an orphan to
delete by hand. It is inert: nothing adopts it unless the server accepts the
token.

Checked on the kind demo with a locally built image against the real
terdut-server v0.43.0: deleting the TerdutServer kept the Secret, recreating it
reached Ready with the same credential (identical hash) and both TerdutTeams
came back Ready with their original ids. The controller specs cover adoption,
a rejected token after a reset, the bootstrapped-and-rejected failure, and
both deletion policies.

Co-authored-by: Claude <noreply@anthropic.com>

api/v1alpha1/terdutserver_types.go

@@ -23,6 +23,37 @@ type SecretKeyRef struct {
 	Key string `json:"key"`
 }
 
+// CredentialsDeletionPolicy is what deleting a TerdutServer does to the
+// instance credential Secret the operator generated for it.
+// +kubebuilder:validation:Enum=Retain;Delete
+type CredentialsDeletionPolicy string
+
+const (
+	// CredentialsRetain keeps the Secret when the TerdutServer is deleted, so
+	// a TerdutServer recreated with the same name and namespace against the
+	// same database adopts it again instead of finding a server it cannot
+	// log in to. Deleting a TerdutServer never touches its database, so the
+	// operator's service account is still there to be reused. The default.
+	CredentialsRetain CredentialsDeletionPolicy = "Retain"
+	// CredentialsDelete removes the Secret with the TerdutServer. Choose it
+	// when the database goes too, or when the credential must not outlive the
+	// object.
+	CredentialsDelete CredentialsDeletionPolicy = "Delete"
+)
+
+// CredentialsSpec configures the lifecycle of the generated instance
+// credential.
+type CredentialsSpec struct {
+	// deletionPolicy: whether the instance credential Secret is kept
+	// (Retain, the default) or removed (Delete) when this TerdutServer is
+	// deleted. A kept Secret is only ever adopted after the server accepts
+	// its token, so one left over from a database that has since been reset
+	// is ignored and replaced.
+	// +kubebuilder:default=Retain
+	// +optional
+	DeletionPolicy CredentialsDeletionPolicy `json:"deletionPolicy,omitempty"`
+}
+
 // ImageSpec is the terdut-server image to run.
 type ImageSpec struct {
 	// +kubebuilder:validation:MinLength=1
@@ -324,6 +355,11 @@ type TerdutServerSpec struct {
 	// +optional
 	Deadman DeadmanSpec `json:"deadman,omitempty"`
 
+	// credentials: what happens to the instance credential this operator
+	// generates for the server.
+	// +optional
+	Credentials CredentialsSpec `json:"credentials,omitempty"`
+
 	// +optional
 	Notify NotifySpec `json:"notify,omitempty"`
 
@@ -382,9 +418,12 @@ const (
 	// ReasonBootstrapStateLost: a checkpointed admin credential
 	// (DESIGN.md §6) was lost after being used but before the lasting
 	// credential it was for could be persisted -- the one genuinely
-	// pathological case in the self-registration flow. Fail-closed, same
-	// recovery as DESIGN.md §5's webhook-Secret-loss rule: delete and
-	// recreate this TerdutServer.
+	// pathological case in the self-registration flow -- or the server's
+	// database is already bootstrapped and no credential for it survives
+	// (spec.credentials.deletionPolicy: Delete, or the Secret removed by
+	// hand). Fail-closed: the operator cannot mint a credential, and
+	// deleting and recreating the TerdutServer does not clear the database.
+	// Restore the Secret, or reset the server's database.
 	ReasonBootstrapStateLost = "BootstrapStateLost"
 	// ReasonAdopted: the happy path. A working credential is in hand, the
 	// Deployment has a ready replica, and the database (if postgresClusterRef)

api/v1alpha1/zz_generated.deepcopy.go

@@ -47,6 +47,21 @@ func (in *AllowedTeamsNamespaces) DeepCopy() *AllowedTeamsNamespaces {
 	return out
 }
 
+// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil.
+func (in *CredentialsSpec) DeepCopyInto(out *CredentialsSpec) {
+	*out = *in
+}
+
+// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new CredentialsSpec.
+func (in *CredentialsSpec) DeepCopy() *CredentialsSpec {
+	if in == nil {
+		return nil
+	}
+	out := new(CredentialsSpec)
+	in.DeepCopyInto(out)
+	return out
+}
+
 // DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil.
 func (in *DatabaseSpec) DeepCopyInto(out *DatabaseSpec) {
 	*out = *in
@@ -764,6 +779,7 @@ func (in *TerdutServerSpec) DeepCopyInto(out *TerdutServerSpec) {
 	in.Database.DeepCopyInto(&out.Database)
 	out.Sweeper = in.Sweeper
 	out.Deadman = in.Deadman
+	out.Credentials = in.Credentials
 	in.Notify.DeepCopyInto(&out.Notify)
 	in.OIDC.DeepCopyInto(&out.OIDC)
 	in.AllowedTeams.DeepCopyInto(&out.AllowedTeams)