# terdut-operator design This is the design reference for implementing terdut-operator. It exists so implementation can start from settled decisions instead of re-litigating them mid-PR. It is deliberately more detailed than the README; the README stays as the short pitch and now points here instead of carrying open questions. Written against terdut-server as of the Postgres-only, per-team-resources version (teams, escalation policies, dead man's switches and integrations are all rows scoped to a team, managed over `internal/api/*` — not env/config-file driven; see that repo's `charts/terdut-server` for the current deploy story this operator supersedes). ## 1. Goals & non-goals **Goal:** let a terdut-server install — the server itself, its teams, their escalation policies, dead man's switches and alert-source integrations — be fully described as Kubernetes objects and managed through gitops, following controller-runtime / Kubebuilder conventions. **The operator creates and owns every `TerdutServer` it manages. It never adopts a pre-existing, independently-deployed terdut-server** — whether deployed by hand or by `charts/terdut-server`. There is no migration path from an existing chart-based install, and none is planned (§10): starting with the operator means applying a fresh `TerdutServer` CR, not converting one. This is the root a few things downstream hang off of — notably §6's bootstrap flow, which only has to handle the operator bootstrapping a server it just created, never a server something else already bootstrapped first. **Non-goals (v1):** - Not a general-purpose Postgres operator. It *consumes* a database that either the Zalando `postgres-operator` or something else already provides. - Not managing Alertmanager itself, or the routing rules that decide which alerts reach which integration webhook — only the terdut-server side (creating the integration and handing back its URL/key). - Not OLM packaging. Plain Kubebuilder manifests + Helm chart for install, matching how terdut-server itself ships. - Cross-namespace references are limited to exactly one edge: `TerdutTeam.spec.serverRef` may name a `TerdutServer` in a different namespace, gated by that `TerdutServer`'s own `spec.allowedTeams` consent field (§4.1, §4.2) — this is the multi-tenant shape the operator exists for (one platform team owns a `TerdutServer`; other teams self-service a `TerdutTeam` against it without needing write access to the server's namespace). Every other reference (`teamRef` on the escalation rule/dead-man-switch/alert-source CRDs) stays same-namespace-as-its-`TerdutTeam` only, in v1 — those manage a specific team's own resources and are expected to live alongside it. - No validating/mutating admission webhooks in v1. CEL validation rules on the CRDs (OpenAPI `x-kubernetes-validations`) cover what they can; anything that needs a live look at another object (e.g. "does this teamRef exist") is a status condition, not an admission rejection — keeps v1 to a controller-only deployment with no cert-manager/webhook dependency. ## 2. The README's open questions, resolved > How do team-crd connect with server-crd? Explicit `spec.serverRef: {name, namespace}` on `TerdutTeam` — same reasoning as before (explicit, greppable, trivially validated), but **`namespace` is deliberately part of the reference**: one team can run and own a `TerdutServer`, and other teams — in their own namespaces, without any write access to the server-owning team's namespace — self-service a `TerdutTeam` against it. `namespace` defaults to the `TerdutTeam`'s own namespace when omitted, so the common single-tenant case (`serverRef: {name: terdut}`) is unchanged. This cross-namespace edge needs the target namespace's explicit consent — otherwise any namespace in the cluster could point a `TerdutTeam` at someone else's `TerdutServer` and have the operator provision a team on its behalf, which is a namespace-boundary violation, not a gitops convenience. (This consent gate is about which `TerdutTeam`s the operator will act on, not about credential exposure — no terdut-server credential is ever placed in a `TerdutTeam`'s own namespace regardless of this setting; see §6.) Kubernetes has two established patterns for this kind of consent, and Gateway API itself uses both, for two different relationships: - **`ReferenceGrant`** (used for a Route reaching into an arbitrary Service/Secret): a separate object, living in the *target* namespace, enumerating exact `{fromNamespace, fromKind} → {toKind, toName}` pairs. No wildcard, no selector — every permitted namespace is spelled out. - **An inline field on the parent** (used for a `ListenerSet` attaching to a shared `Gateway`, GA since Gateway API v1.5): the parent carries `spec.allowedListeners.namespaces: {from: None|Same|All|Selector, selector}` directly, no separate CRD. `TerdutTeam` attaching to a shared `TerdutServer` is structurally the second case, not the first — a bounded set of expected children attaching to a parent they were deliberately made shareable, not an arbitrary backend reference — so this design follows the `ListenerSet` precedent: `TerdutServer.spec.allowedTeams` (§4.1), no extra CRD. §4.2 covers how `TerdutTeam` resolves against it. No other reference in this design (`teamRef` on the child CRDs) crosses a namespace boundary, so this is the only place cross-namespace consent is needed at all (§1, §5, §6, §9). > How does escalationrules, switches and alertsources connect to a team? Explicit `spec.teamRef: {name}` on each of `TerdutEscalationRule`, `TerdutDeadmanSwitch`, `TerdutAlertSource` — same reasoning, and it mirrors terdut-server's own data model, where every one of these rows carries a `team_id` foreign key already. A matcher/selector on `TerdutTeam` would be inventing a second source of truth for an association the server already models as a plain reference. > Support for both postgres-operator (Zalando) and bring-your-own, how do we > design that to be user friendly? `TerdutServer.spec.database` is a oneOf, mirroring the chart's existing `database.dsn` / `database.passwordSecret` contract (see §8): - `dsn` + `passwordSecretRef` — bring-your-own, exactly today's chart inputs. - `postgresClusterRef` — points at a Zalando `postgresql.acid.zalan.do` CR; the operator derives the DSN and resolves the generated credentials Secret itself (see §8). CloudNativePG support is a natural follow-up using the same shape and is called out as deferred (§13), not designed in detail now. ## 3. API group, versions, CRD catalog - Group: `terdut.ryuvia.com`, version: `v1alpha1` (matches the `ryuvia.com` domain terdut-server already uses; bump to `v1beta1`/`v1` per the normal Kubernetes API graduation criteria once the shapes below have proven stable against a real install). - Module: `git.ryuvia.com/niklas/terdut-operator`, scaffolded with Kubebuilder (controller-runtime), matching terdut-server's Go toolchain and house style. | Kind | Scope | Purpose | |---|---|---| | `TerdutServer` | Namespaced | One terdut-server install: Deployment, Service, database wiring, bootstrap, operator credentials, cross-namespace team consent. | | `TerdutTeam` | Namespaced | One team on a `TerdutServer`, possibly in another namespace: name, OIDC group mapping. | | `TerdutEscalationRule` | Namespaced | A team's escalation policy (levels, targets, repeat). | | `TerdutDeadmanSwitch` | Namespaced | One dead man's switch on a team. | | `TerdutAlertSource` | Namespaced | One alert-ingest integration on a team (currently: Alertmanager webhook). | ## 4. Per-CRD spec ### 4.1 `TerdutServer` ```yaml apiVersion: terdut.ryuvia.com/v1alpha1 kind: TerdutServer metadata: name: terdut namespace: oncall spec: image: repository: git.ryuvia.com/niklas/terdut-server tag: v0.9.3 replicas: 1 # terdut-server is not horizontally-scale-tested; keep the field, default 1 networking: hostname: terdut.example.com servicePort: 8080 gatewayListener: "" # same semantics as chart's networking.listener database: dsn: "postgres://terdut@terdut-postgres:5432/terdut?sslmode=require" # mutually exclusive with postgresClusterRef passwordSecretRef: {name: terdut.terdut-postgres.credentials.postgresql.acid.zalan.do, key: password} # --- OR --- postgresClusterRef: {name: terdut-postgres} # Zalando `postgresql` CR in the same namespace sweeper: staleAfter: 6h archiveAfter: 168h deadman: matchers: "alertname=Watchdog" timeout: 15m severity: critical notify: ntfyURL: "http://ntfy.ntfy.svc.cluster.local" fallbackTopic: "" repeatEvery: 15m tokenSecretRef: {name: "", key: token} oidc: enabled: false issuer: "" clientID: "" clientSecretRef: {name: "", key: client-secret} name: SSO scopes: "openid profile email" allowedGroups: [] adminGroup: "" sessionMaxAge: 12h passwordLogin: true # Consent for TerdutTeams in OTHER namespaces to set serverRef at this # TerdutServer. Same-namespace TerdutTeams never need this. Modeled on # Gateway API's Gateway.spec.allowedListeners.namespaces (the ListenerSet # attachment pattern, not ReferenceGrant — see §2 for why). allowedTeams: namespaces: from: None # None (default) | Same | All | Selector # selector: # required, and only meaningful, when from: Selector # matchLabels: # terdut.ryuvia.com/allowed: "true" status: conditions: [...] # Ready, DatabaseReady, Bootstrapped observedGeneration: 3 serviceName: terdut credentialsSecretRef: {name: terdut.platform-oncall-instance-credentials, key: token} # see §6; pure output -- generated by the controller's own self-registration flow, always in the OPERATOR's namespace (always, implicitly -- not stored here, since it's never anything else), under a fixed key ("token"). ``` Field-for-field this is the chart's `values.yaml` reshaped as a spec — not for migrating an existing chart-based install (§1: there is no such path), just because the shape is already familiar from the chart, and the operator absorbs what the chart's Deployment/Service/bootstrap-job templates used to do. `spec.database` fields are `+kubebuilder:validation:XValidation` guarded to be mutually exclusive (`dsn` xor `postgresClusterRef`); mirrors the chart's "chart provisions no database" stance — this operator provisions no database either, only wires up one that exists. `spec.allowedTeams.namespaces.from` defaults to `None`, matching `allowedListeners`'s own default — a fresh `TerdutServer` accepts no cross-namespace `TerdutTeam` until its owner opts in, same-namespace `TerdutTeam`s are unaffected either way. `Selector` deliberately has no per-name allowlist (no "and only these teams") — namespace-level consent is the right granularity here, same reasoning as `ListenerSet`: the namespace is the tenancy boundary, not the object. ### 4.2 `TerdutTeam` ```yaml spec: serverRef: name: terdut namespace: platform-oncall # optional; defaults to this TerdutTeam's own namespace. # Cross-namespace requires that TerdutServer's spec.allowedTeams # (§4.1) to admit this namespace — otherwise Ready: False, reason: RefNotPermitted. displayName: "Platform" # -> POST /api/teams {"name": ...}; server assigns the ID oidc: memberGroup: "terdut-platform-members" ownerGroup: "terdut-platform-owners" status: conditions: [...] teamID: 42 # the server-side ID; needed by every child object's controller credentialsSecretRef: {name: platform-oncall.platform-team-credentials, key: token} # see §6; this team's own scoped key, always in the OPERATOR's own namespace (not stored here — same reasoning as TerdutServer's, §4.1) serverEndpoint: "http://terdut.platform-oncall.svc:8080" # resolved once, here, from spec.serverRef -- see §5: this is what actually makes "a child never needs to chain up to TerdutServer" true, not just a stated intent. Set alongside teamID/credentialsSecretRef, same reconcile. observedGeneration: 1 ``` Team *membership* (which users belong, `team_members`) is explicitly **not** modeled as a CRD field in v1: terdut-server already manages membership via OIDC group sync at login for SSO installs, and manual membership for password-login installs is a people-management action, not infrastructure — forcing it through gitops would mean a human's team change goes through a PR review. Flagged in §13 as revisitable if a real gitops-membership need shows up. ### 4.3 `TerdutEscalationRule` ```yaml spec: teamRef: {name: platform-team} repeatCount: 2 fallbackTopic: "platform-oncall" levels: - timeout: 5m targets: - kind: oncall # "oncall" or "user" - kind: user username: alice # resolved to a user ID by the controller at apply time - timeout: 15m targets: - kind: oncall status: conditions: [...] observedGeneration: 1 ``` One `TerdutEscalationRule` per team — the server itself models a policy as one row (`escalation_policies`) with an owned list of levels, so a one-CRD-to-one-policy mapping (not one-CRD-per-level) matches the server's own aggregate and lets the whole thing be reconciled with the single `PUT /api/teams/{teamID}/escalation` the API actually exposes (see §5). Not enforced at admission (no webhooks in v1, §1) — two `TerdutEscalationRule`s naming the same team would both `PUT` it and clobber each other every reconcile; a footgun worth this one sentence, not a technical guard. A `username` target is resolved to the `user_id` `PUT /api/teams/{teamID}/escalation` actually requires (confirmed against source: `escalationTargetJSON.UserID *int64`, no username field at all) via `GET /api/users` — confirmed open to any authenticated caller, not gated by team membership or admin (`internal/api/router.go`'s own comment: "readable by anyone signed in"), so the team-scoped credential this controller already holds is enough; no extra RBAC-equivalent server-side needed. Re-resolved every reconcile rather than cached, in case a username is renamed. An unresolvable username is `Ready: False, reason: UnknownUser`, naming which one. ### 4.4 `TerdutDeadmanSwitch` ```yaml spec: teamRef: {name: platform-team} name: "prod-watchdog" matcher: "alertname=Watchdog,cluster=prod" timeout: 15m severity: critical status: conditions: [...] switchID: 7 ``` No unique-name constraint server-side (§5's table row, confirmed against source) — the controller's own idempotent-create step is a `GET`-list and name match, not a conflict to recover from. `name` is optional, same as the API: left empty, terdut-server derives it from `matcher`'s own canonical form, and that's what the lookup matches against too. ### 4.5 `TerdutAlertSource` ```yaml spec: teamRef: {name: platform-team} kind: alertmanager name: "prod-alertmanager" status: conditions: [...] integrationID: 3 webhookURLSecretRef: {name: prod-alertmanager-terdut-webhook} # url + key, never in status/spec ``` The integration key is shown by the API exactly once, at creation (`Integration.Key`/`URL` in terdut-server's own model) — never re-readable, same shape as the bootstrap admin key. The controller writes it straight into a generated, owner-referenced Secret on the create it caused and never logs or stores it anywhere else; the CR's `status` carries only the Secret reference, matching how e.g. cert-manager's `Certificate` exposes `spec.secretName` rather than the key material itself. ### 4.6 Cross-namespace consent: `TerdutServer.spec.allowedTeams` No separate CRD — the consent lives on `TerdutServer` itself (§4.1), following Gateway API's `Gateway.spec.allowedListeners` (`ListenerSet` attachment) rather than its `ReferenceGrant`, since a `TerdutTeam` attaching to a shared `TerdutServer` is the same shape of relationship: a bounded set of expected children attaching to a parent explicitly designed to be shared, not an arbitrary cross-namespace backend reference (see §2 for the full comparison of both patterns). - `from: None` (default) — no cross-namespace `TerdutTeam` may resolve a `serverRef` into this `TerdutServer`. Same-namespace `TerdutTeam`s are always allowed regardless of this field. - `from: Same` — equivalent to `None` in effect (same-namespace is already unrestricted) but kept for parity with the upstream enum and to make the policy self-documenting in a diff. - `from: All` — any namespace in the cluster may reference in. Appropriate for a genuinely shared, cluster-wide `TerdutServer`; the audit trail is "check `allowedTeams` plus who has RBAC to create a `TerdutTeam` anywhere," which is materially weaker than `Selector`. - `from: Selector` — only namespaces matching `spec.allowedTeams.namespaces.selector` (a standard `metav1.LabelSelector` over `Namespace` objects, exactly like `allowedListeners`'s own `selector`) may reference in. This is the recommended mode for the platform-team-owns-a-shared-server scenario this design targets: label the consuming namespaces once (e.g. `terdut.ryuvia.com/allowed-server: platform-oncall/terdut`) and new namespaces opt in by carrying the label, without editing the `TerdutServer` again. - A `TerdutTeam`'s controller re-evaluates `allowedTeams` on every reconcile before it will resolve a cross-namespace `serverRef` — for `Selector`, this means a `Get` on its own `Namespace` object plus reading the target `TerdutServer`'s spec, not a List across the cluster. Same-namespace `serverRef` never consults this field at all. - Narrowing or clearing `allowedTeams` (or unlabeling a namespace, under `Selector`) is a live revocation: the next reconcile of any `TerdutTeam` it used to authorize finds itself no longer permitted, flips `Ready: False, reason: RefNotPermitted`, and — deliberately — does **not** delete the team server-side on revocation alone; it stops reconciling further changes until access is restored or the `TerdutTeam` CR itself is deleted (whose finalizer still needs the credentials Secret described in §6 to clean up, so blocking *new* changes rather than forcing an immediate, possibly credential-less deletion is the safer failure mode). ## 5. Reconciliation semantics terdut-server's REST surface (`internal/api/router.go`) does not give every resource a full update verb, so reconciliation strategy is per-resource: | Resource | Verbs available | Strategy | |---|---|---| | Team | POST create, PUT rename, DELETE, PUT oidc-groups, GET by name | Real update-in-place: diff spec vs. last-applied, PUT the changed pieces. `GET /api/teams?name=` (terdut-server's `TEAM-LOOKUP.md`, landed 2026-10-01) is what makes the idempotent-create general rule below actually true for Team — confirmed by checking: until that endpoint existed, an instance-scoped service account had no way to recover a team's id after a 409, unlike every other resource in this table, where the adopt-on-conflict rule had a real lookup to call. | | Escalation policy | GET/PUT whole-policy | Update-in-place: PUT the full desired policy every reconcile that finds drift; cheap because whole-policy is small and already loaded whole server-side. | | Dead man's switch | POST create, **PUT update-in-place** (added in terdut-server v0.33.0), DELETE, GET-list | Real update-in-place, same shape as Team/Escalation: PUT the whole switch every reconcile once its id is known. No unique-name constraint server-side (confirmed against source — `handleCreateTeamDeadman` has no conflict handling at all, unlike Team/service-account creation), so idempotent-create here can't rely on a 409 to adopt from: before POSTing, `GET /api/teams/{teamID}/deadman/switches` and match by `name` first: found → adopt its id; not found → POST. `handleUpdateTeamDeadman`'s own doc comment (terdut-server) confirms the motivation directly: "Added alongside create/delete so an automated caller (terdut-operator) can reconcile a spec change without deleting and recreating the switch, which would otherwise ... needlessly rotate its id for no reason a reconciler's diff should ever manufacture." An earlier draft of this row, written before that endpoint existed, described delete-and-recreate; corrected here, confirmed against source rather than left stale. | | Integration (alert source) | POST create, PATCH rename, DELETE | Rename via PATCH; any other spec change (kind) is delete-and-recreate, which **rotates the webhook key** — called out loudly in the CRD's field docs and in a `Warning` event, since it breaks whatever sends to the old URL/key until the new Secret is picked up. (See the general rule below for what happens if the Secret is lost with *no* spec change.) | General rules for every controller: - **Idempotent create**: before POSTing, check `status.` is unset; if the server already has a same-named object from a previous partial reconcile (e.g. after a crash between POST and status-write), treat a 409/name-conflict as "adopt" — GET-by-name and populate status, rather than erroring forever. terdut-server's list endpoints in each of these areas return objects by name, so this is a straightforward correlation. - **Periodic resync** in addition to watch-triggered reconciles (Kubebuilder default `RequeueAfter` on success, e.g. every 5–10 minutes) to catch drift from **someone changing state directly against the server's API/UI**, since gitops correctness means the CR wins, not "first write wins". - **Finalizers** on every CRD that has a server-side counterpart, so deletion calls the corresponding DELETE before the Kubernetes object disappears. Failure to delete server-side (e.g. server unreachable) blocks finalizer removal and surfaces as a `Degraded` condition + event, rather than silently orphaning a row. - **Generated Secrets holding unrecoverable server-issued material are watched, and their loss is fail-closed, not self-healed.** Currently this is just `TerdutAlertSource`'s webhook Secret (§4.5): the controller adds it to its `Owns()` watches, not just the CR. If it disappears while `status.integrationID` is still set, the controller does **not** attempt to recreate it — the key is genuinely gone (§4.5: never stored anywhere but that one Secret), so silently minting a replacement would rotate a live production webhook URL with no corresponding spec change to explain why. Instead it flips `Ready: False, reason: WebhookSecretLost` and fires a `Warning` event telling the operator to delete and recreate the `TerdutAlertSource`. No new mechanism is needed for recovery: deleting the CR runs the existing finalizer (DELETE the still-live integration server-side, above), and recreating it runs the existing idempotent-create path (this same section) — a fresh POST, a new key, a new Secret. This is deliberately the same recovery motion as the kind-change rotation above, just human-triggered instead of spec-triggered. - **Owner chain for status resolution, not API calls**: `TerdutTeam`'s controller does not call any other controller; every child CRD's controller independently resolves its own `teamRef` → `TerdutTeam.status` for the `teamID`, `credentialsSecretRef`, and `serverEndpoint` it needs to call the API (§6) — it never needs to chain further up to `TerdutServer` at all, not just as a stated intent but literally: `serverEndpoint` is resolved once, by `TerdutTeam`'s own controller, and stored in its status specifically so no child ever needs its own `TerdutServer` RBAC (`get`/ `list`/`watch` on `terdutservers`) to find out where to send a request — the team's own scoped credential plus that one status field is everything a child resource's controller requires. If the referenced `TerdutTeam` isn't `Ready` yet (which includes not having a `credentialsSecretRef` set), the child requeues with backoff and reports `Ready: False, reason: WaitingForTeam` — no cross-controller RPC. - **Cross-namespace `serverRef` is re-checked every reconcile, not just at creation**: `TerdutTeam`'s controller reads the target `TerdutServer`'s `spec.allowedTeams` (and, under `Selector`, a `Get` on its own `Namespace` object for labels) on every pass before touching a cross-namespace `TerdutServer` — revocation (§4.6) takes effect on the team's very next reconcile, not just when the CR is first applied. ## 6. Bootstrap & authentication to terdut-server's API terdut-server's scoped service-account credential type (`terdut-server`'s `SERVICE-ACCOUNTS.md`) has shipped — confirmed against source: `internal/api/service_accounts.go`, migration `014_service_accounts.sql`, and `internal/api/router.go` wiring it in under `AuthMiddleware`. This section is no longer blocked on it; the "v1-blocking" framing here was accurate when this section was first written and is stale now. **`GET /api/service-accounts?name=` is not an unauthenticated lookup — confirmed against `internal/api/middleware.go`'s `AuthMiddleware`, which hard-rejects any request carrying neither a Bearer token nor a session cookie with `401` before any handler ever runs.** This would matter a great deal if the operator's `/api/bootstrap` call could ever lose a race to something else bootstrapping the same server first — a credential-less loser would have no authenticated way to recover. It doesn't matter here, by construction (§1): **the operator only ever calls `/api/bootstrap` against a `TerdutServer` it just created**, so there is nothing else in a position to race it. An earlier draft of this section added a `spec.credentialsSecretRef` bring-your-own input specifically to work around that race, for a world where the operator might adopt a server something else had already bootstrapped. That world doesn't exist (§1), so the field was removed rather than kept as unused flexibility — self-registration (point 1, below) is simply the only path, not one of two. **Every credential the operator holds — the one instance-scoped key per `TerdutServer`, and one team-scoped key per `TerdutTeam` — lives in a Secret in the *operator's own* namespace, never in the namespace of the CR it authenticates for.** Reconciliation happens entirely inside the operator's controller loop, which is a single Deployment/ServiceAccount already watching every namespace it's granted (§9); nothing about calling terdut-server's API on a CR's behalf requires the credential to be physically located near that CR, and no CR owner (human or otherwise) ever needs to see, hold, or have RBAC to read a terdut-server credential. This is a straight simplification of an earlier draft of this section, which mirrored a shared credential into each consenting namespace instead — that version conflated "the CR's owner never needs to see this" (true, and preserved here) with "so the credential must live in the CR's namespace" (a non-sequitur once you don't need to grant *anyone else* namespace-local read access). Dropping that assumption also removes an entire class of complexity: no on-demand mirroring, no garbage-collecting an orphaned copy when `allowedTeams` narrows, no "OwnerReferences can't cross namespaces so track it in status instead" workaround — none of that machinery is needed when nothing ever crosses into a tenant namespace in the first place. 1. **First reconcile, confirmed against source** (`internal/api/users.go`'s `handleBootstrap`): `/api/bootstrap` is single-shot *per install*, gated on `SELECT COUNT(*) FROM users` — once non-zero, every call `403`s regardless of identity. There are two crash windows between "get an admin key" and "have a lasting, usable credential" — getting from `/api/bootstrap`'s key to a minted service-account key, and getting from that key to a persisted Secret — and both get a checkpoint rather than being left as a theoretical gap, the same rigor §5's general idempotent-create rule already applies elsewhere: - `status.credentialsSecretRef` already set: done, nothing to do. - Otherwise, check for an intermediate `.-bootstrap-admin` Secret in the operator's own namespace first. If it exists, its key is a still- valid admin credential from an earlier, interrupted attempt — skip `/api/bootstrap` entirely and reuse it. If not, call `/api/bootstrap` once the Deployment this `TerdutServer` created has a ready replica; on `201`, immediately checkpoint its response's raw admin key (`{"user": ..., "api_key": {"key": "", ...}}`) into that Secret before doing anything else with it. A `403` with neither `status.credentialsSecretRef` nor this checkpoint Secret present is the one genuinely pathological case left (the checkpoint deleted out from under a reconcile already past this point) — handled the same way the design already handles unrecoverable server-issued material elsewhere (§5's webhook-Secret-loss rule): fail closed, `Ready: False, reason: BootstrapStateLost`, with the same recovery as that case, delete and recreate the `TerdutServer` (its finalizer tears down the Deployment/database-backing and server-side rows; a fresh create starts clean) — not a workaround peculiar to this one path. - With an admin key in hand (fresh or checkpointed): `POST /api/service-accounts {name: "terdut-operator", scope: "instance"}`. A `409` here means a prior attempt got this far before being interrupted — adopt rather than error, per §5's general rule: `GET /api/service-accounts?name=terdut-operator` (authenticated with the checkpointed admin key, not an unauthenticated lookup) to find its id, then `POST /api/service-accounts/{id}/keys` to mint a fresh key — an orphaned first key some interrupted attempt minted and never used is inert, not a cleanup obligation. 2. The resulting instance-scoped key — not the checkpointed admin key, which is deleted once this step succeeds — is written to a generated Secret in the **operator's own namespace** (e.g. `.-instance-credentials`, under a fixed data key, `token`), referenced back from `TerdutServer.status.credentialsSecretRef: {name, key}` (§4.1). No `OwnerReference` (those can't cross namespaces, and this Secret doesn't share a namespace with the `TerdutServer` that caused it); the `TerdutServer`'s finalizer deletes this Secret directly as part of its own teardown, the same way it already has to clean up the server-side resources it created (§5's general finalizer rule extends naturally to this Secret). 3. When a `TerdutTeam` first becomes `Ready` (its `serverRef` resolved, `allowedTeams` satisfied if cross-namespace), its controller uses the `TerdutServer`'s instance-scoped credential (read from the operator's own namespace, resolved via the owner chain in §5) to mint a **team-scoped** service account for itself: `POST /api/service-accounts` with `scope: team, teamID: `. The resulting key is written to its own generated Secret, again in the **operator's own namespace** (e.g. `.-team-credentials`), referenced from `TerdutTeam.status.credentialsSecretRef` (§4.2). Same finalizer pattern as point 2: the `TerdutTeam`'s finalizer deletes this Secret as part of its own teardown. 4. Every child controller (`TerdutEscalationRule`, `TerdutDeadmanSwitch`, `TerdutAlertSource`) reads its team's `credentialsSecretRef` — resolved through its `teamRef` → `TerdutTeam.status` (§5) — and never touches the instance-scoped credential at all. Since child CRDs stay same-namespace- as-their-`TerdutTeam` in v1 (§1), and the credential itself lives in the operator's namespace regardless of where the `TerdutTeam` or its children are, this works identically whether the `TerdutTeam` is same-namespace or cross-namespace relative to its `TerdutServer` — there is no separate cross-namespace case to handle here at all, unlike the mirroring design this replaced. 5. **Blast radius**: a team-scoped key can only touch its own `team_id`'s escalation policy, dead-man switches, integrations, schedule and OIDC group bindings server-side (enforced by terdut-server itself, per `SERVICE-ACCOUNTS.md`) — compromising one such Secret (e.g. a bug that leaks operator-namespace Secrets, or an overly broad RBAC grant on that one namespace) exposes exactly one team, never the whole server. This is the real fix for what an earlier draft of this section called out as its weak point (every mirrored copy being server-admin-equivalent); it falls out of team-scoped credentials existing at all, independent of where they're stored — the operator-private storage described above closes the RBAC-footprint half of the problem, team scoping closes the credential- privilege half. 6. **Rotation**: `POST /api/service-accounts/{id}/keys` mints a new key on the existing account without recreating it; the old key is revoked via `DELETE /api/service-accounts/{id}/keys/{keyID}`; the operator's local Secret is updated in place. No DB-level workaround, no re-triggering a single-shot endpoint that can't fire twice (which is what made rotation unworkable under the old `/api/bootstrap`-only design). ## 7. Ownership, status, garbage collection - Every generated object that lives in the *same* namespace as the CR that caused it (Deployment, Service, webhook Secret) carries a standard `metav1.OwnerReference` — GC handles these, no finalizer needed. The two credential Secrets from §6 are the one exception: they live in the operator's own namespace regardless of where their owning CR lives, so `OwnerReference` doesn't apply (cross-namespace) and cleanup instead runs through that CR's finalizer directly, alongside the server-side DELETE it already has to issue (§5). - Status conditions follow the standard `metav1.Condition` shape with at least `Ready` on every kind, plus kind-specific ones (`TerdutServer`: `DatabaseReady`, `Bootstrapped`; children: `Synced`). - `status.observedGeneration` on every kind, bumped only after a successful reconcile against that generation's spec — the standard way a client (or `kubectl wait`) tells "applied" from "seen". - No cluster-scoped aggregation object (e.g. no cluster-wide "all servers" status) in v1 — `kubectl get terdutservers -A` is the aggregate view. ## 8. Postgres integration Mirrors the chart's existing two-path contract (`values.yaml` `database.dsn`/`passwordSecret`), because that contract is already documented and tested operationally: - **Bring-your-own**: `spec.database.dsn` (no password) + `spec.database.passwordSecretRef` — the operator sets `PGPASSWORD` on the Deployment's container env from that Secret, exactly like the chart does today, and does nothing else. No connectivity check beyond what the Deployment's own readiness probe already gives. - **Zalando `postgres-operator`**: `spec.database.postgresClusterRef` names a `postgresql.acid.zalan.do` CR in the same namespace. The controller: - Reads that CR's status for the primary Service name/port to build the DSN host — `..svc:5432` — and database name convention. - Resolves the generated credentials Secret (`..credentials.postgresql.acid.zalan.do`) the same way the chart's comment already documents, and wires it in as `PGPASSWORD` the same way. - Watches that Secret (not just the `postgresql` CR) so a credential rotation triggers a requeue — the chart today requires a manual pod restart for this; the operator can at least detect and report it via a condition even if restarting on rotation is left as a §13 follow-up rather than done automatically (a rolling restart on credential change is a behavior change worth its own design pass, not folded in here). - Requires read RBAC on `postgresql.acid.zalan.do` (optional CRD — the operator's ClusterRole/Role should not hard-fail if the CRD isn't installed and a given `TerdutServer` uses BYO DSN instead). ## 9. RBAC - The operator's own ServiceAccount needs, per namespace it's granted: `get/list/watch/create/update/patch/delete` on `Deployments`, `Services` it owns, and `get/list/watch` on `postgresql.acid.zalan.do` (optional, degrade gracefully if absent per §8), plus cluster-wide `get/list` on `Namespace` (labels only, for `allowedTeams: {from: Selector}` evaluation — §4.1, §4.6). - **Two different `Secret` scopes, not one — corrected from an earlier draft of this section.** That earlier draft said `Secret` access was "scoped to the operator's own namespace only... nowhere else," reasoning that with every credential held privately in the operator's own namespace (§6) there was no legitimate reason to touch a `Secret` anywhere else. That was wrong once §4.5 existed: - The §6 credential Secrets (one instance-scoped key per `TerdutServer`, one team-scoped key per `TerdutTeam`) do live in, and are only ever touched from, the operator's own namespace — `get/list/watch/create/update/patch/delete` there, nowhere else. The rest of the original reasoning stands for *these* Secrets specifically: no human or team's own RBAC is ever granted access to a terdut-server credential by this design, and the operator itself never needs cross-namespace access to reach them. - The §4.5 webhook Secret is different: it's owned by and lives beside its `TerdutAlertSource`, in that CR's own tenant namespace, not the operator's. The per-namespace `Role` already granted for `Deployments`/`Services` in each watched namespace (below) must carry the same `Secret` verbs there too, or the controller cannot create, watch, or even detect the loss of (§5) that Secret at all. - This necessarily widens the operator's footprint in each watched tenant namespace to "any `Secret` in that namespace," not just the ones it created — Kubernetes RBAC has no owner-scoped grant finer than the namespace itself, and the design already accepts this same granularity for Deployments/Services there. Flagged as an accepted trade-off, not a silent gap (§13). - No cluster-scoped resources are created by this operator (namespaced CRDs only, per §1) — a `Role` + `RoleBinding` per watched (tenant) namespace is sufficient for Deployments/Services/the webhook `Secret`/the optional Zalando CRD, plus a separate `Role` + `RoleBinding` in the operator's own namespace for the §6 credential Secrets; a `ClusterRole` is only needed for watching CRDs across all namespaces (the normal Kubebuilder multi-tenant-operator default) — none of the `Secret` access above needs to be cluster-scoped. - terdut-server's own RBAC is unaffected — the operator talks to it purely over HTTP with service-account API keys (§6), never via the Kubernetes API for app-level state. ## 10. Relationship to `charts/terdut-server` The chart's Deployment/Service/bootstrap-job templates are redundant once `TerdutServer` exists — running both would mean two controllers (Helm and this operator) reconciling the same Deployment, which is exactly the conflict Kubernetes operators exist to avoid. The chart is repurposed into an **installer chart**: it installs the operator + CRDs (and optionally one `TerdutServer` CR from `values.yaml`, for users who want "helm install and get a server" without hand-writing a CR) rather than templating the Deployment directly. **No migration path from an existing chart-based install, and none is planned (§1).** An earlier draft of this section spent most of its length on one — `helm template` the current release's `values.yaml` into an equivalent `TerdutServer` CR, uninstall or shrink the old release, and a whole sub-question about who gets to call `/api/bootstrap` first, the chart's Job or the operator — all of which presupposed the operator might end up managing a server the chart had already deployed and bootstrapped. §1 rules that out: the operator only ever manages servers it created itself, so there's nothing to migrate and no bootstrap race to settle (§6 covers why that race doesn't exist either). Adopting the operator means applying a fresh `TerdutServer` CR; whatever the chart deployed before stays exactly what it was, a separate install, until someone deletes it. ## 11. Testing strategy - `envtest` (controller-runtime's fake API server) for every controller's reconcile logic against the Kubernetes side. - terdut-server's REST API is faked with a small `httptest.Server` per controller test driven by fixtures matching the real handlers' request/response shapes (already well-documented in `internal/api/*_test.go` on the server side) — no real Postgres or real terdut-server binary needed for controller unit tests. - A smaller number of true end-to-end tests (`kind` cluster + real terdut-server image + real Postgres) covering the golden path per CRD: create `TerdutServer` → `TerdutTeam` → one of each child kind → verify via terdut-server's own API that the objects exist with the right shape → delete the CR → verify the server-side object is gone. ## 12. Observability - Standard controller-runtime metrics (reconcile duration/error counts) are enough for v1 — no custom metrics. - Every externally-visible action (bootstrap, key rotation-needed, delete- and-recreate on the no-PUT resources, adopt-on-conflict) emits a Kubernetes `Event` on the CR, since that's what shows up in `kubectl describe` and gitops tooling (Argo CD/Flux) surfaces without extra wiring. ## 13. Deferred / explicitly out of scope for this design - **A real scoped service-account/token type in terdut-server — not merely deferred, this is v1-blocking for §6 as written** (verified: without it, §6's bootstrap flow has no working credential-rotation path and no clean answer to the chart-vs-operator bootstrap race; see §6 points 1, 5, 6 and §10). Sequence this server-side change *before* implementing the `TerdutServer` controller's bootstrap logic, not after. - **A version-discovery endpoint on terdut-server** (e.g. `GET /api/version`). Neither this operator nor terdut-tui has one today — both independently detect capability by probing specific routes (terdut-tui via `GET /api/teams` 404-checking; this operator would otherwise need to invent its own equivalent probe). An unattended reconciler is more exposed to a silent breaking API change than an interactive TUI a human is watching; raising this alongside the service-account request rather than inventing another route-probe here. - CloudNativePG support — same `spec.database` shape as Zalando should extend to it, but the concrete field/Secret-naming conventions need their own look. - Cross-namespace `teamRef` on the child CRDs (`TerdutEscalationRule`, `TerdutDeadmanSwitch`, `TerdutAlertSource`) — only `TerdutTeam.serverRef` crosses namespaces in v1 (§2, §4.2, §4.6); these stay same-namespace as their `TerdutTeam` until a real need for splitting them out shows up. - Narrower-than-namespace RBAC for the §4.5 webhook Secret (Kubernetes RBAC has no owner-scoped grant below the namespace itself, per §9) — revisit if the widened per-tenant-namespace `Secret` access proves too broad in practice. - Gitops-managed team *membership* (see §4.2). - Automatic Deployment restart on upstream Postgres credential rotation. - Admission webhooks / CEL-only validation limits (e.g. verifying a `teamRef` exists at admission time rather than surfacing it as a status condition after the fact). - OLM packaging.