# terdut-operator design The design of terdut-operator as built. terdut-server is the system of record; this operator makes one install of it — the server, its teams, their escalation ladders, dead man's switches and alert-source integrations — describable as Kubernetes objects and manageable through gitops. ## 1. Goals & non-goals **Goal:** a terdut-server install fully described as Kubernetes objects, following controller-runtime / Kubebuilder conventions. **The operator creates and owns every `TerdutServer` it manages. It never adopts a pre-existing, independently-deployed terdut-server**, whether deployed by hand or by `charts/terdut-server`. There is no migration path from a chart-based install (§10): starting with the operator means applying a fresh `TerdutServer`. **Non-goals (v1):** - Not a Postgres operator. It consumes a database that the Zalando `postgres-operator` or something else provides (§8). - Not managing Alertmanager or its routing, only the terdut-server side (creating the integration and handing back its URL/key). - Not OLM packaging: plain Kubebuilder manifests and a Helm chart, like terdut-server. - Cross-namespace references are limited to one edge: `TerdutTeam.spec.serverRef` may name a `TerdutServer` in another namespace, gated by that server's `spec.allowedTeams` (§4.6). A `TerdutAlertSource` lives beside its `TerdutTeam`. - No admission webhooks. CEL validation covers what it can; anything needing a live look at another object is a status condition, not an admission rejection. - **Never manages external exposure for a `TerdutServer`** (Ingress, HTTPRoute, VirtualService) — a permanent non-goal. The operator creates a plain `ClusterIP` Service; see `examples/networking`. ## 2. Credentials and CRD shape (the 2026-10 redesign) What the design is, and why — each point replaced something heavier: - **No bootstrap handshake.** The operator generates a key into a Secret named `-operator-key`, in the TerdutServer's own namespace and owned by it (a pod can only mount Secrets of its own namespace, and an owner reference replaces the old finalizer and `credentials.deletionPolicy`). The Deployment hands it to terdut-server as `TERDUT_OPERATOR_KEY`; the server creates or re-keys its instance-scoped service account `terdut-operator` from it at every start. A replaced Secret rolls the pods (key-hash annotation). `/api/bootstrap` stays free for the first human administrator. Gone with it: the checkpoint Secret, `BootstrapStateLost`, `DatabaseReady`/`Bootstrapped` conditions. - **One credential per server.** An instance-scoped account acts as owner of every team's configuration (not a member, so it reads no incidents). The per-team service accounts, Secrets and `status.credentialsSecretRef`/`serverEndpoint` on `TerdutTeam` are gone. - **Team identity is `external_id`.** The operator creates a team with `external_id: /` of its CR; the server returns the existing team for a known id (200) instead of creating one, so crash recovery, a lost status and a deleted team all heal by repeating the same call, and a display name that belongs to another team is a 409 (`TeamNameTaken`) instead of an adoption. - **Three CRDs.** `TerdutServer`, `TerdutTeam` and `TerdutAlertSource`. `TerdutEscalationRule` and `TerdutDeadmanSwitch` are `spec.escalation` and `spec.deadmanSwitches[]` (matched by name, unique per team server-side; ones not listed are removed) on the team: they were one-to-one children with the team's lifecycle, and folding them removes `teamRef`, the two-rules-clobber footgun and three controllers. `TerdutAlertSource` stays separate because it owns a webhook Secret in its own namespace. - **Usernames are resolved by the server** (`PUT .../escalation` accepts `username`), so an unknown user is a `UnknownUser` condition, not a list-and-match. - **Invites are removed.** Membership is not modelled; people get in through the server's own signup/OIDC. - **Env mirrors the chart's where it matters:** OIDC claim names and `trustEmail` are spec fields; `TERDUT_DEADMAN_*` no longer exist on the server. ## 3. CRD catalog Group `terdut.ryuvia.com`, version `v1alpha1`; module `git.ryuvia.com/niklas/terdut-operator`. | Kind | Purpose | |---|---| | `TerdutServer` | One terdut-server install: Deployment, Service, database wiring, operator key, cross-namespace team consent. | | `TerdutTeam` | One team on a server, possibly in another namespace: name, OIDC groups, escalation ladder, dead man's switches. | | `TerdutAlertSource` | One alert-ingest integration on a team; owns a webhook Secret. | ## 4. Per-CRD spec (The escalation and dead man's switch specs that used to be §4.3 and §4.4 are part of §4.2; the section numbers are kept because code comments cite them.) ### 4.1 `TerdutServer` ```yaml apiVersion: terdut.ryuvia.com/v1alpha1 kind: TerdutServer metadata: name: terdut namespace: oncall spec: image: repository: git.ryuvia.com/niklas/terdut-server tag: v0.9.3 replicas: 2 # default since terdut-server v0.36.0's advisory locks; see TerdutServerSpec.Replicas networking: hostname: terdut.example.com servicePort: 8080 database: dsn: "postgres://terdut@terdut-postgres:5432/terdut?sslmode=require" # mutually exclusive with postgresClusterRef passwordSecretRef: {name: terdut.terdut-postgres.credentials.postgresql.acid.zalan.do, key: password} # --- OR --- postgresClusterRef: {name: terdut-postgres} # Zalando `postgresql` CR in the same namespace sweeper: staleAfter: 6h archiveAfter: 168h notify: ntfyURL: "http://ntfy.ntfy.svc.cluster.local" fallbackTopic: "" repeatEvery: 15m tokenSecretRef: {name: "", key: token} oidc: enabled: false issuer: "" clientID: "" clientSecretRef: {name: "", key: client-secret} name: SSO scopes: "openid profile email" allowedGroups: [] adminGroup: "" usernameClaim: preferred_username # also emailClaim, groupsClaim trustEmail: false sessionMaxAge: 12h passwordLogin: true # Consent for TerdutTeams in OTHER namespaces to set serverRef at this # TerdutServer. Same-namespace TerdutTeams never need this. Modeled on # Gateway API's Gateway.spec.allowedListeners.namespaces (the ListenerSet # attachment pattern, not ReferenceGrant — see §2 for why). allowedTeams: namespaces: from: None # None (default) | Same | All | Selector # selector: # required, and only meaningful, when from: Selector # matchLabels: # terdut.ryuvia.com/allowed: "true" # Pod-level customization of the Deployment -- all optional, direct corev1 # passthrough throughout (see PodSpec's own doc comment). A representative # subset: pod: resources: requests: {cpu: 100m, memory: 128Mi} limits: {memory: 256Mi} tolerations: - key: dedicated operator: Equal value: terdut effect: NoSchedule disruptionBudget: minAvailable: 1 # mutually exclusive with maxUnavailable status: conditions: [...] # Ready observedGeneration: 3 credentialsSecretRef: {name: terdut-operator-key, key: token} # see §6; pure output, in this TerdutServer's own namespace ``` Field-for-field this is the chart's `values.yaml` reshaped as a spec — not for migrating an existing chart-based install (§1: there is no such path), just because the shape is already familiar from the chart, and the operator absorbs what the chart's Deployment/Service/bootstrap-job templates used to do. `spec.database` fields are `+kubebuilder:validation:XValidation` guarded to be mutually exclusive (`dsn` xor `postgresClusterRef`); mirrors the chart's "chart provisions no database" stance — this operator provisions no database either, only wires up one that exists. `spec.allowedTeams.namespaces.from` defaults to `None`, matching `allowedListeners`'s own default — a fresh `TerdutServer` accepts no cross-namespace `TerdutTeam` until its owner opts in, same-namespace `TerdutTeam`s are unaffected either way. `Selector` deliberately has no per-name allowlist (no "and only these teams") — namespace-level consent is the right granularity here, same reasoning as `ListenerSet`: the namespace is the tenancy boundary, not the object. `spec.pod` is pod-level customization of the Deployment, all optional and directly reusing corev1 types wherever corev1 already models the knob exactly (`tolerations`, `affinity`, `topologySpreadConstraints`, `resources`, `securityContext`/`containerSecurityContext`, `extraEnv`/ `extraEnvFrom`, `extraVolumes`/`extraVolumeMounts`, `imagePullSecrets`) — no custom wrapper buys anything for any of these, matching how CloudNativePG and the Zalando postgres-operator both expose the same knobs. `affinity` is pure user-supplied passthrough, not a toggle-plus-generated-default the way a multi-replica-aware operator's pod anti-affinity typically is: even though `replicas` now defaults to 2 (terdut-server v0.36.0's advisory locks made that safe, §4.1's own illustrative YAML comment), this operator still never auto-generates affinity of its own. `spec.pod.disruptionBudget` is the one field here that isn't a straight PodTemplateSpec knob — when set, the controller reconciles a `PodDisruptionBudget` selecting this `TerdutServer`'s pods; clearing it deletes any it previously created (§7). `minAvailable`/`maxUnavailable` are mutually exclusive, `+kubebuilder:validation:XValidation`-guarded the same way as `spec.database`'s own `dsn`/`postgresClusterRef` rule. ### 4.2 `TerdutTeam` ```yaml spec: serverRef: {name: terdut, namespace: platform-oncall} # namespace optional; cross-namespace needs allowedTeams displayName: Platform oidc: {memberGroup: terdut-platform-members, ownerGroup: terdut-platform-owners} escalation: repeatCount: 2 fallbackTopic: platform-oncall levels: - timeout: 5m targets: [{kind: oncall}, {kind: user, username: alice}] - timeout: 15m targets: [{kind: oncall}] deadmanSwitches: - {name: watchdog, matcher: "alertname=Watchdog,cluster=prod", timeout: 15m, severity: critical} status: conditions: [...] # Ready; reasons: ServerRefNotFound, RefNotPermitted, WaitingForServer, # TeamNameTaken, UnknownUser, InvalidSpec, Adopted teamID: 42 observedGeneration: 1 ``` The team is created under the identity `/` of its CR (`external_id`), so a retry, a lost status or a team deleted behind the operator's back all heal by repeating the same call; a `displayName` that belongs to a different team is `TeamNameTaken`, never an adoption. Every reconcile applies, in order: create-or-find, rename, OIDC groups, the escalation ladder (one PUT of the whole ladder; absent `spec.escalation` clears it), and the switches. Switches are matched by name (unique per team on the server): missing ones are created, changed ones updated in place, and any not listed are deleted, since in operator mode nobody else can add one. Durations are validated by a CRD pattern and re-checked (`InvalidSpec`). A username the server does not know is `UnknownUser` until that person exists. Team *membership* is not modelled: the server manages it through OIDC group sync and its own UI. ### 4.5 `TerdutAlertSource` ```yaml spec: teamRef: {name: platform-team} kind: alertmanager name: "prod-alertmanager" status: conditions: [...] integrationID: 3 webhookURLSecretRef: {name: prod-alertmanager-terdut-webhook} # url + key, never in status/spec ``` The integration key is shown by the API exactly once, at creation (`Integration.Key`/`URL` in terdut-server's own model) — never re-readable, a one-shot value. The controller writes it straight into a generated, owner-referenced Secret on the create it caused and never logs or stores it anywhere else; the CR's `status` carries only the Secret reference, matching how e.g. cert-manager's `Certificate` exposes `spec.secretName` rather than the key material itself. ### 4.6 Cross-namespace consent: `TerdutServer.spec.allowedTeams` No separate CRD — the consent lives on `TerdutServer` itself (§4.1), following Gateway API's `Gateway.spec.allowedListeners` (`ListenerSet` attachment) rather than its `ReferenceGrant`, since a `TerdutTeam` attaching to a shared `TerdutServer` is the same shape of relationship: a bounded set of expected children attaching to a parent explicitly designed to be shared, not an arbitrary cross-namespace backend reference (see §2 for the full comparison of both patterns). - `from: None` (default) — no cross-namespace `TerdutTeam` may resolve a `serverRef` into this `TerdutServer`. Same-namespace `TerdutTeam`s are always allowed regardless of this field. - `from: Same` — equivalent to `None` in effect (same-namespace is already unrestricted); kept for parity with the upstream enum. - `from: All` — any namespace in the cluster may reference in. Appropriate for a genuinely shared, cluster-wide `TerdutServer`; the audit trail is "check `allowedTeams` plus who has RBAC to create a `TerdutTeam` anywhere," which is materially weaker than `Selector`. - `from: Selector` — only namespaces matching `spec.allowedTeams.namespaces.selector` (a standard `metav1.LabelSelector` over `Namespace` objects, exactly like `allowedListeners`'s own `selector`) may reference in. This is the recommended mode for the platform-team-owns-a-shared-server scenario this design targets: label the consuming namespaces once (e.g. `terdut.ryuvia.com/allowed-server: platform-oncall/terdut`) and new namespaces opt in by carrying the label, without editing the `TerdutServer` again. - A `TerdutTeam`'s controller re-evaluates `allowedTeams` on every reconcile before it will resolve a cross-namespace `serverRef` — for `Selector`, this means a `Get` on its own `Namespace` object plus reading the target `TerdutServer`'s spec, not a List across the cluster. Same-namespace `serverRef` never consults this field at all. - Narrowing or clearing `allowedTeams` (or unlabeling a namespace, under `Selector`) is a live revocation: the next reconcile of any `TerdutTeam` it used to authorize finds itself no longer permitted, flips `Ready: False, reason: RefNotPermitted`, and — deliberately — does **not** delete the team server-side on revocation alone; it stops reconciling further changes until access is restored. Deleting the `TerdutTeam` CR still deletes the team: consent gates what the operator starts acting on, not whether it may clean up after itself. ## 5. Reconciliation semantics | Resource | Server verbs | Strategy | |---|---|---| | Team | POST (idempotent on `external_id`), PUT rename, PUT oidc-groups, DELETE | Repeat the idempotent POST, then PUT the rest, every reconcile. Delete runs in a finalizer; the server refuses (409) while the team has open incidents, which is retried. | | Escalation ladder | PUT whole ladder | PUT the full desired ladder every reconcile; the server resolves usernames. | | Dead man's switch | list, POST, PUT, DELETE; names unique per team | Diff by name against the list. | | Integration (alert source) | POST, PATCH rename, DELETE | Rename via PATCH; a kind change is delete-and-recreate, which **rotates the webhook key** (Warning event). An integration deleted on the server is recreated with a new key (Warning event). | General rules: - **Periodic resync** (5 minutes) besides watch-triggered reconciles, to catch someone changing state directly against the server: the CR wins. - **Finalizers** on `TerdutTeam` and `TerdutAlertSource` call the server's DELETE first. A 404 is success; any other failure blocks removal and surfaces as an event rather than orphaning a row. `TerdutServer` needs none: its Secret is owned and its database is never touched. - **A webhook Secret that is lost fails closed.** The key is never re-readable from the server, so the controller does not mint a replacement for a live URL with no spec change to explain it: `Ready: False, reason: WebhookSecretLost`; delete and recreate the `TerdutAlertSource`. - **Children resolve through the team.** A `TerdutAlertSource` finds its `TerdutTeam` (same namespace), requires it Ready, then reads the server named by the team's `serverRef` and that server's operator key. A `TerdutTeam` re-reconciles when its `TerdutServer` changes. - **Cross-namespace consent is re-checked every reconcile** (§4.6), so revocation takes effect on the team's next pass. ## 6. Authentication to terdut-server's API The operator talks to the server over HTTP with one bearer key per `TerdutServer`: 1. The `TerdutServer` controller creates the Secret `-operator-key` (data key `token`, `tdsa_` + 48 hex characters) in the server's own namespace, owned by the `TerdutServer`. It is created once and never overwritten while it exists: the running server was seeded with it. 2. The Deployment hands it to the pods as `TERDUT_OPERATOR_KEY` through a `secretKeyRef` (the key never appears in the pod spec), plus an annotation holding a hash of it so a replaced Secret rolls the pods. 3. At every start the server creates or re-keys its instance-scoped service account `terdut-operator` from that value. An instance-scoped account acts as owner of every team's configuration but is not a member of any team, so it reads no incidents and is never an administrator. 4. `status.credentialsSecretRef` points at the Secret; `TerdutTeam` and `TerdutAlertSource` controllers read it from the server's namespace. There is no bootstrap handshake: `/api/bootstrap` stays free for the first human administrator. Deleting the `TerdutServer` deletes the key with it; a recreated one gets a new key and the server re-seeds on its next start. Rotating by hand means deleting the Secret: the next reconcile makes a new one and rolls the pods. ## 7. Ownership, status, garbage collection - Everything the controllers generate in a CR's namespace carries an `OwnerReference` (Deployment, Service, PodDisruptionBudget, the operator key Secret, the webhook Secret), so GC cleans up and no finalizer is needed. The PDB exists only while `spec.pod.disruptionBudget` is set; the controller deletes it itself when the field is cleared. - Every kind has a `Ready` condition and `status.observedGeneration`. - No cluster-scoped aggregate object: `kubectl get terdutservers -A` is the overview. ## 8. Postgres integration Mirrors the chart's existing two-path contract (`values.yaml` `database.dsn`/`passwordSecret`), because that contract is already documented and tested operationally: - **Bring-your-own**: `spec.database.dsn` (no password) + `spec.database.passwordSecretRef` — the operator sets `PGPASSWORD` on the Deployment's container env from that Secret, exactly like the chart does today, and does nothing else. No connectivity check beyond what the Deployment's own readiness probe already gives. - **Zalando `postgres-operator`**: `spec.database.postgresClusterRef` names a `postgresql.acid.zalan.do` CR in the same namespace. The controller: - Reads that CR's status for the primary Service name/port to build the DSN host — `..svc:5432` — and database name convention. - Resolves the generated credentials Secret (`..credentials.postgresql.acid.zalan.do`) the same way the chart's comment already documents, and wires it in as `PGPASSWORD` the same way. - Does not watch that Secret: a rotated credential is noticed at the next 5-minute resync, and restarting pods on rotation is a §13 follow-up. - Requires read RBAC on `postgresql.acid.zalan.do` (optional CRD — the operator's ClusterRole/Role should not hard-fail if the CRD isn't installed and a given `TerdutServer` uses BYO DSN instead). ## 9. RBAC - The operator needs `get/list/watch/create/update/patch/delete` on `Deployments`, `Services`, `PodDisruptionBudgets` and `Secrets` it owns, `get/list/watch` on `postgresql.acid.zalan.do` (optional; degraded gracefully when the CRD is absent), and `get/list/watch` on `Namespaces` (labels only, for `allowedTeams: {from: Selector}`). - Secrets live in the `TerdutServer`'s namespace (the operator key) and in each `TerdutAlertSource`'s namespace (the webhook Secret), so the operator needs Secret access in every tenant namespace. Kubernetes RBAC has no owner-scoped grant finer than the namespace. - **As built (2026-10): broader than the above.** The shipped default is a `ClusterRole` with full verbs on `Secrets` in every namespace (the chart's `rbac.namespaced: true` gives a `Role` in the release namespace only, which cannot serve tenant namespaces), and the manager's cache is not restricted to watched namespaces. A per-namespace `Role` split (a Role and RoleBinding per watched namespace, with the cache restricted to them) is the intended end state, not implemented, and the decision (2026-10) is to stay cluster-wide for now: treat this operator as able to read every Secret in the cluster. - terdut-server's own RBAC is unaffected: the operator uses only its HTTP API (§6), never the Kubernetes API for app-level state. ## 10. Relationship to `charts/terdut-server` The server chart's Deployment, Service and bootstrap Job are redundant once `TerdutServer` exists: running both would have two controllers reconciling the same Deployment. The operator's chart (`charts/terdut-operator`) installs the operator, CRDs and RBAC, and optionally one `TerdutServer` from `values.yaml` (`terdutServer.enabled`). There is no migration from a chart-based install and none is planned (§1); whatever the server chart deployed stays a separate install until someone deletes it. The operator's Deployment builder mirrors the chart's env block for the knobs both expose (the chart's `deployment.yaml` and `buildEnv` in `internal/controller/terdutserver_deployment.go`); a new server setting is added in `config.go`, the chart, and the operator, in that order. ## 11. Testing strategy - `envtest` (controller-runtime's fake API server) for every controller's reconcile logic against the Kubernetes side. - terdut-server's REST API is faked with a small `httptest.Server` per controller test driven by fixtures matching the real handlers' request/response shapes (already well-documented in `internal/api/*_test.go` on the server side) — no real Postgres or real terdut-server binary needed for controller unit tests. - The golden path (`kind` cluster + real terdut-server image + real Postgres: create `TerdutServer` → `TerdutTeam` → `TerdutAlertSource`, verify through terdut-server's own API, delete, verify it is gone) is a manual pass via `examples/demo/run-demo.sh`, not a CI job. ## 12. Observability - Standard controller-runtime metrics (reconcile duration/error counts) are enough for v1 — no custom metrics. - Every externally-visible action (a team created or renamed, an integration recreated, a delete that failed) emits a Kubernetes `Event` on the CR, since that's what shows up in `kubectl describe` and gitops tooling (Argo CD/Flux) surfaces without extra wiring. ## 13. Deferred / out of scope - CloudNativePG support, alongside the Zalando `postgresClusterRef` (same `spec.database` shape). - Gitops-managed team membership (§4.2). - Per-namespace RBAC with a restricted cache (§9). - Automatic Deployment restart on upstream Postgres credential rotation. - `spec.pod.priorityClassName`, pod labels beyond annotations, and an HPA for `TerdutServer`. - Admission webhooks beyond CEL (for example, checking a `teamRef` exists at admission time). - A shared API types module or generated client for terdut-server, terdut-operator and terdut-tui. - OLM packaging.