terdut-operator/DESIGN.md

# terdut-operator design

The design of terdut-operator as built. terdut-server is the system of record; this operator
makes one install of it — the server, its teams, their escalation ladders, dead man's switches
and alert-source integrations — describable as Kubernetes objects and manageable through gitops.

## 1. Goals & non-goals

**Goal:** a terdut-server install fully described as Kubernetes objects, following
controller-runtime / Kubebuilder conventions.

**The operator creates and owns every `TerdutServer` it manages. It never adopts a pre-existing,
independently-deployed terdut-server**, whether deployed by hand or by `charts/terdut-server`.
There is no migration path from a chart-based install (§10): starting with the operator means
applying a fresh `TerdutServer`.

**Non-goals (v1):**
- Not a Postgres operator. It consumes a database that the Zalando `postgres-operator` or
  something else provides (§8).
- Not managing Alertmanager or its routing, only the terdut-server side (creating the
  integration and handing back its URL/key).
- Not OLM packaging: plain Kubebuilder manifests and a Helm chart, like terdut-server.
- Cross-namespace references are limited to one edge: `TerdutTeam.spec.serverRef` may name a
  `TerdutServer` in another namespace, gated by that server's `spec.allowedTeams` (§4.6). A
  `TerdutAlertSource` lives beside its `TerdutTeam`.
- No admission webhooks. CEL validation covers what it can; anything needing a live look at
  another object is a status condition, not an admission rejection.
- **Never manages external exposure for a `TerdutServer`** (Ingress, HTTPRoute, VirtualService) —
  a permanent non-goal. The operator creates a plain `ClusterIP` Service; see `examples/networking`.

## 2. Credentials and CRD shape (the 2026-10 redesign)

What the design is, and why — each point replaced something heavier:

- **No bootstrap handshake.** The operator generates a key into a Secret named
  `<TerdutServer>-operator-key`, in the TerdutServer's own namespace and owned by
  it (a pod can only mount Secrets of its own namespace, and an owner reference
  replaces the old finalizer and `credentials.deletionPolicy`). The Deployment
  hands it to terdut-server as `TERDUT_OPERATOR_KEY`; the server creates or
  re-keys its instance-scoped service account `terdut-operator` from it at every
  start. A replaced Secret rolls the pods (key-hash annotation). `/api/bootstrap`
  stays free for the first human administrator. Gone with it: the checkpoint
  Secret, `BootstrapStateLost`, `DatabaseReady`/`Bootstrapped` conditions.
- **One credential per server.** An instance-scoped account acts as owner of every
  team's configuration (not a member, so it reads no incidents). The per-team
  service accounts, Secrets and `status.credentialsSecretRef`/`serverEndpoint` on
  `TerdutTeam` are gone.
- **Team identity is `external_id`.** The operator creates a team with
  `external_id: <namespace>/<name>` of its CR; the server returns the existing team
  for a known id (200) instead of creating one, so crash recovery, a lost status
  and a deleted team all heal by repeating the same call, and a display name that
  belongs to another team is a 409 (`TeamNameTaken`) instead of an adoption.
- **Three CRDs.** `TerdutServer`, `TerdutTeam` and `TerdutAlertSource`.
  `TerdutEscalationRule` and `TerdutDeadmanSwitch` are `spec.escalation` and
  `spec.deadmanSwitches[]` (matched by name, unique per team server-side; ones not
  listed are removed) on the team: they were one-to-one children with the team's
  lifecycle, and folding them removes `teamRef`, the two-rules-clobber footgun and
  three controllers. `TerdutAlertSource` stays separate because it owns a webhook
  Secret in its own namespace.
- **Usernames are resolved by the server** (`PUT .../escalation` accepts
  `username`), so an unknown user is a `UnknownUser` condition, not a list-and-match.
- **Invites are removed.** Membership is not modelled; people get in through the
  server's own signup/OIDC.
- **Env mirrors the chart's where it matters:** OIDC claim names and `trustEmail`
  are spec fields; `TERDUT_DEADMAN_*` no longer exist on the server.

## 3. CRD catalog

Group `terdut.ryuvia.com`, version `v1alpha1`; module `git.ryuvia.com/niklas/terdut-operator`.

| Kind | Purpose |
|---|---|
| `TerdutServer` | One terdut-server install: Deployment, Service, database wiring, operator key, cross-namespace team consent. |
| `TerdutTeam` | One team on a server, possibly in another namespace: name, OIDC groups, escalation ladder, dead man's switches. |
| `TerdutAlertSource` | One alert-ingest integration on a team; owns a webhook Secret. |

## 4. Per-CRD spec

(The escalation and dead man's switch specs that used to be §4.3 and §4.4 are part of §4.2; the section numbers are kept because code comments cite them.)

### 4.1 `TerdutServer`

```yaml
apiVersion: terdut.ryuvia.com/v1alpha1
kind: TerdutServer
metadata:
  name: terdut
  namespace: oncall
spec:
  image:
    repository: git.ryuvia.com/niklas/terdut-server
    tag: v0.9.3
  replicas: 2                      # default since terdut-server v0.36.0's advisory locks; see TerdutServerSpec.Replicas
  networking:
    hostname: terdut.example.com
    servicePort: 8080
  database:
    dsn: "postgres://terdut@terdut-postgres:5432/terdut?sslmode=require"   # mutually exclusive with postgresClusterRef
    passwordSecretRef: {name: terdut.terdut-postgres.credentials.postgresql.acid.zalan.do, key: password}
    # --- OR ---
    postgresClusterRef: {name: terdut-postgres}   # Zalando `postgresql` CR in the same namespace
  sweeper:
    staleAfter: 6h
    archiveAfter: 168h
  notify:
    ntfyURL: "http://ntfy.ntfy.svc.cluster.local"
    fallbackTopic: ""
    repeatEvery: 15m
    tokenSecretRef: {name: "", key: token}
  oidc:
    enabled: false
    issuer: ""
    clientID: ""
    clientSecretRef: {name: "", key: client-secret}
    name: SSO
    scopes: "openid profile email"
    allowedGroups: []
    adminGroup: ""
    usernameClaim: preferred_username   # also emailClaim, groupsClaim
    trustEmail: false
    sessionMaxAge: 12h
  passwordLogin: true
  # Consent for TerdutTeams in OTHER namespaces to set serverRef at this
  # TerdutServer. Same-namespace TerdutTeams never need this. Modeled on
  # Gateway API's Gateway.spec.allowedListeners.namespaces (the ListenerSet
  # attachment pattern, not ReferenceGrant — see §2 for why).
  allowedTeams:
    namespaces:
      from: None        # None (default) | Same | All | Selector
      # selector:       # required, and only meaningful, when from: Selector
      #   matchLabels:
      #     terdut.ryuvia.com/allowed: "true"
  # Pod-level customization of the Deployment -- all optional, direct corev1
  # passthrough throughout (see PodSpec's own doc comment). A representative
  # subset:
  pod:
    resources:
      requests: {cpu: 100m, memory: 128Mi}
      limits: {memory: 256Mi}
    tolerations:
      - key: dedicated
        operator: Equal
        value: terdut
        effect: NoSchedule
    disruptionBudget:
      minAvailable: 1       # mutually exclusive with maxUnavailable
status:
  conditions: [...]                 # Ready
  observedGeneration: 3
  credentialsSecretRef: {name: terdut-operator-key, key: token}   # see §6; pure output, in this TerdutServer's own namespace
```

Field-for-field this is the chart's `values.yaml` reshaped as a spec — not
for migrating an existing chart-based install (§1: there is no such path),
just because the shape is already familiar from the chart, and the operator
absorbs what the chart's Deployment/Service/bootstrap-job templates used to
do.

`spec.database` fields are `+kubebuilder:validation:XValidation` guarded to
be mutually exclusive (`dsn` xor `postgresClusterRef`); mirrors the chart's
"chart provisions no database" stance — this operator provisions no database
either, only wires up one that exists.

`spec.allowedTeams.namespaces.from` defaults to `None`, matching
`allowedListeners`'s own default — a fresh `TerdutServer` accepts no
cross-namespace `TerdutTeam` until its owner opts in, same-namespace
`TerdutTeam`s are unaffected either way. `Selector` deliberately has no
per-name allowlist (no "and only these teams") — namespace-level consent is
the right granularity here, same reasoning as `ListenerSet`: the namespace
is the tenancy boundary, not the object.

`spec.pod` is pod-level customization of the Deployment, all optional and
directly reusing corev1 types wherever corev1 already models the knob
exactly (`tolerations`, `affinity`, `topologySpreadConstraints`,
`resources`, `securityContext`/`containerSecurityContext`, `extraEnv`/
`extraEnvFrom`, `extraVolumes`/`extraVolumeMounts`, `imagePullSecrets`) —
no custom wrapper buys anything for any of these, matching how
CloudNativePG and the Zalando postgres-operator both expose the same
knobs. `affinity` is pure user-supplied passthrough, not a
toggle-plus-generated-default the way a multi-replica-aware operator's
pod anti-affinity typically is: even though `replicas` now defaults to 2
(terdut-server v0.36.0's advisory locks made that safe, §4.1's own
illustrative YAML comment), this operator still never auto-generates
affinity of its own. `spec.pod.disruptionBudget` is the one field here that isn't a
straight PodTemplateSpec knob — when set, the controller reconciles a
`PodDisruptionBudget` selecting this `TerdutServer`'s pods; clearing it
deletes any it previously created (§7). `minAvailable`/`maxUnavailable`
are mutually exclusive, `+kubebuilder:validation:XValidation`-guarded the
same way as `spec.database`'s own `dsn`/`postgresClusterRef` rule.

### 4.2 `TerdutTeam`

```yaml
spec:
  serverRef: {name: terdut, namespace: platform-oncall}   # namespace optional; cross-namespace needs allowedTeams
  displayName: Platform
  oidc: {memberGroup: terdut-platform-members, ownerGroup: terdut-platform-owners}
  escalation:
    repeatCount: 2
    fallbackTopic: platform-oncall
    levels:
      - timeout: 5m
        targets: [{kind: oncall}, {kind: user, username: alice}]
      - timeout: 15m
        targets: [{kind: oncall}]
  deadmanSwitches:
    - {name: watchdog, matcher: "alertname=Watchdog,cluster=prod", timeout: 15m, severity: critical}
status:
  conditions: [...]      # Ready; reasons: ServerRefNotFound, RefNotPermitted, WaitingForServer,
                         # TeamNameTaken, UnknownUser, InvalidSpec, Adopted
  teamID: 42
  observedGeneration: 1
```

The team is created under the identity `<namespace>/<name>` of its CR (`external_id`), so a retry,
a lost status or a team deleted behind the operator's back all heal by repeating the same call; a
`displayName` that belongs to a different team is `TeamNameTaken`, never an adoption. Every
reconcile applies, in order: create-or-find, rename, OIDC groups, the escalation ladder (one PUT
of the whole ladder; absent `spec.escalation` clears it), and the switches. Switches are matched
by name (unique per team on the server): missing ones are created, changed ones updated in
place, and any not listed are deleted, since in operator mode nobody else can add one. Durations
are validated by a CRD pattern and re-checked (`InvalidSpec`). A username the server does not
know is `UnknownUser` until that person exists.

Team *membership* is not modelled: the server manages it through OIDC group sync and its own UI.

### 4.5 `TerdutAlertSource`

```yaml
spec:
  teamRef: {name: platform-team}
  kind: alertmanager
  name: "prod-alertmanager"
status:
  conditions: [...]
  integrationID: 3
  webhookURLSecretRef: {name: prod-alertmanager-terdut-webhook}   # url + key, never in status/spec
```

The integration key is shown by the API exactly once, at creation
(`Integration.Key`/`URL` in terdut-server's own model) — never re-readable,
a one-shot value. The controller writes it straight into
a generated, owner-referenced Secret on the create it caused and never logs
or stores it anywhere else; the CR's `status` carries only the Secret
reference, matching how e.g. cert-manager's `Certificate` exposes
`spec.secretName` rather than the key material itself.

### 4.6 Cross-namespace consent: `TerdutServer.spec.allowedTeams`

No separate CRD — the consent lives on `TerdutServer` itself (§4.1), following
Gateway API's `Gateway.spec.allowedListeners` (`ListenerSet` attachment)
rather than its `ReferenceGrant`, since a `TerdutTeam` attaching to a shared
`TerdutServer` is the same shape of relationship: a bounded set of expected
children attaching to a parent explicitly designed to be shared, not an
arbitrary cross-namespace backend reference (see §2 for the full comparison
of both patterns).

- `from: None` (default) — no cross-namespace `TerdutTeam` may resolve a
  `serverRef` into this `TerdutServer`. Same-namespace `TerdutTeam`s are
  always allowed regardless of this field.
- `from: Same` — equivalent to `None` in effect (same-namespace is already
  unrestricted); kept for parity with the upstream enum.
- `from: All` — any namespace in the cluster may reference in. Appropriate
  for a genuinely shared, cluster-wide `TerdutServer`; the audit trail is
  "check `allowedTeams` plus who has RBAC to create a `TerdutTeam`
  anywhere," which is materially weaker than `Selector`.
- `from: Selector` — only namespaces matching `spec.allowedTeams.namespaces.selector`
  (a standard `metav1.LabelSelector` over `Namespace` objects, exactly like
  `allowedListeners`'s own `selector`) may reference in. This is the
  recommended mode for the platform-team-owns-a-shared-server scenario this
  design targets: label the consuming namespaces once
  (e.g. `terdut.ryuvia.com/allowed-server: platform-oncall/terdut`) and new
  namespaces opt in by carrying the label, without editing the
  `TerdutServer` again.
- A `TerdutTeam`'s controller re-evaluates `allowedTeams` on every
  reconcile before it will resolve a cross-namespace `serverRef` — for
  `Selector`, this means a `Get` on its own `Namespace` object plus
  reading the target `TerdutServer`'s spec, not a List across the cluster.
  Same-namespace `serverRef` never consults this field at all.
- Narrowing or clearing `allowedTeams` (or unlabeling a namespace, under
  `Selector`) is a live revocation: the next reconcile of any `TerdutTeam`
  it used to authorize finds itself no longer permitted, flips
  `Ready: False, reason: RefNotPermitted`, and — deliberately — does
  **not** delete the team server-side on revocation alone; it stops
  reconciling further changes until access is restored. Deleting the
  `TerdutTeam` CR still deletes the team: consent gates what the operator
  starts acting on, not whether it may clean up after itself.

## 5. Reconciliation semantics

| Resource | Server verbs | Strategy |
|---|---|---|
| Team | POST (idempotent on `external_id`), PUT rename, PUT oidc-groups, DELETE | Repeat the idempotent POST, then PUT the rest, every reconcile. Delete runs in a finalizer; the server refuses (409) while the team has open incidents, which is retried. |
| Escalation ladder | PUT whole ladder | PUT the full desired ladder every reconcile; the server resolves usernames. |
| Dead man's switch | list, POST, PUT, DELETE; names unique per team | Diff by name against the list. |
| Integration (alert source) | POST, PATCH rename, DELETE | Rename via PATCH; a kind change is delete-and-recreate, which **rotates the webhook key** (Warning event). An integration deleted on the server is recreated with a new key (Warning event). |

General rules:
- **Periodic resync** (5 minutes) besides watch-triggered reconciles, to catch someone changing
  state directly against the server: the CR wins.
- **Finalizers** on `TerdutTeam` and `TerdutAlertSource` call the server's DELETE first. A 404 is
  success; any other failure blocks removal and surfaces as an event rather than orphaning a row.
  `TerdutServer` needs none: its Secret is owned and its database is never touched.
- **A webhook Secret that is lost fails closed.** The key is never re-readable from the server, so
  the controller does not mint a replacement for a live URL with no spec change to explain it:
  `Ready: False, reason: WebhookSecretLost`; delete and recreate the `TerdutAlertSource`.
- **Children resolve through the team.** A `TerdutAlertSource` finds its `TerdutTeam` (same
  namespace), requires it Ready, then reads the server named by the team's `serverRef` and that
  server's operator key. A `TerdutTeam` re-reconciles when its `TerdutServer` changes.
- **Cross-namespace consent is re-checked every reconcile** (§4.6), so revocation takes effect on
  the team's next pass.

## 6. Authentication to terdut-server's API

The operator talks to the server over HTTP with one bearer key per `TerdutServer`:

1. The `TerdutServer` controller creates the Secret `<name>-operator-key` (data key `token`,
   `tdsa_` + 48 hex characters) in the server's own namespace, owned by the `TerdutServer`. It
   is created once and never overwritten while it exists: the running server was seeded with it.
2. The Deployment hands it to the pods as `TERDUT_OPERATOR_KEY` through a `secretKeyRef` (the key
   never appears in the pod spec), plus an annotation holding a hash of it so a replaced Secret
   rolls the pods.
3. At every start the server creates or re-keys its instance-scoped service account
   `terdut-operator` from that value. An instance-scoped account acts as owner of every team's
   configuration but is not a member of any team, so it reads no incidents and is never an
   administrator.
4. `status.credentialsSecretRef` points at the Secret; `TerdutTeam` and `TerdutAlertSource`
   controllers read it from the server's namespace.

There is no bootstrap handshake: `/api/bootstrap` stays free for the first human administrator.
Deleting the `TerdutServer` deletes the key with it; a recreated one gets a new key and the server
re-seeds on its next start. Rotating by hand means deleting the Secret: the next reconcile makes
a new one and rolls the pods.

## 7. Ownership, status, garbage collection

- Everything the controllers generate in a CR's namespace carries an `OwnerReference` (Deployment,
  Service, PodDisruptionBudget, the operator key Secret, the webhook Secret), so GC cleans up
  and no finalizer is needed. The PDB exists only while `spec.pod.disruptionBudget` is set; the
  controller deletes it itself when the field is cleared.
- Every kind has a `Ready` condition and `status.observedGeneration`.
- No cluster-scoped aggregate object: `kubectl get terdutservers -A` is the overview.

## 8. Postgres integration

Mirrors the chart's existing two-path contract (`values.yaml`
`database.dsn`/`passwordSecret`), because that contract is already
documented and tested operationally:

- **Bring-your-own**: `spec.database.dsn` (no password) +
  `spec.database.passwordSecretRef` — the operator sets `PGPASSWORD` on the
  Deployment's container env from that Secret, exactly like the chart does
  today, and does nothing else. No connectivity check beyond what the
  Deployment's own readiness probe already gives.
- **Zalando `postgres-operator`**: `spec.database.postgresClusterRef` names a
  `postgresql.acid.zalan.do` CR in the same namespace. The controller:
  - Reads that CR's status for the primary Service name/port to build the DSN
    host — `<cluster>.<namespace>.svc:5432` — and database name convention.
  - Resolves the generated credentials Secret
    (`<user>.<cluster>.credentials.postgresql.acid.zalan.do`) the same way
    the chart's comment already documents, and wires it in as
    `PGPASSWORD` the same way.
  - Does not watch that Secret: a rotated credential is noticed at the next
    5-minute resync, and restarting pods on rotation is a §13 follow-up.
  - Requires read RBAC on `postgresql.acid.zalan.do` (optional CRD — the
    operator's ClusterRole/Role should not hard-fail if the CRD isn't
    installed and a given `TerdutServer` uses BYO DSN instead).

## 9. RBAC

- The operator needs `get/list/watch/create/update/patch/delete` on `Deployments`, `Services`,
  `PodDisruptionBudgets` and `Secrets` it owns, `get/list/watch` on `postgresql.acid.zalan.do`
  (optional; degraded gracefully when the CRD is absent), and `get/list/watch` on `Namespaces`
  (labels only, for `allowedTeams: {from: Selector}`).
- Secrets live in the `TerdutServer`'s namespace (the operator key) and in each
  `TerdutAlertSource`'s namespace (the webhook Secret), so the operator needs Secret access in
  every tenant namespace. Kubernetes RBAC has no owner-scoped grant finer than the namespace.
- **As built (2026-10): broader than the above.** The shipped default is a
  `ClusterRole` with full verbs on `Secrets` in every namespace (the chart's
  `rbac.namespaced: true` gives a `Role` in the release namespace only, which
  cannot serve tenant namespaces), and the manager's cache is not restricted
  to watched namespaces. A per-namespace `Role` split (a Role and RoleBinding per watched namespace, with the cache
  restricted to them) is the intended end state, not implemented, and the decision (2026-10) is to stay
  cluster-wide for now: treat this operator as able to read every Secret in the cluster.
- terdut-server's own RBAC is unaffected: the operator uses only its HTTP API (§6), never the
  Kubernetes API for app-level state.

## 10. Relationship to `charts/terdut-server`

The server chart's Deployment, Service and bootstrap Job are redundant once `TerdutServer`
exists: running both would have two controllers reconciling the same Deployment. The operator's
chart (`charts/terdut-operator`) installs the operator, CRDs and RBAC, and optionally one
`TerdutServer` from `values.yaml` (`terdutServer.enabled`). There is no migration from a
chart-based install and none is planned (§1); whatever the server chart deployed stays a separate
install until someone deletes it.

The operator's Deployment builder mirrors the chart's env block for the knobs both expose (the
chart's `deployment.yaml` and `buildEnv` in `internal/controller/terdutserver_deployment.go`);
a new server setting is added in `config.go`, the chart, and the operator, in that order.

## 11. Testing strategy

- `envtest` (controller-runtime's fake API server) for every controller's
  reconcile logic against the Kubernetes side.
- terdut-server's REST API is faked with a small `httptest.Server` per
  controller test driven by fixtures matching the real handlers'
  request/response shapes (already well-documented in
  `internal/api/*_test.go` on the server side) — no real Postgres or real
  terdut-server binary needed for controller unit tests.
- The golden path (`kind` cluster + real terdut-server image + real Postgres:
  create `TerdutServer` → `TerdutTeam` → `TerdutAlertSource`, verify through
  terdut-server's own API, delete, verify it is gone) is a manual pass via
  `examples/demo/run-demo.sh`, not a CI job.

## 12. Observability

- Standard controller-runtime metrics (reconcile duration/error counts) are
  enough for v1 — no custom metrics.
- Every externally-visible action (a team created or renamed, an integration recreated, a delete
  that failed) emits a
  Kubernetes `Event` on the CR, since that's what shows up in `kubectl
  describe` and gitops tooling (Argo CD/Flux) surfaces without extra wiring.

## 13. Deferred / out of scope

- CloudNativePG support, alongside the Zalando `postgresClusterRef` (same `spec.database` shape).
- Gitops-managed team membership (§4.2).
- Per-namespace RBAC with a restricted cache (§9).
- Automatic Deployment restart on upstream Postgres credential rotation.
- `spec.pod.priorityClassName`, pod labels beyond annotations, and an HPA for `TerdutServer`.
- Admission webhooks beyond CEL (for example, checking a `teamRef` exists at admission time).
- A shared API types module or generated client for terdut-server, terdut-operator and terdut-tui.
- OLM packaging.