terdut-operator/DESIGN.md

# terdut-operator design

This is the design reference for implementing terdut-operator. It exists so
implementation can start from settled decisions instead of re-litigating them
mid-PR. It is deliberately more detailed than the README; the README stays as
the short pitch and now points here instead of carrying open questions.

Written against terdut-server as of the Postgres-only, per-team-resources
version (teams, escalation policies, dead man's switches and integrations are
all rows scoped to a team, managed over `internal/api/*` — not env/config-file
driven; see that repo's `charts/terdut-server` for the current deploy story
this operator supersedes).

## 1. Goals & non-goals

**Goal:** let a terdut-server install — the server itself, its teams, their
escalation policies, dead man's switches and alert-source integrations — be
fully described as Kubernetes objects and managed through gitops, following
controller-runtime / Kubebuilder conventions.

**Non-goals (v1):**
- Not a general-purpose Postgres operator. It *consumes* a database that
  either the Zalando `postgres-operator` or something else already provides.
- Not managing Alertmanager itself, or the routing rules that decide which
  alerts reach which integration webhook — only the terdut-server side
  (creating the integration and handing back its URL/key).
- Not OLM packaging. Plain Kubebuilder manifests + Helm chart for
  install, matching how terdut-server itself ships.
- Cross-namespace references are limited to exactly one edge:
  `TerdutTeam.spec.serverRef` may name a `TerdutServer` in a different
  namespace, gated by that `TerdutServer`'s own `spec.allowedTeams` consent
  field (§4.1, §4.2) — this is the multi-tenant shape the operator exists
  for (one platform team owns a `TerdutServer`; other teams self-service a
  `TerdutTeam` against it without needing write access to the server's
  namespace). Every other reference (`teamRef` on the escalation
  rule/dead-man-switch/alert-source CRDs) stays same-namespace-as-its-`TerdutTeam`
  only, in v1 — those manage a specific team's own resources and are
  expected to live alongside it.
- No validating/mutating admission webhooks in v1. CEL validation rules on
  the CRDs (OpenAPI `x-kubernetes-validations`) cover what they can; anything
  that needs a live look at another object (e.g. "does this teamRef exist")
  is a status condition, not an admission rejection — keeps v1 to a
  controller-only deployment with no cert-manager/webhook dependency.

## 2. The README's open questions, resolved

> How do team-crd connect with server-crd?

Explicit `spec.serverRef: {name, namespace}` on `TerdutTeam` — same reasoning
as before (explicit, greppable, trivially validated), but **`namespace` is
deliberately part of the reference**: one team can run and own a
`TerdutServer`, and other teams — in their own namespaces, without any write
access to the server-owning team's namespace — self-service a `TerdutTeam`
against it. `namespace` defaults to the `TerdutTeam`'s own namespace when
omitted, so the common single-tenant case (`serverRef: {name: terdut}`) is
unchanged.

This cross-namespace edge needs the target namespace's explicit consent —
otherwise any namespace in the cluster could point a `TerdutTeam` at
someone else's `TerdutServer` and have the operator provision a team on
its behalf, which is a namespace-boundary violation, not a gitops
convenience. (This consent gate is about which `TerdutTeam`s the operator
will act on, not about credential exposure — no terdut-server credential
is ever placed in a `TerdutTeam`'s own namespace regardless of this
setting; see §6.) Kubernetes has two established patterns for this kind
of consent, and Gateway API itself uses both, for two different
relationships:

- **`ReferenceGrant`** (used for a Route reaching into an arbitrary
  Service/Secret): a separate object, living in the *target* namespace,
  enumerating exact `{fromNamespace, fromKind} → {toKind, toName}` pairs.
  No wildcard, no selector — every permitted namespace is spelled out.
- **An inline field on the parent** (used for a `ListenerSet` attaching to
  a shared `Gateway`, GA since Gateway API v1.5): the parent carries
  `spec.allowedListeners.namespaces: {from: None|Same|All|Selector,
  selector}` directly, no separate CRD.

`TerdutTeam` attaching to a shared `TerdutServer` is structurally the
second case, not the first — a bounded set of expected children attaching
to a parent they were deliberately made shareable, not an arbitrary
backend reference — so this design follows the `ListenerSet` precedent:
`TerdutServer.spec.allowedTeams` (§4.1), no extra CRD. §4.2 covers how
`TerdutTeam` resolves against it. No other reference in this design
(`teamRef` on the child CRDs) crosses a namespace boundary, so this is the
only place cross-namespace consent is needed at all (§1, §5, §6, §9).

> How does escalationrules, switches and alertsources connect to a team?

Explicit `spec.teamRef: {name}` on each of `TerdutEscalationRule`,
`TerdutDeadmanSwitch`, `TerdutAlertSource` — same reasoning, and it mirrors
terdut-server's own data model, where every one of these rows carries a
`team_id` foreign key already. A matcher/selector on `TerdutTeam` would be
inventing a second source of truth for an association the server already
models as a plain reference.

> Support for both postgres-operator (Zalando) and bring-your-own, how do we
> design that to be user friendly?

`TerdutServer.spec.database` is a oneOf, mirroring the chart's existing
`database.dsn` / `database.passwordSecret` contract (see §8):
- `dsn` + `passwordSecretRef` — bring-your-own, exactly today's chart inputs.
- `postgresClusterRef` — points at a Zalando `postgresql.acid.zalan.do` CR;
  the operator derives the DSN and resolves the generated credentials Secret
  itself (see §8). CloudNativePG support is a natural follow-up using the
  same shape and is called out as deferred (§13), not designed in detail now.

## 3. API group, versions, CRD catalog

- Group: `terdut.ryuvia.com`, version: `v1alpha1` (matches the `ryuvia.com`
  domain terdut-server already uses; bump to `v1beta1`/`v1` per the normal
  Kubernetes API graduation criteria once the shapes below have proven
  stable against a real install).
- Module: `git.ryuvia.com/niklas/terdut-operator`, scaffolded with
  Kubebuilder (controller-runtime), matching terdut-server's Go toolchain
  and house style.

| Kind | Scope | Purpose |
|---|---|---|
| `TerdutServer` | Namespaced | One terdut-server install: Deployment, Service, database wiring, bootstrap, operator credentials, cross-namespace team consent. |
| `TerdutTeam` | Namespaced | One team on a `TerdutServer`, possibly in another namespace: name, OIDC group mapping. |
| `TerdutEscalationRule` | Namespaced | A team's escalation policy (levels, targets, repeat). |
| `TerdutDeadmanSwitch` | Namespaced | One dead man's switch on a team. |
| `TerdutAlertSource` | Namespaced | One alert-ingest integration on a team (currently: Alertmanager webhook). |

## 4. Per-CRD spec

### 4.1 `TerdutServer`

```yaml
apiVersion: terdut.ryuvia.com/v1alpha1
kind: TerdutServer
metadata:
  name: terdut
  namespace: oncall
spec:
  image:
    repository: git.ryuvia.com/niklas/terdut-server
    tag: v0.9.3
  replicas: 1                      # terdut-server is not horizontally-scale-tested; keep the field, default 1
  networking:
    hostname: terdut.example.com
    servicePort: 8080
    gatewayListener: ""            # same semantics as chart's networking.listener
  database:
    dsn: "postgres://terdut@terdut-postgres:5432/terdut?sslmode=require"   # mutually exclusive with postgresClusterRef
    passwordSecretRef: {name: terdut.terdut-postgres.credentials.postgresql.acid.zalan.do, key: password}
    # --- OR ---
    postgresClusterRef: {name: terdut-postgres}   # Zalando `postgresql` CR in the same namespace
  sweeper:
    staleAfter: 6h
    archiveAfter: 168h
  deadman:
    matchers: "alertname=Watchdog"
    timeout: 15m
    severity: critical
  notify:
    ntfyURL: "http://ntfy.ntfy.svc.cluster.local"
    fallbackTopic: ""
    repeatEvery: 15m
    tokenSecretRef: {name: "", key: token}
  oidc:
    enabled: false
    issuer: ""
    clientID: ""
    clientSecretRef: {name: "", key: client-secret}
    name: SSO
    scopes: "openid profile email"
    allowedGroups: []
    adminGroup: ""
    sessionMaxAge: 12h
  passwordLogin: true
  # Bring-your-own instance credential: a human mints this once, manually, with
  # their own admin session (`POST /api/service-accounts {name: ..., scope:
  # instance}`) and creates this Secret themselves, in the OPERATOR's own
  # namespace (same namespace status.credentialsSecretRef below would otherwise
  # point into). When set and the Secret exists, the controller adopts it
  # directly and skips bootstrap entirely -- see §6 for why this is required,
  # not optional polish, whenever terdut-server was already bootstrapped by its
  # chart or a human before this CR existed (the normal case, not an edge one).
  credentialsSecretRef: {name: "", key: token}
  # Consent for TerdutTeams in OTHER namespaces to set serverRef at this
  # TerdutServer. Same-namespace TerdutTeams never need this. Modeled on
  # Gateway API's Gateway.spec.allowedListeners.namespaces (the ListenerSet
  # attachment pattern, not ReferenceGrant — see §2 for why).
  allowedTeams:
    namespaces:
      from: None        # None (default) | Same | All | Selector
      # selector:       # required, and only meaningful, when from: Selector
      #   matchLabels:
      #     terdut.ryuvia.com/allowed: "true"
status:
  conditions: [...]                 # Ready, DatabaseReady, Bootstrapped
  observedGeneration: 3
  serviceName: terdut
  credentialsSecretRef: {name: terdut.platform-oncall-instance-credentials, namespace: terdut-operator-system}  # see §6; lives in the OPERATOR's namespace, not this TerdutServer's
```

Field-for-field this is the chart's `values.yaml` reshaped as a spec — the
operator absorbs the chart's Deployment/Service/bootstrap-job templates, so
existing installs have a direct mapping when migrating (see §10).

`spec.credentialsSecretRef` is an *input* (bring-your-own), distinct from
`status.credentialsSecretRef`'s *output* (generated-by-the-controller) —
when the input is set, the controller treats it as the credential outright
and echoes its name back into `status.credentialsSecretRef` rather than
generating a second Secret alongside it. Left unset, the controller falls
back to the self-registration flow (§6) — which only actually completes if
this `TerdutServer`'s own first reconcile is the one that wins the
`/api/bootstrap` race against a genuinely empty install; see §6 for why
that fallback is the exception, not the common case, and why this field
exists at all rather than being deferred as nice-to-have.

`spec.database` fields are `+kubebuilder:validation:XValidation` guarded to
be mutually exclusive (`dsn` xor `postgresClusterRef`); mirrors the chart's
"chart provisions no database" stance — this operator provisions no database
either, only wires up one that exists.

`spec.allowedTeams.namespaces.from` defaults to `None`, matching
`allowedListeners`'s own default — a fresh `TerdutServer` accepts no
cross-namespace `TerdutTeam` until its owner opts in, same-namespace
`TerdutTeam`s are unaffected either way. `Selector` deliberately has no
per-name allowlist (no "and only these teams") — namespace-level consent is
the right granularity here, same reasoning as `ListenerSet`: the namespace
is the tenancy boundary, not the object.

### 4.2 `TerdutTeam`

```yaml
spec:
  serverRef:
    name: terdut
    namespace: platform-oncall      # optional; defaults to this TerdutTeam's own namespace.
                                     # Cross-namespace requires that TerdutServer's spec.allowedTeams
                                     # (§4.1) to admit this namespace — otherwise Ready: False, reason: RefNotPermitted.
  displayName: "Platform"          # -> POST /api/teams {"name": ...}; server assigns the ID
  oidc:
    memberGroup: "terdut-platform-members"
    ownerGroup: "terdut-platform-owners"
status:
  conditions: [...]
  teamID: 42                       # the server-side ID; needed by every child object's controller
  credentialsSecretRef: {name: platform-oncall.platform-team-credentials, namespace: terdut-operator-system}  # see §6; this team's own scoped key, in the OPERATOR's namespace
  observedGeneration: 1
```

Team *membership* (which users belong, `team_members`) is explicitly **not**
modeled as a CRD field in v1: terdut-server already manages membership via
OIDC group sync at login for SSO installs, and manual membership for
password-login installs is a people-management action, not infrastructure —
forcing it through gitops would mean a human's team change goes through a PR
review. Flagged in §13 as revisitable if a real gitops-membership need shows up.

### 4.3 `TerdutEscalationRule`

```yaml
spec:
  teamRef: {name: platform-team}
  repeatCount: 2
  fallbackTopic: "platform-oncall"
  levels:
    - timeout: 5m
      targets:
        - kind: oncall            # "oncall" or "user"
        - kind: user
          username: alice         # resolved to a user ID by the controller at apply time
    - timeout: 15m
      targets:
        - kind: oncall
status:
  conditions: [...]
  observedGeneration: 1
```

One `TerdutEscalationRule` per team — the server itself models a policy as
one row (`escalation_policies`) with an owned list of levels, so a
one-CRD-to-one-policy mapping (not one-CRD-per-level) matches the server's
own aggregate and lets the whole thing be reconciled with the single
`PUT /api/teams/{teamID}/escalation` the API actually exposes (see §5).

### 4.4 `TerdutDeadmanSwitch`

```yaml
spec:
  teamRef: {name: platform-team}
  name: "prod-watchdog"
  matcher: "alertname=Watchdog,cluster=prod"
  timeout: 15m
  severity: critical
status:
  conditions: [...]
  switchID: 7
```

### 4.5 `TerdutAlertSource`

```yaml
spec:
  teamRef: {name: platform-team}
  kind: alertmanager
  name: "prod-alertmanager"
status:
  conditions: [...]
  integrationID: 3
  webhookURLSecretRef: {name: prod-alertmanager-terdut-webhook}   # url + key, never in status/spec
```

The integration key is shown by the API exactly once, at creation
(`Integration.Key`/`URL` in terdut-server's own model) — never re-readable,
same shape as the bootstrap admin key. The controller writes it straight into
a generated, owner-referenced Secret on the create it caused and never logs
or stores it anywhere else; the CR's `status` carries only the Secret
reference, matching how e.g. cert-manager's `Certificate` exposes
`spec.secretName` rather than the key material itself.

### 4.6 Cross-namespace consent: `TerdutServer.spec.allowedTeams`

No separate CRD — the consent lives on `TerdutServer` itself (§4.1), following
Gateway API's `Gateway.spec.allowedListeners` (`ListenerSet` attachment)
rather than its `ReferenceGrant`, since a `TerdutTeam` attaching to a shared
`TerdutServer` is the same shape of relationship: a bounded set of expected
children attaching to a parent explicitly designed to be shared, not an
arbitrary cross-namespace backend reference (see §2 for the full comparison
of both patterns).

- `from: None` (default) — no cross-namespace `TerdutTeam` may resolve a
  `serverRef` into this `TerdutServer`. Same-namespace `TerdutTeam`s are
  always allowed regardless of this field.
- `from: Same` — equivalent to `None` in effect (same-namespace is already
  unrestricted) but kept for parity with the upstream enum and to make the
  policy self-documenting in a diff.
- `from: All` — any namespace in the cluster may reference in. Appropriate
  for a genuinely shared, cluster-wide `TerdutServer`; the audit trail is
  "check `allowedTeams` plus who has RBAC to create a `TerdutTeam`
  anywhere," which is materially weaker than `Selector`.
- `from: Selector` — only namespaces matching `spec.allowedTeams.namespaces.selector`
  (a standard `metav1.LabelSelector` over `Namespace` objects, exactly like
  `allowedListeners`'s own `selector`) may reference in. This is the
  recommended mode for the platform-team-owns-a-shared-server scenario this
  design targets: label the consuming namespaces once
  (e.g. `terdut.ryuvia.com/allowed-server: platform-oncall/terdut`) and new
  namespaces opt in by carrying the label, without editing the
  `TerdutServer` again.
- A `TerdutTeam`'s controller re-evaluates `allowedTeams` on every
  reconcile before it will resolve a cross-namespace `serverRef` — for
  `Selector`, this means a `Get` on its own `Namespace` object plus
  reading the target `TerdutServer`'s spec, not a List across the cluster.
  Same-namespace `serverRef` never consults this field at all.
- Narrowing or clearing `allowedTeams` (or unlabeling a namespace, under
  `Selector`) is a live revocation: the next reconcile of any `TerdutTeam`
  it used to authorize finds itself no longer permitted, flips
  `Ready: False, reason: RefNotPermitted`, and — deliberately — does
  **not** delete the team server-side on revocation alone; it stops
  reconciling further changes until access is restored or the `TerdutTeam`
  CR itself is deleted (whose finalizer still needs the credentials
  Secret described in §6 to clean up, so blocking *new* changes rather
  than forcing an immediate, possibly credential-less deletion is the
  safer failure mode).

## 5. Reconciliation semantics

terdut-server's REST surface (`internal/api/router.go`) does not give every
resource a full update verb, so reconciliation strategy is per-resource:

| Resource | Verbs available | Strategy |
|---|---|---|
| Team | POST create, PUT rename, DELETE, PUT oidc-groups | Real update-in-place: diff spec vs. last-applied, PUT the changed pieces. |
| Escalation policy | GET/PUT whole-policy | Update-in-place: PUT the full desired policy every reconcile that finds drift; cheap because whole-policy is small and already loaded whole server-side. |
| Dead man's switch | POST create, DELETE — **no PUT** | Delete-and-recreate on any spec diff other than `name`. The controller diffs against `status` (which mirrors what was last successfully applied) rather than re-reading the server every reconcile, to avoid a spurious recreate from field reordering. |
| Integration (alert source) | POST create, PATCH rename, DELETE | Rename via PATCH; any other spec change (kind) is delete-and-recreate, which **rotates the webhook key** — called out loudly in the CRD's field docs and in a `Warning` event, since it breaks whatever sends to the old URL/key until the new Secret is picked up. (See the general rule below for what happens if the Secret is lost with *no* spec change.) |

General rules for every controller:
- **Idempotent create**: before POSTing, check `status.<serverSideID>` is
  unset; if the server already has a same-named object from a previous
  partial reconcile (e.g. after a crash between POST and status-write), treat
  a 409/name-conflict as "adopt" — GET-by-name and populate status, rather
  than erroring forever. terdut-server's list endpoints in each of these
  areas return objects by name, so this is a straightforward correlation.
- **Periodic resync** in addition to watch-triggered reconciles (Kubebuilder
  default `RequeueAfter` on success, e.g. every 5–10 minutes) to catch drift
  from **someone changing state directly against the server's API/UI**,
  since gitops correctness means the CR wins, not "first write wins".
- **Finalizers** on every CRD that has a server-side counterpart, so deletion
  calls the corresponding DELETE before the Kubernetes object disappears.
  Failure to delete server-side (e.g. server unreachable) blocks finalizer
  removal and surfaces as a `Degraded` condition + event, rather than
  silently orphaning a row.
- **Generated Secrets holding unrecoverable server-issued material are
  watched, and their loss is fail-closed, not self-healed.** Currently this
  is just `TerdutAlertSource`'s webhook Secret (§4.5): the controller adds
  it to its `Owns()` watches, not just the CR. If it disappears while
  `status.integrationID` is still set, the controller does **not** attempt
  to recreate it — the key is genuinely gone (§4.5: never stored anywhere
  but that one Secret), so silently minting a replacement would rotate a
  live production webhook URL with no corresponding spec change to explain
  why. Instead it flips `Ready: False, reason: WebhookSecretLost` and fires
  a `Warning` event telling the operator to delete and recreate the
  `TerdutAlertSource`. No new mechanism is needed for recovery: deleting the
  CR runs the existing finalizer (DELETE the still-live integration
  server-side, above), and recreating it runs the existing idempotent-create
  path (this same section) — a fresh POST, a new key, a new Secret. This is
  deliberately the same recovery motion as the kind-change rotation above,
  just human-triggered instead of spec-triggered.
- **Owner chain for status resolution, not API calls**: `TerdutTeam`'s
  controller does not call any other controller; every child CRD's
  controller independently resolves its own `teamRef` → `TerdutTeam.status`
  for both the `teamID` and the `credentialsSecretRef` it needs to call the
  API (§6) — it never needs to chain further up to `TerdutServer` at all,
  since the team's own scoped credential is everything a child resource's
  controller requires. If the referenced `TerdutTeam` isn't `Ready` yet
  (which includes not having a `credentialsSecretRef` set), the child
  requeues with backoff and reports `Ready: False, reason: WaitingForTeam`
  — no cross-controller RPC.
- **Cross-namespace `serverRef` is re-checked every reconcile, not just at
  creation**: `TerdutTeam`'s controller reads the target `TerdutServer`'s
  `spec.allowedTeams` (and, under `Selector`, a `Get` on its own `Namespace`
  object for labels) on every pass before touching a cross-namespace
  `TerdutServer` — revocation (§4.6) takes effect on the team's very next
  reconcile, not just when the CR is first applied.

## 6. Bootstrap & authentication to terdut-server's API

terdut-server's scoped service-account credential type
(`terdut-server`'s `SERVICE-ACCOUNTS.md`) has shipped — confirmed against
source: `internal/api/service_accounts.go`, migration
`014_service_accounts.sql`, and `internal/api/router.go` wiring it in under
`AuthMiddleware`. This section is no longer blocked on it; the "v1-blocking"
framing here was accurate when this section was first written and is stale
now.

**`GET /api/service-accounts?name=` is not an unauthenticated lookup, unlike
an earlier draft of this section assumed — confirmed against
`internal/api/middleware.go`'s `AuthMiddleware`, which hard-rejects any
request carrying neither a Bearer token nor a session cookie with `401`
before any handler (including this one's own internal, more permissive
name-filter check) ever runs.** This matters beyond a technicality: it means
the self-registration flow below (point 1) only ever completes for the
`TerdutServer` whose own controller happens to win the `/api/bootstrap` race
on a genuinely empty install. Every other case — including Stage 1's own
setup (`ROADMAP.md`): terdut-server deployed by its existing chart, which
runs its own bootstrap Job, *before* the `TerdutServer` CR or its controller
ever exist — leaves the controller with no credential and no authenticated
way to get one. `spec.credentialsSecretRef` (§4.1) exists to make that the
normal path, not an unhandled edge case: a human mints an instance-scoped
service account once, manually, with their own admin session, and hands the
controller that Secret directly.

**Every credential the operator holds — the one instance-scoped key per
`TerdutServer`, and one team-scoped key per `TerdutTeam` — lives in a Secret
in the *operator's own* namespace, never in the namespace of the CR it
authenticates for.** Reconciliation happens entirely inside the operator's
controller loop, which is a single Deployment/ServiceAccount already
watching every namespace it's granted (§9); nothing about calling
terdut-server's API on a CR's behalf requires the credential to be
physically located near that CR, and no CR owner (human or otherwise) ever
needs to see, hold, or have RBAC to read a terdut-server credential. This is
a straight simplification of an earlier draft of this section, which mirrored
a shared credential into each consenting namespace instead — that version
conflated "the CR's owner never needs to see this" (true, and preserved
here) with "so the credential must live in the CR's namespace" (a
non-sequitur once you don't need to grant *anyone else* namespace-local
read access). Dropping that assumption also removes an entire class of
complexity: no on-demand mirroring, no garbage-collecting an orphaned copy
when `allowedTeams` narrows, no "OwnerReferences can't cross namespaces so
track it in status instead" workaround — none of that machinery is needed
when nothing ever crosses into a tenant namespace in the first place.

1. **First reconcile checks `spec.credentialsSecretRef` before anything
   else.** Set and the Secret exists: adopt it as-is, set
   `status.credentialsSecretRef` to the same reference, `Bootstrapped: True`,
   done — no API call made at all. This is the path every Stage 1 install
   actually takes (`ROADMAP.md`): terdut-server deployed by its existing
   chart, bootstrapped by that chart's own Job, before this CR exists.
   Unset (or the named Secret doesn't exist yet): fall through to
   self-registration, confirmed against source (`internal/api/users.go`'s
   `handleBootstrap`): `/api/bootstrap` is single-shot *per install*, gated
   on `SELECT COUNT(*) FROM users` — once non-zero, every call 403s
   regardless of identity, exactly as `charts/terdut-server`'s own
   `bootstrap-job.yaml` already assumes (403 → "already bootstrapped,
   nothing to do", exit 0). Call `/api/bootstrap`; on `201`, its response
   (`{"user": ..., "api_key": {"key": "<raw>", ...}}`) hands back a real,
   usable admin key directly — use it for exactly one further call,
   `POST /api/service-accounts {name: "terdut-operator", scope: "instance"}`,
   and keep *that* key, not the raw admin one, as the lasting credential.
   **On `403`, there is no further fallback**: per this section's opening
   note, `GET /api/service-accounts?name=` needs a credential this
   controller does not have, so self-lookup cannot run unauthenticated. Set
   `Ready: False, reason: WaitingForCredential` with an event telling the
   human to mint an instance-scoped service account with their own admin
   session and set `spec.credentialsSecretRef`, and requeue with backoff —
   this is the expected, steady-state outcome whenever the operator loses
   (or never entered) the bootstrap race, not a transient error to retry
   past.
2. On the self-registration path only (point 1's bring-your-own path
   generates nothing — it adopts the human-provided Secret directly): the
   resulting instance-scoped key is written to a generated Secret in the
   **operator's own namespace** (e.g. `<serverRef.namespace>.<serverRef.name>-instance-credentials`),
   referenced back from `TerdutServer.status.credentialsSecretRef: {name,
   namespace}` (§4.1) — `namespace` is part of the reference now, since it's
   no longer the same namespace as the `TerdutServer` itself. No
   `OwnerReference` (those can't cross namespaces); the `TerdutServer`'s
   finalizer deletes this Secret directly as part of its own teardown,
   the same way it already has to clean up the server-side resources it
   created (§5's general finalizer rule extends naturally to this Secret).
3. When a `TerdutTeam` first becomes `Ready` (its `serverRef` resolved,
   `allowedTeams` satisfied if cross-namespace), its controller uses the
   `TerdutServer`'s instance-scoped credential (read from the operator's own
   namespace, resolved via the owner chain in §5) to mint a **team-scoped**
   service account for itself: `POST /api/service-accounts` with
   `scope: team, teamID: <status.teamID>`. The resulting key is written to
   its own generated Secret, again in the **operator's own namespace**
   (e.g. `<teamNamespace>.<teamName>-team-credentials`), referenced from
   `TerdutTeam.status.credentialsSecretRef` (§4.2). Same finalizer pattern as
   point 2: the `TerdutTeam`'s finalizer deletes this Secret as part of its
   own teardown.
4. Every child controller (`TerdutEscalationRule`, `TerdutDeadmanSwitch`,
   `TerdutAlertSource`) reads its team's `credentialsSecretRef` — resolved
   through its `teamRef` → `TerdutTeam.status` (§5) — and never touches the
   instance-scoped credential at all. Since child CRDs stay same-namespace-
   as-their-`TerdutTeam` in v1 (§1), and the credential itself lives in the
   operator's namespace regardless of where the `TerdutTeam` or its children
   are, this works identically whether the `TerdutTeam` is same-namespace or
   cross-namespace relative to its `TerdutServer` — there is no separate
   cross-namespace case to handle here at all, unlike the mirroring design
   this replaced.
5. **Blast radius**: a team-scoped key can only touch its own `team_id`'s
   escalation policy, dead-man switches, integrations, schedule and OIDC
   group bindings server-side (enforced by terdut-server itself, per
   `SERVICE-ACCOUNTS.md`) — compromising one such Secret (e.g. a bug that
   leaks operator-namespace Secrets, or an overly broad RBAC grant on that
   one namespace) exposes exactly one team, never the whole server. This is
   the real fix for what an earlier draft of this section called out as its
   weak point (every mirrored copy being server-admin-equivalent); it falls
   out of team-scoped credentials existing at all, independent of where
   they're stored — the operator-private storage described above closes the
   RBAC-footprint half of the problem, team scoping closes the credential-
   privilege half.
6. **Rotation**: `POST /api/service-accounts/{id}/keys` mints a new key on
   the existing account without recreating it; the old key is revoked via
   `DELETE /api/service-accounts/{id}/keys/{keyID}`; the operator's local
   Secret is updated in place. No DB-level workaround, no re-triggering a
   single-shot endpoint that can't fire twice (which is what made rotation
   unworkable under the old `/api/bootstrap`-only design).

## 7. Ownership, status, garbage collection

- Every generated object that lives in the *same* namespace as the CR that
  caused it (Deployment, Service, webhook Secret) carries a standard
  `metav1.OwnerReference` — GC handles these, no finalizer needed. The two
  credential Secrets from §6 are the one exception: they live in the
  operator's own namespace regardless of where their owning CR lives, so
  `OwnerReference` doesn't apply (cross-namespace) and cleanup instead runs
  through that CR's finalizer directly, alongside the server-side DELETE
  it already has to issue (§5).
- Status conditions follow the standard `metav1.Condition` shape with at
  least `Ready` on every kind, plus kind-specific ones (`TerdutServer`:
  `DatabaseReady`, `Bootstrapped`; children: `Synced`).
- `status.observedGeneration` on every kind, bumped only after a successful
  reconcile against that generation's spec — the standard way a client
  (or `kubectl wait`) tells "applied" from "seen".
- No cluster-scoped aggregation object (e.g. no cluster-wide "all servers"
  status) in v1 — `kubectl get terdutservers -A` is the aggregate view.

## 8. Postgres integration

Mirrors the chart's existing two-path contract (`values.yaml`
`database.dsn`/`passwordSecret`), because that contract is already
documented and tested operationally:

- **Bring-your-own**: `spec.database.dsn` (no password) +
  `spec.database.passwordSecretRef` — the operator sets `PGPASSWORD` on the
  Deployment's container env from that Secret, exactly like the chart does
  today, and does nothing else. No connectivity check beyond what the
  Deployment's own readiness probe already gives.
- **Zalando `postgres-operator`**: `spec.database.postgresClusterRef` names a
  `postgresql.acid.zalan.do` CR in the same namespace. The controller:
  - Reads that CR's status for the primary Service name/port to build the DSN
    host — `<cluster>.<namespace>.svc:5432` — and database name convention.
  - Resolves the generated credentials Secret
    (`<user>.<cluster>.credentials.postgresql.acid.zalan.do`) the same way
    the chart's comment already documents, and wires it in as
    `PGPASSWORD` the same way.
  - Watches that Secret (not just the `postgresql` CR) so a credential
    rotation triggers a requeue — the chart today requires a manual pod
    restart for this; the operator can at least detect and report it via a
    condition even if restarting on rotation is left as a §13 follow-up
    rather than done automatically (a rolling restart on credential change
    is a behavior change worth its own design pass, not folded in here).
  - Requires read RBAC on `postgresql.acid.zalan.do` (optional CRD — the
    operator's ClusterRole/Role should not hard-fail if the CRD isn't
    installed and a given `TerdutServer` uses BYO DSN instead).

## 9. RBAC

- The operator's own ServiceAccount needs, per namespace it's granted:
  `get/list/watch/create/update/patch/delete` on `Deployments`, `Services`
  it owns, and `get/list/watch` on `postgresql.acid.zalan.do` (optional,
  degrade gracefully if absent per §8), plus cluster-wide `get/list` on
  `Namespace` (labels only, for `allowedTeams: {from: Selector}` evaluation
  — §4.1, §4.6).
- **Two different `Secret` scopes, not one — corrected from an earlier draft
  of this section.** That earlier draft said `Secret` access was "scoped to
  the operator's own namespace only... nowhere else," reasoning that with
  every credential held privately in the operator's own namespace (§6)
  there was no legitimate reason to touch a `Secret` anywhere else. That was
  wrong once §4.5 existed:
  - The §6 credential Secrets (one instance-scoped key per `TerdutServer`,
    one team-scoped key per `TerdutTeam`) do live in, and are only ever
    touched from, the operator's own namespace —
    `get/list/watch/create/update/patch/delete` there, nowhere else. The
    rest of the original reasoning stands for *these* Secrets specifically:
    no human or team's own RBAC is ever granted access to a terdut-server
    credential by this design, and the operator itself never needs
    cross-namespace access to reach them.
  - The §4.5 webhook Secret is different: it's owned by and lives beside
    its `TerdutAlertSource`, in that CR's own tenant namespace, not the
    operator's. The per-namespace `Role` already granted for
    `Deployments`/`Services` in each watched namespace (below) must carry
    the same `Secret` verbs there too, or the controller cannot create,
    watch, or even detect the loss of (§5) that Secret at all.
  - This necessarily widens the operator's footprint in each watched
    tenant namespace to "any `Secret` in that namespace," not just the ones
    it created — Kubernetes RBAC has no owner-scoped grant finer than the
    namespace itself, and the design already accepts this same granularity
    for Deployments/Services there. Flagged as an accepted trade-off, not a
    silent gap (§13).
- No cluster-scoped resources are created by this operator (namespaced CRDs
  only, per §1) — a `Role` + `RoleBinding` per watched (tenant) namespace is
  sufficient for Deployments/Services/the webhook `Secret`/the optional
  Zalando CRD, plus a separate `Role` + `RoleBinding` in the operator's own
  namespace for the §6 credential Secrets; a `ClusterRole` is only needed
  for watching CRDs across all namespaces (the normal Kubebuilder
  multi-tenant-operator default) — none of the `Secret` access above needs
  to be cluster-scoped.
- terdut-server's own RBAC is unaffected — the operator talks to it purely
  over HTTP with service-account API keys (§6), never via the Kubernetes
  API for app-level state.

## 10. Relationship to `charts/terdut-server`

**Recommendation** (flagged explicitly as a decision to confirm before
implementation starts, not settled by this document alone): the chart's
Deployment/Service/bootstrap-job templates become redundant once
`TerdutServer` exists — running both would mean two controllers (Helm and
this operator) reconciling the same Deployment, which is exactly the
conflict Kubernetes operators exist to avoid. Proposed path:
- The chart is repurposed into an **installer chart**: it installs the
  operator + CRDs (and optionally one `TerdutServer` CR from `values.yaml`,
  for users who want "helm install and get a server" without hand-writing a
  CR) rather than templating the Deployment directly.
- Existing installs migrate by: `helm template` the current release's
  `values.yaml` into an equivalent `TerdutServer` CR (mechanical, since §4.1
  is deliberately shaped to make that mapping 1:1), install the operator,
  apply the CR, then let Helm's release be uninstalled or reduced to just
  the CRD/operator subchart.
- This is a breaking change to the chart's contract and needs its own
  migration guide and probably a major chart version bump — out of scope
  for this design doc beyond flagging it; do not start that migration work
  without separately confirming this recommendation.
- **This decision isn't only about the migration — it also decides who
  owns bootstrap.** §6 assumed the operator could always get its own
  bootstrap identity separately from the chart's; §6 point 1 shows that's
  false, so whichever of {chart's Job, operator controller} is expected to
  call `/api/bootstrap` first has to be settled explicitly (a one-paragraph
  call, not the full migration plan) *before* writing any operator
  bootstrap/credential code, not deferred alongside the rest of this
  section.

## 11. Testing strategy

- `envtest` (controller-runtime's fake API server) for every controller's
  reconcile logic against the Kubernetes side.
- terdut-server's REST API is faked with a small `httptest.Server` per
  controller test driven by fixtures matching the real handlers'
  request/response shapes (already well-documented in
  `internal/api/*_test.go` on the server side) — no real Postgres or real
  terdut-server binary needed for controller unit tests.
- A smaller number of true end-to-end tests (`kind` cluster + real
  terdut-server image + real Postgres) covering the golden path per CRD:
  create `TerdutServer` → `TerdutTeam` → one of each child kind → verify via
  terdut-server's own API that the objects exist with the right shape →
  delete the CR → verify the server-side object is gone.

## 12. Observability

- Standard controller-runtime metrics (reconcile duration/error counts) are
  enough for v1 — no custom metrics.
- Every externally-visible action (bootstrap, key rotation-needed, delete-
  and-recreate on the no-PUT resources, adopt-on-conflict) emits a
  Kubernetes `Event` on the CR, since that's what shows up in `kubectl
  describe` and gitops tooling (Argo CD/Flux) surfaces without extra wiring.

## 13. Deferred / explicitly out of scope for this design

- **A real scoped service-account/token type in terdut-server — not merely
  deferred, this is v1-blocking for §6 as written** (verified: without it,
  §6's bootstrap flow has no working credential-rotation path and no clean
  answer to the chart-vs-operator bootstrap race; see §6 points 1, 5, 6 and
  §10). Sequence this server-side change *before* implementing the
  `TerdutServer` controller's bootstrap logic, not after.
- **A version-discovery endpoint on terdut-server** (e.g. `GET /api/version`).
  Neither this operator nor terdut-tui has one today — both independently
  detect capability by probing specific routes (terdut-tui via `GET
  /api/teams` 404-checking; this operator would otherwise need to invent
  its own equivalent probe). An unattended reconciler is more exposed to a
  silent breaking API change than an interactive TUI a human is watching;
  raising this alongside the service-account request rather than inventing
  another route-probe here.
- CloudNativePG support — same `spec.database` shape as Zalando should
  extend to it, but the concrete field/Secret-naming conventions need their
  own look.
- Cross-namespace `teamRef` on the child CRDs (`TerdutEscalationRule`,
  `TerdutDeadmanSwitch`, `TerdutAlertSource`) — only `TerdutTeam.serverRef`
  crosses namespaces in v1 (§2, §4.2, §4.6); these stay same-namespace as
  their `TerdutTeam` until a real need for splitting them out shows up.
- Narrower-than-namespace RBAC for the §4.5 webhook Secret (Kubernetes RBAC
  has no owner-scoped grant below the namespace itself, per §9) — revisit
  if the widened per-tenant-namespace `Secret` access proves too broad in
  practice.
- Gitops-managed team *membership* (see §4.2).
- Automatic Deployment restart on upstream Postgres credential rotation.
- Admission webhooks / CEL-only validation limits (e.g. verifying a
  `teamRef` exists at admission time rather than surfacing it as a status
  condition after the fact).
- OLM packaging, Helm chart migration execution (§10 is a recommendation,
  not a plan to execute).