terdut-operator/DESIGN.md

# terdut-operator design

This is the design reference for implementing terdut-operator. It exists so
implementation can start from settled decisions instead of re-litigating them
mid-PR. It is deliberately more detailed than the README; the README stays as
the short pitch and now points here instead of carrying open questions.

Written against terdut-server as of the Postgres-only, per-team-resources
version (teams, escalation policies, dead man's switches and integrations are
all rows scoped to a team, managed over `internal/api/*` — not env/config-file
driven; see that repo's `charts/terdut-server` for the current deploy story
this operator supersedes).

## 1. Goals & non-goals

**Goal:** let a terdut-server install — the server itself, its teams, their
escalation policies, dead man's switches and alert-source integrations — be
fully described as Kubernetes objects and managed through gitops, following
controller-runtime / Kubebuilder conventions.

**Non-goals (v1):**
- Not a general-purpose Postgres operator. It *consumes* a database that
  either the Zalando `postgres-operator` or something else already provides.
- Not managing Alertmanager itself, or the routing rules that decide which
  alerts reach which integration webhook — only the terdut-server side
  (creating the integration and handing back its URL/key).
- Not OLM packaging. Plain Kubebuilder manifests + Helm chart for
  install, matching how terdut-server itself ships.
- Cross-namespace references are limited to exactly one edge:
  `TerdutTeam.spec.serverRef` may name a `TerdutServer` in a different
  namespace, gated by an explicit consent object in the target namespace
  (§4.2, §4.6) — this is the multi-tenant shape the operator exists for
  (one platform team owns a `TerdutServer`; other teams self-service a
  `TerdutTeam` against it without needing write access to the server's
  namespace). Every other reference (`teamRef` on the escalation
  rule/dead-man-switch/alert-source CRDs) stays same-namespace-as-its-`TerdutTeam`
  only, in v1 — those manage a specific team's own resources and are
  expected to live alongside it.
- No validating/mutating admission webhooks in v1. CEL validation rules on
  the CRDs (OpenAPI `x-kubernetes-validations`) cover what they can; anything
  that needs a live look at another object (e.g. "does this teamRef exist")
  is a status condition, not an admission rejection — keeps v1 to a
  controller-only deployment with no cert-manager/webhook dependency.

## 2. The README's open questions, resolved

> How do team-crd connect with server-crd?

Explicit `spec.serverRef: {name, namespace}` on `TerdutTeam` — same reasoning
as before (explicit, greppable, trivially validated), but **`namespace` is
deliberately part of the reference**: one team can run and own a
`TerdutServer`, and other teams — in their own namespaces, without any write
access to the server-owning team's namespace — self-service a `TerdutTeam`
against it. `namespace` defaults to the `TerdutTeam`'s own namespace when
omitted, so the common single-tenant case (`serverRef: {name: terdut}`) is
unchanged.

This cross-namespace edge needs the target namespace's explicit consent —
otherwise any namespace in the cluster could point a `TerdutTeam` at
someone else's `TerdutServer` and have the operator provision a team +
credentials against it, which is a namespace-boundary violation, not a
gitops convenience. §4.6 covers the consent object
(`TerdutServerReferenceGrant`), modeled directly on Gateway API's
`ReferenceGrant` — the established Kubernetes pattern for exactly this
"object A in namespace X wants to reference object B in namespace Y"
problem. No other reference in this design (`teamRef` on the child CRDs)
crosses a namespace boundary, so this is the only place `ReferenceGrant`-style
consent is needed (§1, §4.2, §5, §6, §9).

> How does escalationrules, switches and alertsources connect to a team?

Explicit `spec.teamRef: {name}` on each of `TerdutEscalationRule`,
`TerdutDeadmanSwitch`, `TerdutAlertSource` — same reasoning, and it mirrors
terdut-server's own data model, where every one of these rows carries a
`team_id` foreign key already. A matcher/selector on `TerdutTeam` would be
inventing a second source of truth for an association the server already
models as a plain reference.

> Support for both postgres-operator (Zalando) and bring-your-own, how do we
> design that to be user friendly?

`TerdutServer.spec.database` is a oneOf, mirroring the chart's existing
`database.dsn` / `database.passwordSecret` contract (see §8):
- `dsn` + `passwordSecretRef` — bring-your-own, exactly today's chart inputs.
- `postgresClusterRef` — points at a Zalando `postgresql.acid.zalan.do` CR;
  the operator derives the DSN and resolves the generated credentials Secret
  itself (see §8). CloudNativePG support is a natural follow-up using the
  same shape and is called out as deferred (§13), not designed in detail now.

## 3. API group, versions, CRD catalog

- Group: `terdut.ryuvia.com`, version: `v1alpha1` (matches the `ryuvia.com`
  domain terdut-server already uses; bump to `v1beta1`/`v1` per the normal
  Kubernetes API graduation criteria once the shapes below have proven
  stable against a real install).
- Module: `git.ryuvia.com/niklas/terdut-operator`, scaffolded with
  Kubebuilder (controller-runtime), matching terdut-server's Go toolchain
  and house style.

| Kind | Scope | Purpose |
|---|---|---|
| `TerdutServer` | Namespaced | One terdut-server install: Deployment, Service, database wiring, bootstrap, operator credentials. |
| `TerdutTeam` | Namespaced | One team on a `TerdutServer`, possibly in another namespace: name, OIDC group mapping. |
| `TerdutServerReferenceGrant` | Namespaced (lives with the `TerdutServer`) | Consent for a `TerdutTeam` in another namespace to reference this namespace's `TerdutServer`(s). |
| `TerdutEscalationRule` | Namespaced | A team's escalation policy (levels, targets, repeat). |
| `TerdutDeadmanSwitch` | Namespaced | One dead man's switch on a team. |
| `TerdutAlertSource` | Namespaced | One alert-ingest integration on a team (currently: Alertmanager webhook). |

## 4. Per-CRD spec

### 4.1 `TerdutServer`

```yaml
apiVersion: terdut.ryuvia.com/v1alpha1
kind: TerdutServer
metadata:
  name: terdut
  namespace: oncall
spec:
  image:
    repository: git.ryuvia.com/niklas/terdut-server
    tag: v0.9.3
  replicas: 1                      # terdut-server is not horizontally-scale-tested; keep the field, default 1
  networking:
    hostname: terdut.example.com
    servicePort: 8080
    gatewayListener: ""            # same semantics as chart's networking.listener
  database:
    dsn: "postgres://terdut@terdut-postgres:5432/terdut?sslmode=require"   # mutually exclusive with postgresClusterRef
    passwordSecretRef: {name: terdut.terdut-postgres.credentials.postgresql.acid.zalan.do, key: password}
    # --- OR ---
    postgresClusterRef: {name: terdut-postgres}   # Zalando `postgresql` CR in the same namespace
  sweeper:
    staleAfter: 6h
    archiveAfter: 168h
  deadman:
    matchers: "alertname=Watchdog"
    timeout: 15m
    severity: critical
  notify:
    ntfyURL: "http://ntfy.ntfy.svc.cluster.local"
    fallbackTopic: ""
    repeatEvery: 15m
    tokenSecretRef: {name: "", key: token}
  oidc:
    enabled: false
    issuer: ""
    clientID: ""
    clientSecretRef: {name: "", key: client-secret}
    name: SSO
    scopes: "openid profile email"
    allowedGroups: []
    adminGroup: ""
    sessionMaxAge: 12h
  passwordLogin: true
status:
  conditions: [...]                 # Ready, DatabaseReady, Bootstrapped
  observedGeneration: 3
  serviceName: terdut
  operatorCredentialsSecretRef: {name: terdut-operator-credentials}  # see §6
```

Field-for-field this is the chart's `values.yaml` reshaped as a spec — the
operator absorbs the chart's Deployment/Service/bootstrap-job templates, so
existing installs have a direct mapping when migrating (see §10).

`spec.database` fields are `+kubebuilder:validation:XValidation` guarded to
be mutually exclusive (`dsn` xor `postgresClusterRef`); mirrors the chart's
"chart provisions no database" stance — this operator provisions no database
either, only wires up one that exists.

### 4.2 `TerdutTeam`

```yaml
spec:
  serverRef:
    name: terdut
    namespace: platform-oncall      # optional; defaults to this TerdutTeam's own namespace.
                                     # Cross-namespace requires a matching TerdutServerReferenceGrant
                                     # in that namespace (§4.6) — otherwise Ready: False, reason: RefNotPermitted.
  displayName: "Platform"          # -> POST /api/teams {"name": ...}; server assigns the ID
  oidc:
    memberGroup: "terdut-platform-members"
    ownerGroup: "terdut-platform-owners"
status:
  conditions: [...]
  teamID: 42                       # the server-side ID; needed by every child object's controller
  observedGeneration: 1
```

Team *membership* (which users belong, `team_members`) is explicitly **not**
modeled as a CRD field in v1: terdut-server already manages membership via
OIDC group sync at login for SSO installs, and manual membership for
password-login installs is a people-management action, not infrastructure —
forcing it through gitops would mean a human's team change goes through a PR
review. Flagged in §13 as revisitable if a real gitops-membership need shows up.

### 4.3 `TerdutEscalationRule`

```yaml
spec:
  teamRef: {name: platform-team}
  repeatCount: 2
  fallbackTopic: "platform-oncall"
  levels:
    - timeout: 5m
      targets:
        - kind: oncall            # "oncall" or "user"
        - kind: user
          username: alice         # resolved to a user ID by the controller at apply time
    - timeout: 15m
      targets:
        - kind: oncall
status:
  conditions: [...]
  observedGeneration: 1
```

One `TerdutEscalationRule` per team — the server itself models a policy as
one row (`escalation_policies`) with an owned list of levels, so a
one-CRD-to-one-policy mapping (not one-CRD-per-level) matches the server's
own aggregate and lets the whole thing be reconciled with the single
`PUT /api/teams/{teamID}/escalation` the API actually exposes (see §5).

### 4.4 `TerdutDeadmanSwitch`

```yaml
spec:
  teamRef: {name: platform-team}
  name: "prod-watchdog"
  matcher: "alertname=Watchdog,cluster=prod"
  timeout: 15m
  severity: critical
status:
  conditions: [...]
  switchID: 7
```

### 4.5 `TerdutAlertSource`

```yaml
spec:
  teamRef: {name: platform-team}
  kind: alertmanager
  name: "prod-alertmanager"
status:
  conditions: [...]
  integrationID: 3
  webhookURLSecretRef: {name: prod-alertmanager-terdut-webhook}   # url + key, never in status/spec
```

The integration key is shown by the API exactly once, at creation
(`Integration.Key`/`URL` in terdut-server's own model) — never re-readable,
same shape as the bootstrap admin key. The controller writes it straight into
a generated, owner-referenced Secret on the create it caused and never logs
or stores it anywhere else; the CR's `status` carries only the Secret
reference, matching how e.g. cert-manager's `Certificate` exposes
`spec.secretName` rather than the key material itself.

### 4.6 `TerdutServerReferenceGrant`

Consent object living in the *`TerdutServer`'s* namespace, modeled directly
on Gateway API's `ReferenceGrant` (`gateway.networking.k8s.io/v1beta1`):
without one, no `TerdutTeam` outside this namespace may resolve a
`serverRef` into it, no matter what it names.

```yaml
apiVersion: terdut.ryuvia.com/v1alpha1
kind: TerdutServerReferenceGrant
metadata:
  name: allow-app-teams
  namespace: platform-oncall        # the TerdutServer's namespace
spec:
  from:
    - group: terdut.ryuvia.com
      kind: TerdutTeam
      namespace: team-checkout      # one entry per namespace permitted to reference in
    - group: terdut.ryuvia.com
      kind: TerdutTeam
      namespace: team-payments
  to:
    - group: terdut.ryuvia.com
      kind: TerdutServer
      name: terdut                  # optional: omit to permit any TerdutServer in this namespace
```

- **No selector/wildcard-across-namespaces field** (e.g. no `namespaceSelector`)
  in v1, matching upstream `ReferenceGrant`'s own choice: the namespace owner
  enumerates exactly which namespaces may reference in, which is the whole
  point of requiring affirmative, auditable consent rather than an implicit
  or pattern-matched grant.
- A `TerdutTeam`'s controller checks for a matching grant (namespace +
  optionally name) on every reconcile before it will resolve `serverRef`
  cross-namespace, exactly mirroring how `HTTPRoute`/`Certificate`
  controllers check `ReferenceGrant` before honouring a cross-namespace
  `backendRef`/`secretRef`. Same-namespace `serverRef` never needs a grant.
- Deleting the grant is a live revocation: the next reconcile of any
  `TerdutTeam` it used to authorize finds no matching grant, flips
  `Ready: False, reason: RefNotPermitted`, and — deliberately — does **not**
  delete the team server-side on revocation alone; it stops reconciling
  further changes until access is restored or the `TerdutTeam` CR itself is
  deleted (whose finalizer still needs the credentials Secret described in
  §6 to clean up, so a revoked grant blocking *new* changes rather than
  forcing an immediate, possibly credential-less deletion is the safer
  failure mode).

## 5. Reconciliation semantics

terdut-server's REST surface (`internal/api/router.go`) does not give every
resource a full update verb, so reconciliation strategy is per-resource:

| Resource | Verbs available | Strategy |
|---|---|---|
| Team | POST create, PUT rename, DELETE, PUT oidc-groups | Real update-in-place: diff spec vs. last-applied, PUT the changed pieces. |
| Escalation policy | GET/PUT whole-policy | Update-in-place: PUT the full desired policy every reconcile that finds drift; cheap because whole-policy is small and already loaded whole server-side. |
| Dead man's switch | POST create, DELETE — **no PUT** | Delete-and-recreate on any spec diff other than `name`. The controller diffs against `status` (which mirrors what was last successfully applied) rather than re-reading the server every reconcile, to avoid a spurious recreate from field reordering. |
| Integration (alert source) | POST create, PATCH rename, DELETE | Rename via PATCH; any other spec change (kind) is delete-and-recreate, which **rotates the webhook key** — called out loudly in the CRD's field docs and in a `Warning` event, since it breaks whatever sends to the old URL/key until the new Secret is picked up. |

General rules for every controller:
- **Idempotent create**: before POSTing, check `status.<serverSideID>` is
  unset; if the server already has a same-named object from a previous
  partial reconcile (e.g. after a crash between POST and status-write), treat
  a 409/name-conflict as "adopt" — GET-by-name and populate status, rather
  than erroring forever. terdut-server's list endpoints in each of these
  areas return objects by name, so this is a straightforward correlation.
- **Periodic resync** in addition to watch-triggered reconciles (Kubebuilder
  default `RequeueAfter` on success, e.g. every 5–10 minutes) to catch drift
  from **someone changing state directly against the server's API/UI**,
  since gitops correctness means the CR wins, not "first write wins".
- **Finalizers** on every CRD that has a server-side counterpart, so deletion
  calls the corresponding DELETE before the Kubernetes object disappears.
  Failure to delete server-side (e.g. server unreachable) blocks finalizer
  removal and surfaces as a `Degraded` condition + event, rather than
  silently orphaning a row.
- **Owner chain for status resolution, not API calls**: `TerdutTeam`'s
  controller does not call any other controller; every child CRD's
  controller independently resolves its own `teamRef` → `TerdutTeam.status.teamID`
  and `serverRef` chain down to `TerdutServer.status.operatorCredentialsSecretRef`,
  the way any two independent controller-runtime reconcilers would. If the
  referenced parent isn't `Ready` yet, the child requeues with backoff and
  reports `Ready: False, reason: WaitingForTeam` — no cross-controller RPC.
- **Cross-namespace `serverRef` is re-checked every reconcile, not just at
  creation**: `TerdutTeam`'s controller lists `TerdutServerReferenceGrant`
  objects in the target namespace on every pass (cheap: a namespaced List
  with a field/label index, not a full-cluster scan) before touching the
  cross-namespace `TerdutServer` — revocation (§4.6) takes effect on the
  team's very next reconcile, not just when the CR is first applied.

## 6. Bootstrap & authentication to terdut-server's API

terdut-server has no first-class "service account" token type — API keys
belong to a real user row (`api_keys.user_id`). The chart's current answer is
a one-shot Job that calls `POST /api/bootstrap`, gets a one-time admin key
back, and stores it in a Secret (`bootstrap-job.yaml`). The operator absorbs
this rather than shelling out to curl:

1. On a `TerdutServer`'s first reconcile after its Deployment reports Ready
   (`/healthz` reachable through the Service), the controller calls
   `POST /api/bootstrap` itself with a fixed, recognizable identity —
   e.g. username `terdut-operator`, email `terdut-operator@<serverRef>` —
   distinct from `spec.bootstrap.username/email` used for the *human* first
   admin the chart bootstraps today. Two separate bootstrap identities are
   needed only if terdut-server's `/api/bootstrap` is single-shot per
   install; if it already returns 403 after the first caller regardless of
   identity, the operator instead **creates its bot user via
   `POST /api/users`** once a human admin exists — this needs confirming
   against the endpoint's actual behavior before implementation and is
   flagged as a first task, not assumed here.
2. The returned API key is written to a generated Secret
   (`<name>-operator-credentials`), owner-referenced to the `TerdutServer`,
   referenced back from `status.operatorCredentialsSecretRef`.
3. Every other **same-namespace** controller (EscalationRule, DeadmanSwitch,
   AlertSource, and any same-namespace `TerdutTeam`) reads that Secret
   directly to call the API — never its own credentials.
4. **Cross-namespace `TerdutTeam` never reads the source Secret directly.**
   Granting arbitrary consuming namespaces `get`/`list`/`watch` on a Secret
   in the server's namespace would mean *any* future workload in that
   namespace with Secret-read RBAC could be pointed at it too — the
   `TerdutServerReferenceGrant` (§4.6) only authorizes the `TerdutTeam`
   *kind*, not "read this Secret". Instead, once a `TerdutServer` observes at
   least one valid grant for a namespace, its own controller (which already
   holds the real credentials, and whose ServiceAccount is the only thing
   with legitimate cross-namespace write access — see §9) mirrors a copy of
   the Secret into that consenting namespace, named
   `<serverRef.name>.<serverRef.namespace>-terdut-credentials`, owned not by
   an `OwnerReference` (those can't cross namespaces) but tracked in the
   `TerdutServer`'s status and cleaned up explicitly when the grant
   authorizing that namespace is removed. The remote `TerdutTeam`'s
   controller reads only this local mirror, never the original.
   - This is one shared mirrored Secret per (server, consuming namespace)
     pair, not one per `TerdutTeam` — terdut-server's own API key isn't
     team-scoped (§13), so there is nothing finer to hand out; "scoped" here
     means scoped by *namespace boundary*, not by team permission.
5. **Rotation**: the key is a bearer credential with no expiry modeled
   server-side today. Rotation is manual (delete the Secret + the
   `api_keys` row via `DELETE /api/users/{id}/api-keys/{keyID}`, let the
   controller re-bootstrap) until/unless terdut-server grows key expiry; a
   rotation invalidates every mirror too, which the `TerdutServer` controller
   re-copies on its own next reconcile.
   Documented as an operational runbook note, not automated in v1.
6. This is explicitly a stand-in for a real scoped service-account token
   type; see §13.

## 7. Ownership, status, garbage collection

- Every generated object (Deployment, Service, credentials Secret, webhook
  Secret) carries a `metav1.OwnerReference` to the CR that caused it, in the
  same namespace — standard GC, no finalizer needed for these (only for the
  server-side REST resources, per §5).
- Status conditions follow the standard `metav1.Condition` shape with at
  least `Ready` on every kind, plus kind-specific ones (`TerdutServer`:
  `DatabaseReady`, `Bootstrapped`; children: `Synced`).
- `status.observedGeneration` on every kind, bumped only after a successful
  reconcile against that generation's spec — the standard way a client
  (or `kubectl wait`) tells "applied" from "seen".
- No cluster-scoped aggregation object (e.g. no cluster-wide "all servers"
  status) in v1 — `kubectl get terdutservers -A` is the aggregate view.

## 8. Postgres integration

Mirrors the chart's existing two-path contract (`values.yaml`
`database.dsn`/`passwordSecret`), because that contract is already
documented and tested operationally:

- **Bring-your-own**: `spec.database.dsn` (no password) +
  `spec.database.passwordSecretRef` — the operator sets `PGPASSWORD` on the
  Deployment's container env from that Secret, exactly like the chart does
  today, and does nothing else. No connectivity check beyond what the
  Deployment's own readiness probe already gives.
- **Zalando `postgres-operator`**: `spec.database.postgresClusterRef` names a
  `postgresql.acid.zalan.do` CR in the same namespace. The controller:
  - Reads that CR's status for the primary Service name/port to build the DSN
    host — `<cluster>.<namespace>.svc:5432` — and database name convention.
  - Resolves the generated credentials Secret
    (`<user>.<cluster>.credentials.postgresql.acid.zalan.do`) the same way
    the chart's comment already documents, and wires it in as
    `PGPASSWORD` the same way.
  - Watches that Secret (not just the `postgresql` CR) so a credential
    rotation triggers a requeue — the chart today requires a manual pod
    restart for this; the operator can at least detect and report it via a
    condition even if restarting on rotation is left as a §13 follow-up
    rather than done automatically (a rolling restart on credential change
    is a behavior change worth its own design pass, not folded in here).
  - Requires read RBAC on `postgresql.acid.zalan.do` (optional CRD — the
    operator's ClusterRole/Role should not hard-fail if the CRD isn't
    installed and a given `TerdutServer` uses BYO DSN instead).

## 9. RBAC

- The operator's own ServiceAccount needs, per namespace it's granted:
  `get/list/watch/create/update/patch/delete` on `Deployments`, `Services`,
  `Secrets` it owns, and `get/list/watch` on `postgresql.acid.zalan.do`
  (optional, degrade gracefully if absent per §8), plus `get/list/watch` on
  `TerdutServerReferenceGrant` (§4.6).
- No cluster-scoped resources are created by this operator (namespaced CRDs
  only, per §1) — a `Role` + `RoleBinding` per watched namespace is
  sufficient; a `ClusterRole` is only needed for watching CRDs across all
  namespaces, which is the normal Kubebuilder multi-tenant-operator default
  and doesn't imply cluster-scoped *managed* resources.
- **The operator's ServiceAccount is the only thing that ever holds
  cross-namespace `Secret` write.** It is a single Deployment/binary already
  watching every namespace it's granted (the normal Kubebuilder shape), so
  mirroring a credentials Secret into a consenting namespace (§6) is not a
  new privilege *boundary* — it's the same ServiceAccount that already
  reconciles objects there — but it is new *scope* (`create`/`update` on
  `Secrets` cluster-wide rather than only within each object's own
  namespace), and should be called out explicitly in the operator's
  ClusterRole/RBAC review, not left implicit. No human or team's own
  RBAC is ever granted cross-namespace Secret access by this design —
  `TerdutServerReferenceGrant` only ever authorizes the operator to act,
  never a person.
- terdut-server's own RBAC is unaffected — the operator talks to it purely
  over HTTP with the bot user's API key, never via the Kubernetes API for
  app-level state.

## 10. Relationship to `charts/terdut-server`

**Recommendation** (flagged explicitly as a decision to confirm before
implementation starts, not settled by this document alone): the chart's
Deployment/Service/bootstrap-job templates become redundant once
`TerdutServer` exists — running both would mean two controllers (Helm and
this operator) reconciling the same Deployment, which is exactly the
conflict Kubernetes operators exist to avoid. Proposed path:
- The chart is repurposed into an **installer chart**: it installs the
  operator + CRDs (and optionally one `TerdutServer` CR from `values.yaml`,
  for users who want "helm install and get a server" without hand-writing a
  CR) rather than templating the Deployment directly.
- Existing installs migrate by: `helm template` the current release's
  `values.yaml` into an equivalent `TerdutServer` CR (mechanical, since §4.1
  is deliberately shaped to make that mapping 1:1), install the operator,
  apply the CR, then let Helm's release be uninstalled or reduced to just
  the CRD/operator subchart.
- This is a breaking change to the chart's contract and needs its own
  migration guide and probably a major chart version bump — out of scope
  for this design doc beyond flagging it; do not start that migration work
  without separately confirming this recommendation.

## 11. Testing strategy

- `envtest` (controller-runtime's fake API server) for every controller's
  reconcile logic against the Kubernetes side.
- terdut-server's REST API is faked with a small `httptest.Server` per
  controller test driven by fixtures matching the real handlers'
  request/response shapes (already well-documented in
  `internal/api/*_test.go` on the server side) — no real Postgres or real
  terdut-server binary needed for controller unit tests.
- A smaller number of true end-to-end tests (`kind` cluster + real
  terdut-server image + real Postgres) covering the golden path per CRD:
  create `TerdutServer` → `TerdutTeam` → one of each child kind → verify via
  terdut-server's own API that the objects exist with the right shape →
  delete the CR → verify the server-side object is gone.

## 12. Observability

- Standard controller-runtime metrics (reconcile duration/error counts) are
  enough for v1 — no custom metrics.
- Every externally-visible action (bootstrap, key rotation-needed, delete-
  and-recreate on the no-PUT resources, adopt-on-conflict) emits a
  Kubernetes `Event` on the CR, since that's what shows up in `kubectl
  describe` and gitops tooling (Argo CD/Flux) surfaces without extra wiring.

## 13. Deferred / explicitly out of scope for this design

- A real scoped service-account/token type in terdut-server (least-privilege
  operator identity instead of a bot admin user) — worth raising as a
  terdut-server feature request, not designed here.
- CloudNativePG support — same `spec.database` shape as Zalando should
  extend to it, but the concrete field/Secret-naming conventions need their
  own look.
- Cross-namespace `teamRef` on the child CRDs (`TerdutEscalationRule`,
  `TerdutDeadmanSwitch`, `TerdutAlertSource`) — only `TerdutTeam.serverRef`
  crosses namespaces in v1 (§2, §4.2, §4.6); these stay same-namespace as
  their `TerdutTeam` until a real need for splitting them out shows up.
- A `TerdutServerReferenceGrant`-style namespace selector (rather than an
  enumerated list of namespaces) — deliberately omitted to match upstream
  `ReferenceGrant`'s own choice of explicit, auditable consent over
  pattern-matching (§4.6); revisit only if enumerating namespaces becomes
  genuinely unworkable at scale.
- Gitops-managed team *membership* (see §4.2).
- Automatic Deployment restart on upstream Postgres credential rotation.
- Admission webhooks / CEL-only validation limits (e.g. verifying a
  `teamRef` exists at admission time rather than surfacing it as a status
  condition after the fact).
- OLM packaging, Helm chart migration execution (§10 is a recommendation,
  not a plan to execute).