niklas/terdut-operator
1
0
Fork 0
Code Issues 6 Pull Requests Actions Packages Projects Releases Wiki Activity

Authenticate with a seeded operator key; fold escalation and switches into TerdutTeam

Browse Source 
Credentials: the TerdutServer controller generates <name>-operator-key in
the server's own namespace (owned by it) and hands it to the pods as
TERDUT_OPERATOR_KEY; the server creates its instance-scoped account from it
at every start. A replaced Secret rolls the pods. The bootstrap handshake,
the checkpoint Secret, per-team service accounts and credentials Secrets,
BootstrapStateLost and credentials.deletionPolicy are gone.

CRDs: TerdutServer, TerdutTeam and TerdutAlertSource. TerdutEscalationRule
and TerdutDeadmanSwitch become spec.escalation and spec.deadmanSwitches[]
on the team (matched by name, extras removed); team invites are removed.
A team is created under the identity <namespace>/<name> (external_id), so a
retry, a lost status or a deleted team heal by repeating the same call, and
a display name owned by another team is TeamNameTaken instead of an
adoption. The server resolves escalation usernames (UnknownUser condition).
OIDC claim names and trustEmail are spec fields.

Fixes: query values are URL-escaped; every delete treats 404 as success;
deleting a team no longer depends on allowedTeams consent; a switch or
integration deleted on the server is recreated; unnamed switches take the
CR's name.

Cleanup: scaffold e2e test, AGENTS.md, devcontainer, unused config/ pieces
and Client.Version() removed; DESIGN.md, README, ROADMAP and the demo
(run-demo.sh, manifests) rewritten for the new design. Secret RBAC stays
cluster-wide, now stated in DESIGN.md section 9.

Claude-Session: https://claude.ai/code/session_016mBLURvJoMuUEr9cB2RpUN
This commit is contained in:
Niklas Ye
2026-10-09 14:56:22 +02:00
parent b0a431f2a4
commit e1103f2b7d
92 changed files with 2082 additions and 7561 deletions
Download Patch File Download Diff File Expand all files Collapse all files
DESIGN.md
+180 -616
View File
@@ -1,146 +1,83 @@
# terdut-operator design
This is the design reference for implementing terdut-operator. It exists so
implementation can start from settled decisions instead of re-litigating them
mid-PR. It is deliberately more detailed than the README; the README stays as
the short pitch and now points here instead of carrying open questions.
Written against terdut-server as of the Postgres-only, per-team-resources
version (teams, escalation policies, dead man's switches and integrations are
all rows scoped to a team, managed over `internal/api/*` — not env/config-file
driven; see that repo's `charts/terdut-server` for the current deploy story
this operator supersedes).
The design of terdut-operator as built. terdut-server is the system of record; this operator
makes one install of it — the server, its teams, their escalation ladders, dead man's switches
and alert-source integrations — describable as Kubernetes objects and manageable through gitops.
## 1. Goals & non-goals
**Goal:** let a terdut-server install — the server itself, its teams, their
escalation policies, dead man's switches and alert-source integrations — be
fully described as Kubernetes objects and managed through gitops, following
**Goal:** a terdut-server install fully described as Kubernetes objects, following
controller-runtime / Kubebuilder conventions.
**The operator creates and owns every `TerdutServer` it manages. It never
adopts a pre-existing, independently-deployed terdut-server** — whether
deployed by hand or by `charts/terdut-server`. There is no migration path
from an existing chart-based install, and none is planned (§10): starting
with the operator means applying a fresh `TerdutServer` CR, not converting
one. This is the root a few things downstream hang off of — notably §6's
bootstrap flow, which only has to handle the operator bootstrapping a server
it just created, never a server something else already bootstrapped first.
**The operator creates and owns every `TerdutServer` it manages. It never adopts a pre-existing,
independently-deployed terdut-server**, whether deployed by hand or by `charts/terdut-server`.
There is no migration path from a chart-based install (§10): starting with the operator means
applying a fresh `TerdutServer`.
**Non-goals (v1):**
- Not a general-purpose Postgres operator. It *consumes* a database that
either the Zalando `postgres-operator` or something else already provides.
- Not managing Alertmanager itself, or the routing rules that decide which
alerts reach which integration webhook — only the terdut-server side
(creating the integration and handing back its URL/key).
- Not OLM packaging. Plain Kubebuilder manifests + Helm chart for
install, matching how terdut-server itself ships.
- Cross-namespace references are limited to exactly one edge:
`TerdutTeam.spec.serverRef` may name a `TerdutServer` in a different
namespace, gated by that `TerdutServer`'s own `spec.allowedTeams` consent
field (§4.1, §4.2) — this is the multi-tenant shape the operator exists
for (one platform team owns a `TerdutServer`; other teams self-service a
`TerdutTeam` against it without needing write access to the server's
namespace). Every other reference (`teamRef` on the escalation
rule/dead-man-switch/alert-source CRDs) stays same-namespace-as-its-`TerdutTeam`
only, in v1 — those manage a specific team's own resources and are
expected to live alongside it.
- No validating/mutating admission webhooks in v1. CEL validation rules on
the CRDs (OpenAPI `x-kubernetes-validations`) cover what they can; anything
that needs a live look at another object (e.g. "does this teamRef exist")
is a status condition, not an admission rejection — keeps v1 to a
controller-only deployment with no cert-manager/webhook dependency.
- **Never manages external exposure/ingress for `TerdutServer`, in any
form — a permanent non-goal, not a staged one.** The operator creates a
plain `ClusterIP` Service (§4.1) and stops there: no `Ingress`, no
Gateway API `HTTPRoute`, no Istio `VirtualService`, nothing. Some
installs won't expose `TerdutServer` outside the cluster at all; others
will use whichever of those mechanisms already fits their cluster. That
choice belongs to whoever deploys it, not to this operator — see
`examples/networking` for worked (but not operator-managed) examples.
- Not a Postgres operator. It consumes a database that the Zalando `postgres-operator` or
something else provides (§8).
- Not managing Alertmanager or its routing, only the terdut-server side (creating the
integration and handing back its URL/key).
- Not OLM packaging: plain Kubebuilder manifests and a Helm chart, like terdut-server.
- Cross-namespace references are limited to one edge: `TerdutTeam.spec.serverRef` may name a
`TerdutServer` in another namespace, gated by that server's `spec.allowedTeams` (§4.6). A
`TerdutAlertSource` lives beside its `TerdutTeam`.
- No admission webhooks. CEL validation covers what it can; anything needing a live look at
another object is a status condition, not an admission rejection.
- **Never manages external exposure for a `TerdutServer`** (Ingress, HTTPRoute, VirtualService) —
a permanent non-goal. The operator creates a plain `ClusterIP` Service; see `examples/networking`.
## 2. The README's open questions, resolved
## 2. Credentials and CRD shape (the 2026-10 redesign)
> How do team-crd connect with server-crd?
What the design is, and why — each point replaced something heavier:
Explicit `spec.serverRef: {name, namespace}` on `TerdutTeam` — same reasoning
as before (explicit, greppable, trivially validated), but **`namespace` is
deliberately part of the reference**: one team can run and own a
`TerdutServer`, and other teams — in their own namespaces, without any write
access to the server-owning team's namespace — self-service a `TerdutTeam`
against it. `namespace` defaults to the `TerdutTeam`'s own namespace when
omitted, so the common single-tenant case (`serverRef: {name: terdut}`) is
unchanged.
- **No bootstrap handshake.** The operator generates a key into a Secret named
`<TerdutServer>-operator-key`, in the TerdutServer's own namespace and owned by
it (a pod can only mount Secrets of its own namespace, and an owner reference
replaces the old finalizer and `credentials.deletionPolicy`). The Deployment
hands it to terdut-server as `TERDUT_OPERATOR_KEY`; the server creates or
re-keys its instance-scoped service account `terdut-operator` from it at every
start. A replaced Secret rolls the pods (key-hash annotation). `/api/bootstrap`
stays free for the first human administrator. Gone with it: the checkpoint
Secret, `BootstrapStateLost`, `DatabaseReady`/`Bootstrapped` conditions.
- **One credential per server.** An instance-scoped account acts as owner of every
team's configuration (not a member, so it reads no incidents). The per-team
service accounts, Secrets and `status.credentialsSecretRef`/`serverEndpoint` on
`TerdutTeam` are gone.
- **Team identity is `external_id`.** The operator creates a team with
`external_id: <namespace>/<name>` of its CR; the server returns the existing team
for a known id (200) instead of creating one, so crash recovery, a lost status
and a deleted team all heal by repeating the same call, and a display name that
belongs to another team is a 409 (`TeamNameTaken`) instead of an adoption.
- **Three CRDs.** `TerdutServer`, `TerdutTeam` and `TerdutAlertSource`.
`TerdutEscalationRule` and `TerdutDeadmanSwitch` are `spec.escalation` and
`spec.deadmanSwitches[]` (matched by name, unique per team server-side; ones not
listed are removed) on the team: they were one-to-one children with the team's
lifecycle, and folding them removes `teamRef`, the two-rules-clobber footgun and
three controllers. `TerdutAlertSource` stays separate because it owns a webhook
Secret in its own namespace.
- **Usernames are resolved by the server** (`PUT .../escalation` accepts
`username`), so an unknown user is a `UnknownUser` condition, not a list-and-match.
- **Invites are removed.** Membership is not modelled; people get in through the
server's own signup/OIDC.
- **Env mirrors the chart's where it matters:** OIDC claim names and `trustEmail`
are spec fields; `TERDUT_DEADMAN_*` no longer exist on the server.
This cross-namespace edge needs the target namespace's explicit consent —
otherwise any namespace in the cluster could point a `TerdutTeam` at
someone else's `TerdutServer` and have the operator provision a team on
its behalf, which is a namespace-boundary violation, not a gitops
convenience. (This consent gate is about which `TerdutTeam`s the operator
will act on, not about credential exposure — no terdut-server credential
is ever placed in a `TerdutTeam`'s own namespace regardless of this
setting; see §6.) Kubernetes has two established patterns for this kind
of consent, and Gateway API itself uses both, for two different
relationships:
## 3. CRD catalog
- **`ReferenceGrant`** (used for a Route reaching into an arbitrary
Service/Secret): a separate object, living in the *target* namespace,
enumerating exact `{fromNamespace, fromKind} → {toKind, toName}` pairs.
No wildcard, no selector — every permitted namespace is spelled out.
- **An inline field on the parent** (used for a `ListenerSet` attaching to
a shared `Gateway`, GA since Gateway API v1.5): the parent carries
`spec.allowedListeners.namespaces: {from: None|Same|All|Selector,
selector}` directly, no separate CRD.
Group `terdut.ryuvia.com`, version `v1alpha1`; module `git.ryuvia.com/niklas/terdut-operator`.
`TerdutTeam` attaching to a shared `TerdutServer` is structurally the
second case, not the first — a bounded set of expected children attaching
to a parent they were deliberately made shareable, not an arbitrary
backend reference — so this design follows the `ListenerSet` precedent:
`TerdutServer.spec.allowedTeams` (§4.1), no extra CRD. §4.2 covers how
`TerdutTeam` resolves against it. No other reference in this design
(`teamRef` on the child CRDs) crosses a namespace boundary, so this is the
only place cross-namespace consent is needed at all (§1, §5, §6, §9).
> How does escalationrules, switches and alertsources connect to a team?
Explicit `spec.teamRef: {name}` on each of `TerdutEscalationRule`,
`TerdutDeadmanSwitch`, `TerdutAlertSource` — same reasoning, and it mirrors
terdut-server's own data model, where every one of these rows carries a
`team_id` foreign key already. A matcher/selector on `TerdutTeam` would be
inventing a second source of truth for an association the server already
models as a plain reference.
> Support for both postgres-operator (Zalando) and bring-your-own, how do we
> design that to be user friendly?
`TerdutServer.spec.database` is a oneOf, mirroring the chart's existing
`database.dsn` / `database.passwordSecret` contract (see §8):
- `dsn` + `passwordSecretRef` — bring-your-own, exactly today's chart inputs.
- `postgresClusterRef` — points at a Zalando `postgresql.acid.zalan.do` CR;
the operator derives the DSN and resolves the generated credentials Secret
itself (see §8). CloudNativePG support is a natural follow-up using the
same shape and is called out as deferred (§13), not designed in detail now.
## 3. API group, versions, CRD catalog
- Group: `terdut.ryuvia.com`, version: `v1alpha1` (matches the `ryuvia.com`
domain terdut-server already uses; bump to `v1beta1`/`v1` per the normal
Kubernetes API graduation criteria once the shapes below have proven
stable against a real install).
- Module: `git.ryuvia.com/niklas/terdut-operator`, scaffolded with
Kubebuilder (controller-runtime), matching terdut-server's Go toolchain
and house style.
| Kind | Scope | Purpose |
|---|---|---|
| `TerdutServer` | Namespaced | One terdut-server install: Deployment, Service, database wiring, bootstrap, operator credentials, cross-namespace team consent. |
| `TerdutTeam` | Namespaced | One team on a `TerdutServer`, possibly in another namespace: name, OIDC group mapping. |
| `TerdutEscalationRule` | Namespaced | A team's escalation policy (levels, targets, repeat). |
| `TerdutDeadmanSwitch` | Namespaced | One dead man's switch on a team. |
| `TerdutAlertSource` | Namespaced | One alert-ingest integration on a team (currently: Alertmanager webhook). |
| Kind | Purpose |
|---|---|
| `TerdutServer` | One terdut-server install: Deployment, Service, database wiring, operator key, cross-namespace team consent. |
| `TerdutTeam` | One team on a server, possibly in another namespace: name, OIDC groups, escalation ladder, dead man's switches. |
| `TerdutAlertSource` | One alert-ingest integration on a team; owns a webhook Secret. |
## 4. Per-CRD spec
(The escalation and dead man's switch specs that used to be §4.3 and §4.4 are part of §4.2; the section numbers are kept because code comments cite them.)
### 4.1 `TerdutServer`
```yaml
@@ -165,12 +102,6 @@ spec:
sweeper:
staleAfter: 6h
archiveAfter: 168h
deadman:
matchers: "alertname=Watchdog"
timeout: 15m
severity: critical
credentials:
deletionPolicy: Retain # Retain (default) | Delete -- what deleting this CR does to the instance credential Secret; see §6 point 2
notify:
ntfyURL: "http://ntfy.ntfy.svc.cluster.local"
fallbackTopic: ""
@@ -185,6 +116,8 @@ spec:
scopes: "openid profile email"
allowedGroups: []
adminGroup: ""
usernameClaim: preferred_username # also emailClaim, groupsClaim
trustEmail: false
sessionMaxAge: 12h
passwordLogin: true
# Consent for TerdutTeams in OTHER namespaces to set serverRef at this
@@ -212,10 +145,9 @@ spec:
disruptionBudget:
minAvailable: 1 # mutually exclusive with maxUnavailable
status:
conditions: [...] # Ready, DatabaseReady, Bootstrapped
conditions: [...] # Ready
observedGeneration: 3
serviceName: terdut
credentialsSecretRef: {name: terdut.platform-oncall-instance-credentials, key: token} # see §6; pure output -- generated by the controller's own self-registration flow, always in the OPERATOR's namespace (always, implicitly -- not stored here, since it's never anything else), under a fixed key ("token").
credentialsSecretRef: {name: terdut-operator-key, key: token} # see §6; pure output, in this TerdutServer's own namespace
```
Field-for-field this is the chart's `values.yaml` reshaped as a spec — not
@@ -260,89 +192,37 @@ same way as `spec.database`'s own `dsn`/`postgresClusterRef` rule.
```yaml
spec:
serverRef:
name: terdut
namespace: platform-oncall # optional; defaults to this TerdutTeam's own namespace.
# Cross-namespace requires that TerdutServer's spec.allowedTeams
# (§4.1) to admit this namespace — otherwise Ready: False, reason: RefNotPermitted.
displayName: "Platform" # -> POST /api/teams {"name": ...}; server assigns the ID
oidc:
memberGroup: "terdut-platform-members"
ownerGroup: "terdut-platform-owners"
serverRef: {name: terdut, namespace: platform-oncall} # namespace optional; cross-namespace needs allowedTeams
displayName: Platform
oidc: {memberGroup: terdut-platform-members, ownerGroup: terdut-platform-owners}
escalation:
repeatCount: 2
fallbackTopic: platform-oncall
levels:
- timeout: 5m
targets: [{kind: oncall}, {kind: user, username: alice}]
- timeout: 15m
targets: [{kind: oncall}]
deadmanSwitches:
- {name: watchdog, matcher: "alertname=Watchdog,cluster=prod", timeout: 15m, severity: critical}
status:
conditions: [...]
teamID: 42 # the server-side ID; needed by every child object's controller
credentialsSecretRef: {name: platform-oncall.platform-team-credentials, key: token} # see §6; this team's own scoped key, always in the OPERATOR's own namespace (not stored here — same reasoning as TerdutServer's, §4.1)
serverEndpoint: "http://terdut.platform-oncall.svc:8080" # resolved once, here, from spec.serverRef -- see §5: this is what actually makes "a child never needs to chain up to TerdutServer" true, not just a stated intent. Set alongside teamID/credentialsSecretRef, same reconcile.
conditions: [...] # Ready; reasons: ServerRefNotFound, RefNotPermitted, WaitingForServer,
# TeamNameTaken, UnknownUser, InvalidSpec, Adopted
teamID: 42
observedGeneration: 1
```
Team *membership* (which users belong, `team_members`) is explicitly **not**
modeled as a CRD field in v1: terdut-server already manages membership via
OIDC group sync at login for SSO installs, and manual membership for
password-login installs is a people-management action, not infrastructure —
forcing it through gitops would mean a human's team change goes through a PR
review. Flagged in §13 as revisitable if a real gitops-membership need shows up.
The team is created under the identity `<namespace>/<name>` of its CR (`external_id`), so a retry,
a lost status or a team deleted behind the operator's back all heal by repeating the same call; a
`displayName` that belongs to a different team is `TeamNameTaken`, never an adoption. Every
reconcile applies, in order: create-or-find, rename, OIDC groups, the escalation ladder (one PUT
of the whole ladder; absent `spec.escalation` clears it), and the switches. Switches are matched
by name (unique per team on the server): missing ones are created, changed ones updated in
place, and any not listed are deleted, since in operator mode nobody else can add one. Durations
are validated by a CRD pattern and re-checked (`InvalidSpec`). A username the server does not
know is `UnknownUser` until that person exists.
### 4.3 `TerdutEscalationRule`
```yaml
spec:
teamRef: {name: platform-team}
repeatCount: 2
fallbackTopic: "platform-oncall"
levels:
- timeout: 5m
targets:
- kind: oncall # "oncall" or "user"
- kind: user
username: alice # resolved to a user ID by the controller at apply time
- timeout: 15m
targets:
- kind: oncall
status:
conditions: [...]
observedGeneration: 1
```
One `TerdutEscalationRule` per team — the server itself models a policy as
one row (`escalation_policies`) with an owned list of levels, so a
one-CRD-to-one-policy mapping (not one-CRD-per-level) matches the server's
own aggregate and lets the whole thing be reconciled with the single
`PUT /api/teams/{teamID}/escalation` the API actually exposes (see §5). Not
enforced at admission (no webhooks in v1, §1) — two `TerdutEscalationRule`s
naming the same team would both `PUT` it and clobber each other every
reconcile; a footgun worth this one sentence, not a technical guard.
A `username` target is resolved to the `user_id` `PUT /api/teams/{teamID}/escalation`
actually requires (confirmed against source: `escalationTargetJSON.UserID
*int64`, no username field at all) via `GET /api/users` — confirmed open to
any authenticated caller, not gated by team membership or admin
(`internal/api/router.go`'s own comment: "readable by anyone signed in"), so
the team-scoped credential this controller already holds is enough; no
extra RBAC-equivalent server-side needed. Re-resolved every reconcile rather
than cached, in case a username is renamed. An unresolvable username is
`Ready: False, reason: UnknownUser`, naming which one.
### 4.4 `TerdutDeadmanSwitch`
```yaml
spec:
teamRef: {name: platform-team}
name: "prod-watchdog"
matcher: "alertname=Watchdog,cluster=prod"
timeout: 15m
severity: critical
status:
conditions: [...]
switchID: 7
```
No unique-name constraint server-side (§5's table row, confirmed against
source) — the controller's own idempotent-create step is a `GET`-list and
name match, not a conflict to recover from. `name` is optional, same as the
API: left empty, terdut-server derives it from `matcher`'s own canonical
form, and that's what the lookup matches against too.
Team *membership* is not modelled: the server manages it through OIDC group sync and its own UI.
### 4.5 `TerdutAlertSource`
@@ -359,7 +239,7 @@ status:
The integration key is shown by the API exactly once, at creation
(`Integration.Key`/`URL` in terdut-server's own model) — never re-readable,
same shape as the bootstrap admin key. The controller writes it straight into
a one-shot value. The controller writes it straight into
a generated, owner-referenced Secret on the create it caused and never logs
or stores it anywhere else; the CR's `status` carries only the Secret
reference, matching how e.g. cert-manager's `Certificate` exposes
@@ -379,8 +259,7 @@ of both patterns).
`serverRef` into this `TerdutServer`. Same-namespace `TerdutTeam`s are
always allowed regardless of this field.
- `from: Same` — equivalent to `None` in effect (same-namespace is already
unrestricted) but kept for parity with the upstream enum and to make the
policy self-documenting in a diff.
unrestricted); kept for parity with the upstream enum.
- `from: All` — any namespace in the cluster may reference in. Appropriate
for a genuinely shared, cluster-wide `TerdutServer`; the audit trail is
"check `allowedTeams` plus who has RBAC to create a `TerdutTeam`
@@ -403,280 +282,64 @@ of both patterns).
it used to authorize finds itself no longer permitted, flips
`Ready: False, reason: RefNotPermitted`, and — deliberately — does
**not** delete the team server-side on revocation alone; it stops
reconciling further changes until access is restored or the `TerdutTeam`
CR itself is deleted (whose finalizer still needs the credentials
Secret described in §6 to clean up, so blocking *new* changes rather
than forcing an immediate, possibly credential-less deletion is the
safer failure mode).
reconciling further changes until access is restored. Deleting the
`TerdutTeam` CR still deletes the team: consent gates what the operator
starts acting on, not whether it may clean up after itself.
## 5. Reconciliation semantics
terdut-server's REST surface (`internal/api/router.go`) does not give every
resource a full update verb, so reconciliation strategy is per-resource:
| Resource | Verbs available | Strategy |
| Resource | Server verbs | Strategy |
|---|---|---|
| Team | POST create, PUT rename, DELETE, PUT oidc-groups, GET by name | Real update-in-place: diff spec vs. last-applied, PUT the changed pieces. `GET /api/teams?name=` (terdut-server's `TEAM-LOOKUP.md`, landed 2026-10-01) is what makes the idempotent-create general rule below actually true for Team — confirmed by checking: until that endpoint existed, an instance-scoped service account had no way to recover a team's id after a 409, unlike every other resource in this table, where the adopt-on-conflict rule had a real lookup to call. |
| Escalation policy | GET/PUT whole-policy | Update-in-place: PUT the full desired policy every reconcile that finds drift; cheap because whole-policy is small and already loaded whole server-side. |
| Dead man's switch | POST create, **PUT update-in-place** (added in terdut-server v0.33.0), DELETE, GET-list | Real update-in-place, same shape as Team/Escalation: PUT the whole switch every reconcile once its id is known. No unique-name constraint server-side (confirmed against source — `handleCreateTeamDeadman` has no conflict handling at all, unlike Team/service-account creation), so idempotent-create here can't rely on a 409 to adopt from: before POSTing, `GET /api/teams/{teamID}/deadman/switches` and match by `name` first: found → adopt its id; not found → POST. `handleUpdateTeamDeadman`'s own doc comment (terdut-server) confirms the motivation directly: "Added alongside create/delete so an automated caller (terdut-operator) can reconcile a spec change without deleting and recreating the switch, which would otherwise ... needlessly rotate its id for no reason a reconciler's diff should ever manufacture." An earlier draft of this row, written before that endpoint existed, described delete-and-recreate; corrected here, confirmed against source rather than left stale. |
| Integration (alert source) | POST create, PATCH rename, DELETE | Rename via PATCH; any other spec change (kind) is delete-and-recreate, which **rotates the webhook key** — called out loudly in the CRD's field docs and in a `Warning` event, since it breaks whatever sends to the old URL/key until the new Secret is picked up. (See the general rule below for what happens if the Secret is lost with *no* spec change.) |
| Team | POST (idempotent on `external_id`), PUT rename, PUT oidc-groups, DELETE | Repeat the idempotent POST, then PUT the rest, every reconcile. Delete runs in a finalizer; the server refuses (409) while the team has open incidents, which is retried. |
| Escalation ladder | PUT whole ladder | PUT the full desired ladder every reconcile; the server resolves usernames. |
| Dead man's switch | list, POST, PUT, DELETE; names unique per team | Diff by name against the list. |
| Integration (alert source) | POST, PATCH rename, DELETE | Rename via PATCH; a kind change is delete-and-recreate, which **rotates the webhook key** (Warning event). An integration deleted on the server is recreated with a new key (Warning event). |
General rules for every controller:
- **Idempotent create**: before POSTing, check `status.<serverSideID>` is
unset; if the server already has a same-named object from a previous
partial reconcile (e.g. after a crash between POST and status-write), treat
a 409/name-conflict as "adopt" — GET-by-name and populate status, rather
than erroring forever. terdut-server's list endpoints in each of these
areas return objects by name, so this is a straightforward correlation.
- **Periodic resync** in addition to watch-triggered reconciles (Kubebuilder
default `RequeueAfter` on success, e.g. every 5–10 minutes) to catch drift
from **someone changing state directly against the server's API/UI**,
since gitops correctness means the CR wins, not "first write wins".
- **Finalizers** on every CRD that has a server-side counterpart, so deletion
calls the corresponding DELETE before the Kubernetes object disappears.
Failure to delete server-side (e.g. server unreachable) blocks finalizer
removal and surfaces as a `Degraded` condition + event, rather than
silently orphaning a row.
- **Generated Secrets holding unrecoverable server-issued material are
watched, and their loss is fail-closed, not self-healed.** Currently this
is just `TerdutAlertSource`'s webhook Secret (§4.5): the controller adds
it to its `Owns()` watches, not just the CR. If it disappears while
`status.integrationID` is still set, the controller does **not** attempt
to recreate it — the key is genuinely gone (§4.5: never stored anywhere
but that one Secret), so silently minting a replacement would rotate a
live production webhook URL with no corresponding spec change to explain
why. Instead it flips `Ready: False, reason: WebhookSecretLost` and fires
a `Warning` event telling the operator to delete and recreate the
`TerdutAlertSource`. No new mechanism is needed for recovery: deleting the
CR runs the existing finalizer (DELETE the still-live integration
server-side, above), and recreating it runs the existing idempotent-create
path (this same section) — a fresh POST, a new key, a new Secret. This is
deliberately the same recovery motion as the kind-change rotation above,
just human-triggered instead of spec-triggered.
- **Owner chain for status resolution, not API calls**: `TerdutTeam`'s
controller does not call any other controller; every child CRD's
controller independently resolves its own `teamRef` → `TerdutTeam.status`
for the `teamID`, `credentialsSecretRef`, and `serverEndpoint` it needs to
call the API (§6) — it never needs to chain further up to `TerdutServer`
at all, not just as a stated intent but literally: `serverEndpoint` is
resolved once, by `TerdutTeam`'s own controller, and stored in its status
specifically so no child ever needs its own `TerdutServer` RBAC (`get`/
`list`/`watch` on `terdutservers`) to find out where to send a request —
the team's own scoped credential plus that one status field is everything
a child resource's controller requires. If the referenced `TerdutTeam`
isn't `Ready` yet (which includes not having a `credentialsSecretRef` set),
the child requeues with backoff and reports `Ready: False, reason:
WaitingForTeam` — no cross-controller RPC.
- **Cross-namespace `serverRef` is re-checked every reconcile, not just at
creation**: `TerdutTeam`'s controller reads the target `TerdutServer`'s
`spec.allowedTeams` (and, under `Selector`, a `Get` on its own `Namespace`
object for labels) on every pass before touching a cross-namespace
`TerdutServer` — revocation (§4.6) takes effect on the team's very next
reconcile, not just when the CR is first applied.
General rules:
- **Periodic resync** (5 minutes) besides watch-triggered reconciles, to catch someone changing
state directly against the server: the CR wins.
- **Finalizers** on `TerdutTeam` and `TerdutAlertSource` call the server's DELETE first. A 404 is
success; any other failure blocks removal and surfaces as an event rather than orphaning a row.
`TerdutServer` needs none: its Secret is owned and its database is never touched.
- **A webhook Secret that is lost fails closed.** The key is never re-readable from the server, so
the controller does not mint a replacement for a live URL with no spec change to explain it:
`Ready: False, reason: WebhookSecretLost`; delete and recreate the `TerdutAlertSource`.
- **Children resolve through the team.** A `TerdutAlertSource` finds its `TerdutTeam` (same
namespace), requires it Ready, then reads the server named by the team's `serverRef` and that
server's operator key. A `TerdutTeam` re-reconciles when its `TerdutServer` changes.
- **Cross-namespace consent is re-checked every reconcile** (§4.6), so revocation takes effect on
the team's next pass.
## 6. Bootstrap & authentication to terdut-server's API
## 6. Authentication to terdut-server's API
terdut-server's scoped service-account credential type
(`terdut-server`'s `SERVICE-ACCOUNTS.md`) has shipped — confirmed against
source: `internal/api/service_accounts.go`, migration
`014_service_accounts.sql`, and `internal/api/router.go` wiring it in under
`AuthMiddleware`. This section is no longer blocked on it; the "v1-blocking"
framing here was accurate when this section was first written and is stale
now.
The operator talks to the server over HTTP with one bearer key per `TerdutServer`:
**`GET /api/service-accounts?name=` is not an unauthenticated lookup —
confirmed against `internal/api/middleware.go`'s `AuthMiddleware`, which
hard-rejects any request carrying neither a Bearer token nor a session
cookie with `401` before any handler ever runs.** This would matter a great
deal if the operator's `/api/bootstrap` call could ever lose a race to
something else bootstrapping the same server first — a credential-less
loser would have no authenticated way to recover. It doesn't matter here,
by construction (§1): **the operator only ever calls `/api/bootstrap`
against a `TerdutServer` it just created**, so there is nothing else in a
position to race it. An earlier draft of this section added a
`spec.credentialsSecretRef` bring-your-own input specifically to work around
that race, for a world where the operator might adopt a server something
else had already bootstrapped. That world doesn't exist (§1), so the field
was removed rather than kept as unused flexibility — self-registration
(point 1, below) is simply the only path, not one of two.
1. The `TerdutServer` controller creates the Secret `<name>-operator-key` (data key `token`,
`tdsa_` + 48 hex characters) in the server's own namespace, owned by the `TerdutServer`. It
is created once and never overwritten while it exists: the running server was seeded with it.
2. The Deployment hands it to the pods as `TERDUT_OPERATOR_KEY` through a `secretKeyRef` (the key
never appears in the pod spec), plus an annotation holding a hash of it so a replaced Secret
rolls the pods.
3. At every start the server creates or re-keys its instance-scoped service account
`terdut-operator` from that value. An instance-scoped account acts as owner of every team's
configuration but is not a member of any team, so it reads no incidents and is never an
administrator.
4. `status.credentialsSecretRef` points at the Secret; `TerdutTeam` and `TerdutAlertSource`
controllers read it from the server's namespace.
**Every credential the operator holds — the one instance-scoped key per
`TerdutServer`, and one team-scoped key per `TerdutTeam` — lives in a Secret
in the *operator's own* namespace, never in the namespace of the CR it
authenticates for.** Reconciliation happens entirely inside the operator's
controller loop, which is a single Deployment/ServiceAccount already
watching every namespace it's granted (§9); nothing about calling
terdut-server's API on a CR's behalf requires the credential to be
physically located near that CR, and no CR owner (human or otherwise) ever
needs to see, hold, or have RBAC to read a terdut-server credential. This is
a straight simplification of an earlier draft of this section, which mirrored
a shared credential into each consenting namespace instead — that version
conflated "the CR's owner never needs to see this" (true, and preserved
here) with "so the credential must live in the CR's namespace" (a
non-sequitur once you don't need to grant *anyone else* namespace-local
read access). Dropping that assumption also removes an entire class of
complexity: no on-demand mirroring, no garbage-collecting an orphaned copy
when `allowedTeams` narrows, no "OwnerReferences can't cross namespaces so
track it in status instead" workaround — none of that machinery is needed
when nothing ever crosses into a tenant namespace in the first place.
1. **First reconcile, confirmed against source**
(`internal/api/users.go`'s `handleBootstrap`): `/api/bootstrap` is
single-shot *per install*, gated on `SELECT COUNT(*) FROM users` — once
non-zero, every call `403`s regardless of identity. There are two crash
windows between "get an admin key" and "have a lasting, usable
credential" — getting from `/api/bootstrap`'s key to a minted
service-account key, and getting from that key to a persisted Secret —
and both get a checkpoint rather than being left as a theoretical gap,
the same rigor §5's general idempotent-create rule already applies
elsewhere:
- `status.credentialsSecretRef` already set: done, nothing to do.
- Otherwise, look for the instance credential Secret itself
(`<namespace>.<name>-instance-credentials`, point 2 below): a
`TerdutServer` deleted and recreated under the same name leaves it
behind by default (`spec.credentials.deletionPolicy: Retain`), and the
server it logs in to has not changed, because deleting a
`TerdutServer` never touches its database. If it exists, ask the server
for the operator's own service account with that token
(`GET /api/service-accounts?name=terdut-operator`). Accepted: adopt it,
set `status.credentialsSecretRef`, and skip everything below. Rejected
with a `401`/`403`: it is stale (a database reset since), so ignore it
and carry on; the steps below replace it. Any other failure is a retry,
not a guess.
- Otherwise, check for an intermediate
`<namespace>.<name>-bootstrap-admin` Secret in
the operator's own namespace first. If it exists, its key is a still-
valid admin credential from an earlier, interrupted attempt — skip
`/api/bootstrap` entirely and reuse it. If not, call `/api/bootstrap`
once the Deployment this `TerdutServer` created has a ready replica;
on `201`, immediately checkpoint its response's raw admin key
(`{"user": ..., "api_key": {"key": "<raw>", ...}}`) into that Secret
before doing anything else with it. A `403` with neither
`status.credentialsSecretRef`, nor an instance credential the server
accepts, nor this checkpoint Secret present is the one genuinely
pathological case left: the server's database is already bootstrapped
and no credential for it survives here (the checkpoint deleted out from
under a reconcile already past this point, `deletionPolicy: Delete`, or
the Secret removed by hand). Fail closed, `Ready: False, reason:
BootstrapStateLost`, with a message that names the instance credential
Secret to restore. Deleting and recreating the `TerdutServer` does **not**
recover from it: the finalizer removes Secrets only, and the database
(the one thing that still says "already bootstrapped") is not the
operator's to reset. Restore the Secret from a copy, or reset the
server's database and then recreate the `TerdutServer`, which then
bootstraps like a first install. (This section used to say a fresh
create "starts clean"; that was only ever true when the database went
with it.)
- With an admin key in hand (fresh or checkpointed): `POST
/api/service-accounts {name: "terdut-operator", scope: "instance"}`.
A `409` here means a prior attempt got this far before being
interrupted — adopt rather than error, per §5's general rule:
`GET /api/service-accounts?name=terdut-operator` (authenticated with
the checkpointed admin key, not an unauthenticated lookup) to find its
id, then `POST /api/service-accounts/{id}/keys` to mint a fresh key —
an orphaned first key some interrupted attempt minted and never used
is inert, not a cleanup obligation.
2. The resulting instance-scoped key — not the checkpointed admin key, which
is deleted once this step succeeds — is written to a generated Secret in
the **operator's own namespace** (e.g.
`<serverRef.namespace>.<serverRef.name>-instance-credentials`,
under a fixed data key, `token`), referenced back from
`TerdutServer.status.credentialsSecretRef: {name, key}` (§4.1). No
`OwnerReference` (those can't cross namespaces, and this Secret doesn't
share a namespace with the `TerdutServer` that caused it). What the
`TerdutServer`'s finalizer does with it is `spec.credentials.deletionPolicy`:
`Retain` (the default) leaves it for a recreated `TerdutServer` to adopt
(point 1), `Delete` removes it directly as part of the teardown. There are
no server-side resources to undo either way: the bootstrap user and
service account have no delete verb in terdut-server's API. The bootstrap
checkpoint Secret is always removed. A retained Secret of a `TerdutServer`
that is gone for good is an orphan to delete by hand, and it is inert:
adoption asks the server to accept the token first.
3. When a `TerdutTeam` first becomes `Ready` (its `serverRef` resolved,
`allowedTeams` satisfied if cross-namespace), its controller uses the
`TerdutServer`'s instance-scoped credential (read from the operator's own
namespace, resolved via the owner chain in §5) to mint a **team-scoped**
service account for itself: `POST /api/service-accounts` with
`scope: team, teamID: <status.teamID>`. The resulting key is written to
its own generated Secret, again in the **operator's own namespace**
(e.g. `<teamNamespace>.<teamName>-team-credentials`), referenced from
`TerdutTeam.status.credentialsSecretRef` (§4.2). Same finalizer pattern as
point 2: the `TerdutTeam`'s finalizer deletes this Secret as part of its
own teardown.
4. Every child controller (`TerdutEscalationRule`, `TerdutDeadmanSwitch`,
`TerdutAlertSource`) reads its team's `credentialsSecretRef` — resolved
through its `teamRef` → `TerdutTeam.status` (§5) — and never touches the
instance-scoped credential at all. Since child CRDs stay same-namespace-
as-their-`TerdutTeam` in v1 (§1), and the credential itself lives in the
operator's namespace regardless of where the `TerdutTeam` or its children
are, this works identically whether the `TerdutTeam` is same-namespace or
cross-namespace relative to its `TerdutServer` — there is no separate
cross-namespace case to handle here at all, unlike the mirroring design
this replaced.
5. **Blast radius**: a team-scoped key can only touch its own `team_id`'s
escalation policy, dead-man switches, integrations, schedule and OIDC
group bindings server-side (enforced by terdut-server itself, per
`SERVICE-ACCOUNTS.md`) — compromising one such Secret (e.g. a bug that
leaks operator-namespace Secrets, or an overly broad RBAC grant on that
one namespace) exposes exactly one team, never the whole server. This is
the real fix for what an earlier draft of this section called out as its
weak point (every mirrored copy being server-admin-equivalent); it falls
out of team-scoped credentials existing at all, independent of where
they're stored — the operator-private storage described above closes the
RBAC-footprint half of the problem, team scoping closes the credential-
privilege half.
6. **Rotation**: `POST /api/service-accounts/{id}/keys` mints a new key on
the existing account without recreating it; the old key is revoked via
`DELETE /api/service-accounts/{id}/keys/{keyID}`; the operator's local
Secret is updated in place. No DB-level workaround, no re-triggering a
single-shot endpoint that can't fire twice (which is what made rotation
unworkable under the old `/api/bootstrap`-only design).
7. **Operator mode** (`TERDUT_OPERATOR_MODE`, terdut-server's own
deploy-time flag, off by default) is the complementary half of this
trust model: it makes terdut-server itself refuse a *human* write (a
session or a user's own API key) on a route, while a service account's
— this operator's — still goes through. Confirmed against source
(`internal/api/middleware.go`'s `OperatorModeBlock`,
`internal/api/router.go`'s `opMode` wrapper): the blocked set is exactly
team create/rename/delete, a team's OIDC-group binding, its escalation
policy, its dead man's switches, and its integrations — precisely the
resources `TerdutTeam`, `TerdutEscalationRule`, `TerdutDeadmanSwitch` and
`TerdutAlertSource` manage, and nothing more. **Deliberately not
blocked, confirmed against the same router**: team membership and
invites (the router's own comment: "membership is deliberately never
gitops-managed"), and the on-call schedule/rota
(`/api/teams/{teamID}/schedule`, `/api/schedule/current`) — neither
route carries the `opMode` wrapper at all. A human can still add or
remove a team member, or assign who's on call, on a server running in
operator mode; only the CRD-shaped resources above are locked to
GitOps. This isn't a gap to close — it's the same boundary §4.2 already
draws for membership, confirmed to hold on the server side too, not
just stated as an intent here.
There is no bootstrap handshake: `/api/bootstrap` stays free for the first human administrator.
Deleting the `TerdutServer` deletes the key with it; a recreated one gets a new key and the server
re-seeds on its next start. Rotating by hand means deleting the Secret: the next reconcile makes
a new one and rolls the pods.
## 7. Ownership, status, garbage collection
- Every generated object that lives in the *same* namespace as the CR that
caused it (Deployment, Service, webhook Secret, and `TerdutServer`'s own
PodDisruptionBudget) carries a standard `metav1.OwnerReference` — GC
handles these, no finalizer needed. PodDisruptionBudget is the one
member of that list that's conditionally created/deleted rather than
always present: it exists only while `spec.pod.disruptionBudget` is set,
and the controller deletes it itself the moment that field is cleared
(it doesn't wait on GC for that case, only for the `TerdutServer` being
deleted outright). The two credential Secrets from §6 are the one
exception to OwnerReference-based cleanup generally: they live in the
operator's own namespace regardless of where their owning CR lives, so
`OwnerReference` doesn't apply (cross-namespace) and cleanup instead runs
through that CR's finalizer directly, alongside the server-side DELETE
it already has to issue (§5).
- Status conditions follow the standard `metav1.Condition` shape with at
least `Ready` on every kind, plus kind-specific ones (`TerdutServer`:
`DatabaseReady`, `Bootstrapped`; children: `Synced`).
- `status.observedGeneration` on every kind, bumped only after a successful
reconcile against that generation's spec — the standard way a client
(or `kubectl wait`) tells "applied" from "seen".
- No cluster-scoped aggregation object (e.g. no cluster-wide "all servers"
status) in v1 — `kubectl get terdutservers -A` is the aggregate view.
- Everything the controllers generate in a CR's namespace carries an `OwnerReference` (Deployment,
Service, PodDisruptionBudget, the operator key Secret, the webhook Secret), so GC cleans up
and no finalizer is needed. The PDB exists only while `spec.pod.disruptionBudget` is set; the
controller deletes it itself when the field is cleared.
- Every kind has a `Ready` condition and `status.observedGeneration`.
- No cluster-scoped aggregate object: `kubectl get terdutservers -A` is the overview.
## 8. Postgres integration
@@ -697,85 +360,43 @@ documented and tested operationally:
(`<user>.<cluster>.credentials.postgresql.acid.zalan.do`) the same way
the chart's comment already documents, and wires it in as
`PGPASSWORD` the same way.
- Watches that Secret (not just the `postgresql` CR) so a credential
rotation triggers a requeue — the chart today requires a manual pod
restart for this; the operator can at least detect and report it via a
condition even if restarting on rotation is left as a §13 follow-up
rather than done automatically (a rolling restart on credential change
is a behavior change worth its own design pass, not folded in here).
- Does not watch that Secret: a rotated credential is noticed at the next
5-minute resync, and restarting pods on rotation is a §13 follow-up.
- Requires read RBAC on `postgresql.acid.zalan.do` (optional CRD — the
operator's ClusterRole/Role should not hard-fail if the CRD isn't
installed and a given `TerdutServer` uses BYO DSN instead).
## 9. RBAC
- The operator's own ServiceAccount needs, per namespace it's granted:
`get/list/watch/create/update/patch/delete` on `Deployments`, `Services`
and `PodDisruptionBudgets` it owns, and `get/list/watch` on
`postgresql.acid.zalan.do` (optional, degrade gracefully if absent per
§8), plus cluster-wide `get/list` on `Namespace` (labels only, for
`allowedTeams: {from: Selector}` evaluation — §4.1, §4.6).
- **Two different `Secret` scopes, not one — corrected from an earlier draft
of this section.** That earlier draft said `Secret` access was "scoped to
the operator's own namespace only... nowhere else," reasoning that with
every credential held privately in the operator's own namespace (§6)
there was no legitimate reason to touch a `Secret` anywhere else. That was
wrong once §4.5 existed:
- The §6 credential Secrets (one instance-scoped key per `TerdutServer`,
one team-scoped key per `TerdutTeam`) do live in, and are only ever
touched from, the operator's own namespace —
`get/list/watch/create/update/patch/delete` there, nowhere else. The
rest of the original reasoning stands for *these* Secrets specifically:
no human or team's own RBAC is ever granted access to a terdut-server
credential by this design, and the operator itself never needs
cross-namespace access to reach them.
- The §4.5 webhook Secret is different: it's owned by and lives beside
its `TerdutAlertSource`, in that CR's own tenant namespace, not the
operator's. The per-namespace `Role` already granted for
`Deployments`/`Services` in each watched namespace (below) must carry
the same `Secret` verbs there too, or the controller cannot create,
watch, or even detect the loss of (§5) that Secret at all.
- This necessarily widens the operator's footprint in each watched
tenant namespace to "any `Secret` in that namespace," not just the ones
it created — Kubernetes RBAC has no owner-scoped grant finer than the
namespace itself, and the design already accepts this same granularity
for Deployments/Services there. Flagged as an accepted trade-off, not a
silent gap (§13).
- No cluster-scoped resources are created by this operator (namespaced CRDs
only, per §1) — a `Role` + `RoleBinding` per watched (tenant) namespace is
sufficient for Deployments/Services/the webhook `Secret`/the optional
Zalando CRD, plus a separate `Role` + `RoleBinding` in the operator's own
namespace for the §6 credential Secrets; a `ClusterRole` is only needed
for watching CRDs across all namespaces (the normal Kubebuilder
multi-tenant-operator default) — none of the `Secret` access above needs
to be cluster-scoped.
- terdut-server's own RBAC is unaffected — the operator talks to it purely
over HTTP with service-account API keys (§6), never via the Kubernetes
API for app-level state.
- The operator needs `get/list/watch/create/update/patch/delete` on `Deployments`, `Services`,
`PodDisruptionBudgets` and `Secrets` it owns, `get/list/watch` on `postgresql.acid.zalan.do`
(optional; degraded gracefully when the CRD is absent), and `get/list/watch` on `Namespaces`
(labels only, for `allowedTeams: {from: Selector}`).
- Secrets live in the `TerdutServer`'s namespace (the operator key) and in each
`TerdutAlertSource`'s namespace (the webhook Secret), so the operator needs Secret access in
every tenant namespace. Kubernetes RBAC has no owner-scoped grant finer than the namespace.
- **As built (2026-10): broader than the above.** The shipped default is a
`ClusterRole` with full verbs on `Secrets` in every namespace (the chart's
`rbac.namespaced: true` gives a `Role` in the release namespace only, which
cannot serve tenant namespaces), and the manager's cache is not restricted
to watched namespaces. A per-namespace `Role` split (a Role and RoleBinding per watched namespace, with the cache
restricted to them) is the intended end state, not implemented, and the decision (2026-10) is to stay
cluster-wide for now: treat this operator as able to read every Secret in the cluster.
- terdut-server's own RBAC is unaffected: the operator uses only its HTTP API (§6), never the
Kubernetes API for app-level state.
## 10. Relationship to `charts/terdut-server`
The chart's Deployment/Service/bootstrap-job templates are redundant once
`TerdutServer` exists — running both would mean two controllers (Helm and
this operator) reconciling the same Deployment, which is exactly the
conflict Kubernetes operators exist to avoid. The chart is repurposed into
an **installer chart**: it installs the operator + CRDs (and optionally one
`TerdutServer` CR from `values.yaml`, for users who want "helm install and
get a server" without hand-writing a CR) rather than templating the
Deployment directly.
The server chart's Deployment, Service and bootstrap Job are redundant once `TerdutServer`
exists: running both would have two controllers reconciling the same Deployment. The operator's
chart (`charts/terdut-operator`) installs the operator, CRDs and RBAC, and optionally one
`TerdutServer` from `values.yaml` (`terdutServer.enabled`). There is no migration from a
chart-based install and none is planned (§1); whatever the server chart deployed stays a separate
install until someone deletes it.
**No migration path from an existing chart-based install, and none is
planned (§1).** An earlier draft of this section spent most of its length on
one — `helm template` the current release's `values.yaml` into an equivalent
`TerdutServer` CR, uninstall or shrink the old release, and a whole
sub-question about who gets to call `/api/bootstrap` first, the chart's Job
or the operator — all of which presupposed the operator might end up
managing a server the chart had already deployed and bootstrapped. §1 rules
that out: the operator only ever manages servers it created itself, so
there's nothing to migrate and no bootstrap race to settle (§6 covers why
that race doesn't exist either). Adopting the operator means applying a
fresh `TerdutServer` CR; whatever the chart deployed before stays exactly
what it was, a separate install, until someone deletes it.
The operator's Deployment builder mirrors the chart's env block for the knobs both expose (the
chart's `deployment.yaml` and `buildEnv` in `internal/controller/terdutserver_deployment.go`);
a new server setting is added in `config.go`, the chart, and the operator, in that order.
## 11. Testing strategy
@@ -786,84 +407,27 @@ what it was, a separate install, until someone deletes it.
request/response shapes (already well-documented in
`internal/api/*_test.go` on the server side) — no real Postgres or real
terdut-server binary needed for controller unit tests.
- A smaller number of true end-to-end tests (`kind` cluster + real
terdut-server image + real Postgres) covering the golden path per CRD:
create `TerdutServer` → `TerdutTeam` → one of each child kind → verify via
terdut-server's own API that the objects exist with the right shape →
delete the CR → verify the server-side object is gone.
- The golden path (`kind` cluster + real terdut-server image + real Postgres:
create `TerdutServer` → `TerdutTeam` → `TerdutAlertSource`, verify through
terdut-server's own API, delete, verify it is gone) is a manual pass via
`examples/demo/run-demo.sh`, not a CI job.
## 12. Observability
- Standard controller-runtime metrics (reconcile duration/error counts) are
enough for v1 — no custom metrics.
- Every externally-visible action (bootstrap, key rotation-needed, delete-
and-recreate on the no-PUT resources, adopt-on-conflict) emits a
- Every externally-visible action (a team created or renamed, an integration recreated, a delete
that failed) emits a
Kubernetes `Event` on the CR, since that's what shows up in `kubectl
describe` and gitops tooling (Argo CD/Flux) surfaces without extra wiring.
## 13. Deferred / explicitly out of scope for this design
## 13. Deferred / out of scope
- **A real scoped service-account/token type in terdut-server — not merely
deferred, this is v1-blocking for §6 as written** (verified: without it,
§6's bootstrap flow has no working credential-rotation path and no clean
answer to the chart-vs-operator bootstrap race; see §6 points 1, 5, 6 and
§10). Sequence this server-side change *before* implementing the
`TerdutServer` controller's bootstrap logic, not after.
- **A version-discovery endpoint on terdut-server** (e.g. `GET /api/version`).
Neither this operator nor terdut-tui has one today — both independently
detect capability by probing specific routes (terdut-tui via `GET
/api/teams` 404-checking; this operator would otherwise need to invent
its own equivalent probe). An unattended reconciler is more exposed to a
silent breaking API change than an interactive TUI a human is watching;
raising this alongside the service-account request rather than inventing
another route-probe here.
- CloudNativePG support — same `spec.database` shape as Zalando should
extend to it, but the concrete field/Secret-naming conventions need their
own look.
- Cross-namespace `teamRef` on the child CRDs (`TerdutEscalationRule`,
`TerdutDeadmanSwitch`, `TerdutAlertSource`) — only `TerdutTeam.serverRef`
crosses namespaces in v1 (§2, §4.2, §4.6); these stay same-namespace as
their `TerdutTeam` until a real need for splitting them out shows up.
- Narrower-than-namespace RBAC for the §4.5 webhook Secret (Kubernetes RBAC
has no owner-scoped grant below the namespace itself, per §9) — revisit
if the widened per-tenant-namespace `Secret` access proves too broad in
practice.
- A mid-life `spec.teamRef` change on a child CRD (`TerdutEscalationRule`,
`TerdutDeadmanSwitch`, `TerdutAlertSource`) isn't specially detected —
noticed while grounding Stage 4 against source, not newly introduced by
it: all three controllers always resolve `spec.teamRef` fresh every
reconcile and act against whatever `TerdutTeam` that currently names,
trusting the server-side id already stored in `status` remains valid
there. There's no server-side verb that could move an existing
integration/policy/switch to a different team in place regardless, so
retargeting one onto a live child isn't a supported operation in v1 —
delete and recreate the CR instead.
- `TerdutServerSpec.OIDC` has no `trustEmail` field (nor `usernameClaim`,
`emailClaim`, `groupsClaim` — the "rather than being added here
speculatively" fields its own doc comment already names), unlike
`charts/terdut-server`'s own chart, which sets `oidc.trustEmail: true`
for the production install specifically because Authentik reports
`email_verified: false` and without it a user's first SSO sign-in
creates a second, empty account instead of linking to their existing
one (`terdut-server/README.md`'s own account of this). Found by
actually trying to stand up a second real `TerdutServer` against the
same Authentik provider (`terdut-demo`, `Ryuvia/charts#275`), not by
inspection: that install's OIDC config is otherwise a straight copy of
production's and runs with terdut-server's own default (`trustEmail:
false`) regardless, since the CRD has nowhere to put the override.
Worth closing if a second real OIDC install becomes routine rather than
a one-off exercise.
- Gitops-managed team *membership* (see §4.2).
- CloudNativePG support, alongside the Zalando `postgresClusterRef` (same `spec.database` shape).
- Gitops-managed team membership (§4.2).
- Per-namespace RBAC with a restricted cache (§9).
- Automatic Deployment restart on upstream Postgres credential rotation.
- `spec.pod.priorityClassName`, pod-label passthrough beyond
`spec.pod.annotations`, and a HorizontalPodAutoscaler for `TerdutServer`
— all considered alongside §4.1's `spec.pod` and left out of that round.
An HPA no longer contradicts anything now that `spec.replicas` defaults
to 2 (terdut-server v0.36.0's advisory locks), but it is still a
separate, not-yet-made decision: a fixed replica count has no scaling
metric, min/max bounds, or cooldown behaviour to get right, and nobody
has asked for it yet.
- Admission webhooks / CEL-only validation limits (e.g. verifying a
`teamRef` exists at admission time rather than surfacing it as a status
condition after the fact).
- `spec.pod.priorityClassName`, pod labels beyond annotations, and an HPA for `TerdutServer`.
- Admission webhooks beyond CEL (for example, checking a `teamRef` exists at admission time).
- A shared API types module or generated client for terdut-server, terdut-operator and terdut-tui.
- OLM packaging.