Authenticate with a seeded operator key; fold escalation and switches into TerdutTeam
Credentials: the TerdutServer controller generates <name>-operator-key in the server's own namespace (owned by it) and hands it to the pods as TERDUT_OPERATOR_KEY; the server creates its instance-scoped account from it at every start. A replaced Secret rolls the pods. The bootstrap handshake, the checkpoint Secret, per-team service accounts and credentials Secrets, BootstrapStateLost and credentials.deletionPolicy are gone. CRDs: TerdutServer, TerdutTeam and TerdutAlertSource. TerdutEscalationRule and TerdutDeadmanSwitch become spec.escalation and spec.deadmanSwitches[] on the team (matched by name, extras removed); team invites are removed. A team is created under the identity <namespace>/<name> (external_id), so a retry, a lost status or a deleted team heal by repeating the same call, and a display name owned by another team is TeamNameTaken instead of an adoption. The server resolves escalation usernames (UnknownUser condition). OIDC claim names and trustEmail are spec fields. Fixes: query values are URL-escaped; every delete treats 404 as success; deleting a team no longer depends on allowedTeams consent; a switch or integration deleted on the server is recreated; unnamed switches take the CR's name. Cleanup: scaffold e2e test, AGENTS.md, devcontainer, unused config/ pieces and Client.Version() removed; DESIGN.md, README, ROADMAP and the demo (run-demo.sh, manifests) rewritten for the new design. Secret RBAC stays cluster-wide, now stated in DESIGN.md section 9. Claude-Session: https://claude.ai/code/session_016mBLURvJoMuUEr9cB2RpUN
This commit is contained in:
Niklas Ye
+20
-235
@@ -1,240 +1,25 @@
|
||||
# terdut-operator build roadmap
|
||||
# terdut-operator status and deferred work
|
||||
|
||||
This is the staging plan for implementing the operator against `DESIGN.md`'s
|
||||
settled decisions. It exists for the same reason `DESIGN.md` and
|
||||
`SERVICE-ACCOUNTS.md` do: so each stage starts from an agreed sequencing
|
||||
instead of re-litigating "what do we build first" mid-PR.
|
||||
The staged build plan this file used to hold (scaffolding, TerdutServer, TerdutTeam, the child
|
||||
kinds, the installer chart and first release) is done and shipped; the history is in git. The
|
||||
design it implemented is in `DESIGN.md`, and its revision section at the top is the current
|
||||
shape of credentials and the CRD catalog.
|
||||
|
||||
## Sequencing call this roadmap makes
|
||||
## Validation
|
||||
|
||||
**The operator creates and owns every `TerdutServer` it manages — it never
|
||||
adopts one deployed independently, by hand or by `charts/terdut-server`**
|
||||
(`DESIGN.md` §1). An earlier version of this roadmap staged `TerdutServer`'s
|
||||
Deployment/Service/bootstrap takeover separately (old Stage 5), behind a
|
||||
hand-deployed server the simpler CRDs could be proven against first.
|
||||
That staging existed only because a credential-less operator couldn't
|
||||
`/api/bootstrap` its way into a server something else had already
|
||||
bootstrapped (`DESIGN.md` §6's original gap). With no server to adopt at
|
||||
all, that split has nothing left to justify it: `TerdutServer` now builds
|
||||
its full lifecycle — Deployment, Service, database wiring, bootstrap,
|
||||
credentials — in one stage, Stage 1, since bootstrap only has something to
|
||||
bootstrap once the Deployment exists.
|
||||
There is no CI end-to-end job. The golden path (create every kind against a real terdut-server on
|
||||
`kind`, check the server's own API, delete, check it is gone) is a manual pass, using
|
||||
`examples/demo/run-demo.sh`. It has not been re-run since the credential and CRD redesign
|
||||
(2026-10): do that before the next release.
|
||||
|
||||
## Stage 0 — Scaffolding & CI
|
||||
## Deferred
|
||||
|
||||
- `go.mod` (`git.ryuvia.com/niklas/terdut-operator`) + Kubebuilder v4
|
||||
scaffold (`cmd/main.go`, `config/`, `Makefile`, `PROJECT`), matching
|
||||
terdut-server's Go toolchain and house style (§3). Kubebuilder's own
|
||||
scaffolded `Makefile` already wires `manifests`/`generate`
|
||||
(`controller-gen`) and `setup-envtest` into `test`, and `golangci-lint`
|
||||
into `lint`, all fetched on demand into `bin/` — no separate install
|
||||
step needed beyond what `make test`/`make lint` already do.
|
||||
- `.release.conf` deliberately **not** added yet: it names a `HELM_CHART`
|
||||
this repo doesn't have until Stage 5. Adding it now would either be a
|
||||
stub that lies about what's releasable or dead config nobody can run —
|
||||
it lands in Stage 5, alongside the chart it describes.
|
||||
- Gitea Actions CI calling `fmt lint test`, mirroring terdut-server's
|
||||
`ci.yaml` convention (its `CLAUDE.md`: "a green gate here and a green
|
||||
pipeline are the same code, not two descriptions of it") minus the
|
||||
`chart`/`security` jobs, which need a chart (Stage 5) and real controller
|
||||
code (Stage 1+) respectively to have anything to check.
|
||||
- Drop kubebuilder's default `.github/workflows/*` scaffold — this org
|
||||
runs on Gitea, not GitHub; `.gitea/workflows/ci.yaml` is the only CI this
|
||||
repo has.
|
||||
- Housekeeping: drop the stray `.DESIGN.md.swp` (leftover vim swapfile,
|
||||
shouldn't be committed); correct `DESIGN.md` §6/§13's "v1-blocking, not
|
||||
v1-shippable" language — the service-account feature it was blocking on
|
||||
has since shipped in terdut-server.
|
||||
|
||||
**Done when:** CI is green on an otherwise-empty scaffold.
|
||||
|
||||
## Stage 1 — `TerdutServer`, full lifecycle
|
||||
|
||||
Supersedes the Stage 1 shipped before this redesign (commit `1be7cf2`)
|
||||
outright — that `TerdutServerSpec`/`Status`/controller/tests implemented the
|
||||
now-removed bring-your-own path and get replaced wholesale, not extended.
|
||||
New commits build forward over the old ones; no git history rewrite.
|
||||
|
||||
- Full §4.1 spec: `image`, `replicas`, `networking`, `database`, `sweeper`,
|
||||
`deadman`, `notify`, `oidc`, `passwordLogin`, `allowedTeams`, all together
|
||||
— no narrowing, since bootstrap needs the Deployment it's narrowed away
|
||||
from in the version this replaces.
|
||||
- Controller manages the Deployment + Service, both Postgres paths from §8
|
||||
at once (bring-your-own DSN *and* the Zalando `postgres-operator`
|
||||
`postgresClusterRef` integration — not sequenced, per the user's call),
|
||||
and bootstrap/credentials per §6's self-registration flow: `/api/bootstrap`
|
||||
once the Deployment has a ready replica, checkpoint the admin key, mint
|
||||
the instance-scoped service account, generated credentials Secret in the
|
||||
operator's own namespace, `status.credentialsSecretRef`. Finalizer cleans
|
||||
up that Secret (and the checkpoint, if one's still there) on delete —
|
||||
there's no server-side row to clean up alongside it: terdut-server's API
|
||||
has no way to delete a user or a service account, only to revoke
|
||||
individual keys, so there's nothing to undo there regardless.
|
||||
- RBAC: read-only watch on `postgresql.acid.zalan.do`, degrading gracefully
|
||||
if that CRD isn't installed (§8, §9).
|
||||
- Shipped, scoped down from §8's full ambition in one way, called out in
|
||||
code rather than silently dropped: no live watch on the Zalando-
|
||||
generated credentials Secret for rotation (relies on the periodic resync
|
||||
to notice eventually, higher latency than a watch). A near-term
|
||||
follow-up, not deferred to a later stage.
|
||||
- External exposure (a Gateway API `HTTPRoute` from
|
||||
`spec.networking.hostname`) was originally sketched here too, as a
|
||||
second near-term follow-up alongside the one above. It's since become an
|
||||
explicit, permanent non-goal instead (DESIGN.md §1): the operator will
|
||||
never manage ingress/exposure for `TerdutServer` in any form. See
|
||||
`examples/networking` for how to do that yourself.
|
||||
- `envtest` covering Deployment/Service reconciliation and both database
|
||||
paths — the Zalando path needs that CRD's schema vendored into the test
|
||||
environment (there's no real `postgres-operator` controller in `envtest`,
|
||||
only the CRD shape to create fixture objects against) — plus the
|
||||
self-registration flow against an `httptest.Server` fake of
|
||||
`/api/bootstrap` and `/api/service-accounts` (§11), including the
|
||||
adopt-on-409 recovery path and the one fail-closed case
|
||||
(`BootstrapStateLost`), not just the happy path.
|
||||
- **Done, 2026-10-01**: a real `kind` end-to-end pass (bring-your-own DSN,
|
||||
real terdut-server `v0.33.0` image, operator built into a real image and
|
||||
deployed as a real Pod, not `go run` against the cluster). `TerdutServer`
|
||||
went `Ready`; the generated credential authenticated and exercised its
|
||||
real capability against the actual server
|
||||
(`GET`/`POST /api/teams` → `200`/`201`, confirmed from terdut-server's own
|
||||
access log). Caught two real bugs no `envtest` suite could have (its
|
||||
client bypasses RBAC): `.dockerignore`'s `!**/*.go` not working under
|
||||
podman, and missing RBAC for `events.k8s.io` (the new events API
|
||||
`GetEventRecorder` uses) — both fixed. The Zalando path can additionally
|
||||
be validated for real against the org's own cluster later, where
|
||||
`postgres-operator` already runs, rather than only in a disposable `kind`
|
||||
stand-in — not done in this pass.
|
||||
|
||||
## Stage 2 — `TerdutTeam`
|
||||
|
||||
- §4.2: `serverRef` resolution, real update-in-place (POST create / PUT
|
||||
rename / PUT oidc-groups), team-scoped service-account minting once
|
||||
`Ready` (§6 point 3), finalizer that DELETEs the team server-side and its
|
||||
credential Secret.
|
||||
- First place the "every child resolves its own `teamRef` →
|
||||
`TerdutTeam.status`, never chains up to `TerdutServer`" pattern (§5) gets
|
||||
proven end to end.
|
||||
|
||||
## Stage 3 — `TerdutEscalationRule` + `TerdutDeadmanSwitch`
|
||||
|
||||
- Built together: both stay same-namespace-as-their-`TerdutTeam` (§1), so
|
||||
neither exercises cross-namespace complexity, but together they cover the
|
||||
two different reconciliation shapes §5's table calls out — whole-policy
|
||||
PUT-upsert for the escalation policy (no separate create step at all),
|
||||
real create/update-in-place/delete for the dead man's switch (PUT added
|
||||
in terdut-server `v0.33.0` specifically for this operator) — against the
|
||||
same shared create/finalizer/resync scaffolding Stage 2 already built.
|
||||
- New shared `resolveTeamAndClient` helper (`childref.go`) implements §5's
|
||||
"every child resolves its own `teamRef` → `TerdutTeam.status`, never
|
||||
chains up to `TerdutServer`" rule once, for both controllers —
|
||||
`TerdutTeam.status.serverEndpoint`, added in this stage, is what makes
|
||||
that literally true rather than just a stated intent.
|
||||
- `TerdutEscalationRule` resolves each "user" target's username to a
|
||||
user_id via `GET /api/users` (confirmed open to any authenticated
|
||||
caller) and reports `Ready: False, reason: UnknownUser` if it doesn't
|
||||
resolve. No `DELETE` exists for this resource, so its delete path `PUT`s
|
||||
an empty policy as the closest available undo.
|
||||
- `TerdutDeadmanSwitch` has no unique-name constraint server-side, so its
|
||||
idempotent-create is `GET`-list-and-match-by-name rather than
|
||||
adopt-on-409 (unlike every other resource in this operator).
|
||||
- **Done, 2026-10-01**: `envtest` coverage for both controllers' happy
|
||||
path, `TeamRefNotFound`/`WaitingForTeam`, `UnknownUser`, list-and-match
|
||||
adoption, update-in-place on spec drift, and deletion. `make fmt lint
|
||||
test build` all clean; `internal/controller` envtest coverage
|
||||
50.5% → 71.7%. No `kind` e2e pass for this stage — Stage 1's already
|
||||
proved the real-cluster mechanics (RBAC, image, bootstrap) these two
|
||||
controllers reuse unchanged, and neither introduces a new mechanism that
|
||||
pass would exercise differently (same reasoning Stage 2 used to skip
|
||||
one).
|
||||
|
||||
## Stage 4 — `TerdutAlertSource`
|
||||
|
||||
- Last of the children on purpose: it has the subtlest failure mode of the
|
||||
four. Covers webhook Secret generation/ownership (§4.5, §7), the
|
||||
`WebhookSecretLost` fail-closed condition + `Warning` event (§5, added
|
||||
2026-09-30), and the kind-change delete-and-recreate rotation path — all
|
||||
easier to get right with the other three controllers' patterns already
|
||||
in place to build on.
|
||||
- Idempotent-create here is neither adopt-on-409 (Team/service-account) nor
|
||||
list-and-match-by-name (`TerdutDeadmanSwitch`): terdut-server shows the
|
||||
webhook key exactly once, at creation, and never again, so no server-side
|
||||
lookup could ever recover it after a crash. The generated webhook Secret
|
||||
itself — written immediately after the POST, before `status` is ever
|
||||
touched — is this CR's only durable record that a create already
|
||||
succeeded; found on a later reconcile with `status.integrationID` still
|
||||
unset, it's read back directly rather than POSTing again. Found missing
|
||||
with `status.integrationID` *set*, that's the already-designed
|
||||
`WebhookSecretLost` fail-closed case instead.
|
||||
- **Done, 2026-10-01**: `envtest` coverage for the happy path, rename
|
||||
(PATCH, no key rotation), a kind change (delete-and-recreate, new id and
|
||||
key), crash recovery between POST and the Secret write, `WebhookSecretLost`,
|
||||
`TeamRefNotFound`/`WaitingForTeam`, and deletion. `make fmt lint test
|
||||
build` all clean; `internal/controller` envtest coverage holds at 71.6%.
|
||||
No `kind` e2e pass for this stage, same reasoning as Stage 3 (reuses
|
||||
Stage 1's already-proven real-cluster mechanics unchanged).
|
||||
|
||||
## Stage 5 — Installer chart + real release
|
||||
|
||||
- Package CRDs + the operator's own Deployment/RBAC into the installer
|
||||
chart §10 describes; wire `.release.conf`/release-vars the same way
|
||||
terdut-server does; run it through the `release` skill for a real first
|
||||
cut.
|
||||
- Full `kind` end-to-end test per §11: create `TerdutServer` → `TerdutTeam`
|
||||
→ one of each child kind → verify via terdut-server's own API that each
|
||||
object exists with the right shape → delete the CR → verify the
|
||||
server-side object is gone.
|
||||
- Chart built via kubebuilder's own `helm/v2-alpha` plugin from `config/`'s
|
||||
kustomize output (`charts/terdut-operator`), not hand-rolled -- CRDs +
|
||||
manager Deployment/RBAC come from the same markers/manifests every other
|
||||
stage already generates, so there's exactly one source of truth for
|
||||
them. Hand-added on top: the optional `terdutServer` values block (§10's
|
||||
"helm install and get a server" path), `.release.conf`, and the
|
||||
`release-vars`/`helm-lint`/`push`/`helm-package`/`helm-push`/`release`
|
||||
Makefile targets `.gitea/workflows/release.yaml` calls, mirroring
|
||||
terdut-server's own shape end to end (same registry/namespace
|
||||
convention, same multi-arch buildx push, same trivy/govulncheck/gitleaks
|
||||
scans). Also fixed while wiring this: the Dockerfile's builder stage
|
||||
didn't pin `--platform=$BUILDPLATFORM`, which would have made a
|
||||
multi-arch release build fail outright on this org's runners (no binfmt
|
||||
registration) -- caught before it ever shipped, not discovered mid-release;
|
||||
and govulncheck surfaced one real, reachable finding (`google.golang.org/grpc`
|
||||
v1.82.1, transitive via controller-runtime's otel exporter), fixed by
|
||||
bumping to v1.83.1.
|
||||
- **Done, 2026-10-01**: the full golden-path pass above, run for real
|
||||
against a `kind` cluster, installed via `helm install` (not raw
|
||||
kustomize/kubectl apply -- the first time the chart itself, not just
|
||||
`config/`, was exercised): `TerdutServer` (real terdut-server `v0.33.0`
|
||||
image, bring-your-own DSN against a throwaway in-cluster Postgres) →
|
||||
`TerdutTeam` → one `TerdutEscalationRule` + `TerdutDeadmanSwitch` +
|
||||
`TerdutAlertSource`, each confirmed `Ready` and then confirmed a second
|
||||
way, independent of the operator's own status: a `curl` pod inside the
|
||||
cluster, authenticated with the generated team credential, hit
|
||||
terdut-server's real API directly (`GET /api/teams/{id}/escalation`,
|
||||
`.../deadman/switches`, `.../integrations`) and got back exactly the
|
||||
policy/switch/integration each spec declared. Deleting every CR in
|
||||
reverse order was verified the same way: the escalation policy came back
|
||||
empty (its only available "undo"), the switch and the integration were
|
||||
both gone from their list endpoints, the team no longer resolved by
|
||||
name, and the Deployment/Service/every generated Secret were gone from
|
||||
the cluster. No new bugs found this pass -- Stage 1's own kind e2e pass
|
||||
already caught the two issues (`events.k8s.io` RBAC, the podman
|
||||
`.dockerignore` fix) a real cluster catches and `envtest` can't, and
|
||||
nothing since has touched that surface.
|
||||
- Not done in this pass, deliberately: an actual tagged release. `make
|
||||
release-vars`/`helm-lint`/`push`/`helm-package`/`helm-push` all work
|
||||
locally and `.gitea/workflows/release.yaml` is wired, but
|
||||
`release-preflight` found there is no `terdut-operator/` entry under
|
||||
`Ryuvia/charts` yet to bump -- every other onboarded repo had that
|
||||
one-time wrapper-chart bootstrap done for it before its own first
|
||||
release, and this one doesn't, since deploying this operator for real is
|
||||
a decision for whoever runs the cluster, not a side effect of finishing
|
||||
this stage. Cutting the first real release (and creating that wrapper
|
||||
entry) is therefore the next action, not yet taken.
|
||||
|
||||
## Deferred (§13, unchanged by this roadmap)
|
||||
|
||||
Cross-namespace `allowedTeams` exercised against a real second namespace,
|
||||
CloudNativePG support, narrower-than-namespace Secret RBAC for the webhook
|
||||
Secret, gitops-managed team membership, automatic Deployment restart on
|
||||
upstream Postgres credential rotation, admission webhooks/CEL-only
|
||||
validation limits, OLM packaging.
|
||||
- Narrower Secret RBAC: per-namespace Roles and a restricted cache (today a ClusterRole with
|
||||
Secret access cluster-wide, DESIGN.md §9).
|
||||
- CloudNativePG support alongside the Zalando `postgresClusterRef`.
|
||||
- Gitops-managed team membership.
|
||||
- Automatic Deployment restart on upstream Postgres credential rotation.
|
||||
- Admission webhooks beyond CEL validation.
|
||||
- OLM packaging.
|
||||
- A shared API types module (or generated client) between terdut-server, terdut-operator and
|
||||
terdut-tui, so contract drift is a compile error and not a manual mirror.
|
||||
|
||||
Reference in New Issue
Block a user