Ran a full kind end-to-end pass per ROADMAP.md's open item: real kind
cluster, real disposable Postgres, the real terdut-server v0.33.0 image,
the operator built into a real image and deployed as a real Pod (not
`go run` against the cluster -- that was tried first and correctly failed
on cluster DNS not resolving from outside the cluster network, which is
expected, not a bug).
Result: TerdutServer went Ready, the generated credentials Secret held a
real tdsa_-prefixed service-account key, and that key successfully
authenticated and exercised its real intended capability against the
actual server (GET/POST /api/teams -> 200/201) -- confirmed from
terdut-server's own access log, not just our side. Stage 1's actual goal
(ROADMAP.md) is proven, not just asserted.
Two real bugs surfaced that no envtest suite could have caught, since
envtest's client bypasses RBAC entirely:
- .dockerignore's `!**/*.go` doesn't work under podman (the scaffold's
own comment already named this exact gotcha, buildah/containers#6417,
and pointed at the fix) -- `docker build` was silently building from an
empty source tree ("package cmd/main.go is not in std") until this was
pinned down. Fixed by re-including cmd/api/internal by name, as that
comment suggested doing if this happened.
- The controller had no RBAC for events.k8s.io (the new events API
GetEventRecorder uses, unlike the deprecated GetEventRecorderFor) --
every Event emission failed server-side ("Server rejected event (will
not retry!)"), silently, since event-recording failure doesn't fail
reconciliation. Reconciliation itself was never affected, but DESIGN.md
§12's observability goal (every externally-visible action emits an
Event) silently wasn't being met in any real deployment. Added
+kubebuilder:rbac for events.k8s.io/events (create, patch); confirmed
fixed by restarting the operator and checking `kubectl describe
terdutserver` actually shows the Event afterward, not just that the log
line stopped.
Also noted, not fixed here (a different repo's bug): terdut-server's own
GET /api/me 500s for a service-account caller rather than a clean 4xx --
that endpoint assumes a human user in context. Worth a terdut-server
issue, not an operator concern.
paths, self-registration bootstrap)
Replaces the bring-your-own-only Stage 1 (commit 1be7cf2) wholesale, per
the redesign in the previous two commits: the operator creates every
server it manages, so self-registration (DESIGN.md §6) is the only
bootstrap path, and Deployment/Service/database management builds
together with it (ROADMAP.md Stage 1) rather than behind a separate
later stage.
Grounded in terdut-server's actual chart (charts/terdut-server/templates/
deployment.yaml, values.yaml), not reconstructed from DESIGN.md's
illustrative YAML alone -- env var names, the password-via-PGPASSWORD
convention, the Recreate deployment strategy, /healthz probes, and the
TERDUT_OPERATOR_MODE=true decision (always on here, unlike the chart's
default-off: every write this operator's own future controllers make
goes through a service account already) all match that source exactly.
- api/v1alpha1: full TerdutServerSpec (image, replicas, networking,
database, sweeper, deadman, notify, oidc, passwordLogin, allowedTeams).
spec.database is a oneOf (dsn xor postgresClusterRef) via CEL
XValidation. No spec.credentialsSecretRef -- removed entirely in the
prior redesign commit, not carried forward.
- internal/controller:
- terdutserver_deployment.go: Deployment + Service via CreateOrUpdate,
owned (OwnerReference), env built field-for-field against the chart.
- terdutserver_database.go: both §8 paths. The Zalando path resolves
the postgresql.acid.zalan.do CR by convention (database/role both
"terdut", matching every DESIGN.md example) and only ever confirms
its generated credentials Secret exists -- never reads the value,
same "wire a secretKeyRef, don't read it" posture the DSN path takes.
classifyClusterGetError is its own function specifically so the
CRD-not-installed case (meta.IsNoMatchError) is unit-testable without
a real client.
- terdutserver_bootstrap.go: self-registration, checkpointed against
both real crash windows (DESIGN.md §6 point 1) -- an admin-key
checkpoint Secret, and adopt-via-GET+mint-new-key on a 409 from
creating the service account. BootstrapStateLost is its own error
type so Reconcile can route it to a condition instead of an infinite
retry.
- terdutserver_controller.go: ties it together -- finalizer add, DB
resolution, Deployment/Service reconcile, wait for a ready replica,
bootstrap, Ready/Bootstrapped/DatabaseReady conditions. Finalizer on
delete only removes the generated Secrets: terdut-server's API can't
delete a user or service account, only revoke keys, so there's
nothing server-side to undo.
- internal/tdclient: added Bootstrap, CreateInstanceServiceAccount,
GetServiceAccountByName, CreateServiceAccountKey, matching
terdut-server's real handlers' request/response shapes (internal/api/
users.go, service_accounts.go in that repo) field-for-field.
- Tests: envtest suite covering the full DSN-path lifecycle end to end
(finalizer -> Deployment/Service -> simulated readiness -> real
bootstrap against an httptest.Server fake), the adopt-on-409 recovery
path, BootstrapStateLost, both Zalando outcomes (cluster not found;
cluster + Secret found -> real DSN -> Ready), and deletion. A minimal
test-only stub of the Zalando CRD (internal/controller/testdata) lets
envtest create fixture objects without a real postgres-operator
installed. 74.0%/44.7% coverage, 0 lint issues.
- Two things scoped down from §8's full ambition, called out in code and
ROADMAP.md rather than silently dropped: no live watch on the
Zalando-generated Secret for rotation (periodic resync notices
eventually, not immediately), no Gateway API HTTPRoute creation from
spec.networking (would add a new dependency; nothing about proving
bootstrap works depends on external ingress existing). Both are
near-term follow-ups.
Verified locally: make fmt lint test build all clean.
Follows DESIGN.md's redesign (previous commit): with no hand-deployed
server to prove the simpler CRDs against, there's no reason left to defer
TerdutServer's Deployment/Service/database management behind a separate
later stage. Stage 1 now covers TerdutServer's full lifecycle --
Deployment, Service, both Postgres paths from §8 at once (bring-your-own
DSN and Zalando, per the user's call, not sequenced), bootstrap,
credentials -- built together, since bootstrap only has something to
bootstrap once the Deployment exists.
Old Stage 5 (TerdutServer absorbs Deployment/Service/bootstrap) is gone,
folded into Stage 1. Old Stage 6 (installer chart + release) renumbers to
Stage 5. Stages 2-4 (TerdutTeam, EscalationRule+DeadmanSwitch,
AlertSource) are unchanged in content, renumbering only where old Stage 5
disappears from ahead of them.
Explicitly supersedes the Stage 1 shipped before this redesign (commit
1be7cf2): that TerdutServerSpec/Status/controller/tests implemented the
now-removed bring-your-own path and get replaced wholesale when Stage 1
is actually implemented next, not extended.
Removes the premise Stage 1's bring-your-own credential design was built
on. Confirmed with the user directly: this operator creates and owns
every TerdutServer it manages; there is no hand-deployed or chart-deployed
install it's expected to target or migrate.
- §1: states this explicitly -- the root the rest of this commit hangs off.
- §4.1: spec.credentialsSecretRef (bring-your-own input) removed entirely,
not kept as unused flexibility. status.credentialsSecretRef stays as
pure output.
- §6: self-registration is now the *only* bootstrap path, not one of two --
and, since it's now load-bearing rather than a fallback with an easy
escape hatch, closed the two real crash windows in it properly rather
than leaving them as theoretical gaps: a checkpoint Secret for the raw
admin key between /api/bootstrap and minting the service account, and
adopt-on-409 (§5's general rule) if a prior interrupted attempt already
got that far. A checkpoint lost after being used crosses into the same
fail-closed territory §5's webhook-Secret-loss rule already established
-- same recovery (delete and recreate), not a new, one-off workaround.
- §10: dropped the migrate-an-existing-install narrative and the
chart-Job-vs-operator bootstrap race question entirely -- both
presupposed an install the operator might adopt or race against, which
doesn't exist. Kept the installer-chart framing on its own.
- §13: dropped the now-stale "Helm chart migration execution" deferred item.
§8 (Postgres) needed no change -- it already described both the DSN and
Zalando paths as co-equal, full-design detail, with no sequencing between
them to remove.
Implements the narrowed Stage 1 scope from ROADMAP.md, against the
bootstrap-flow fix from DESIGN.md §4.1/§6 (the earlier self-registration
flow couldn't work unauthenticated against terdut-server's real
AuthMiddleware -- see that commit for the full trace).
- api/v1alpha1: TerdutServer with spec.endpoint + spec.credentialsSecretRef
+ spec.allowedTeams (image/replicas/networking/database deferred to
Stage 5, per DESIGN.md's own narrowing). SecretKeyRef has no namespace
field -- always the operator's own, by construction.
- internal/controller: TerdutServerReconciler implements exactly the
bring-your-own path -- adopt spec.credentialsSecretRef if the Secret
exists and has data under the given key, probe GET /api/version as a
reachability check, set Ready/Bootstrapped conditions accordingly.
Self-registration (the /api/bootstrap race) is not implemented; unset
spec.credentialsSecretRef reports Ready: False, reason:
CredentialsSecretRefRequired, not an attempt at a flow that would fail
unauthenticated anyway. No finalizer: this stage creates nothing
server-side and adopts rather than generates its Secret, so there's
nothing to clean up on delete yet.
- internal/tdclient: minimal terdut-server API client (Version only, the
one call this stage needs), styled after terdut-tui's own
internal/api/client.go per terdut/CLAUDE.md's mirroring convention.
- Tests: envtest suite covering all four not-ready paths plus the happy
path (fake terdut-server via httptest.Server, per DESIGN.md §11), and a
focused unit suite for tdclient. 75.6%/82.4% coverage.
- Event recording uses the new events.k8s.io/v1 recorder API
(mgr.GetEventRecorder), not the deprecated GetEventRecorderFor --
caught by golangci-lint's staticcheck before it shipped.
Verified locally: make fmt lint test build all clean, 0 lint issues, all
specs pass.
Stage 1's own setup (chart bootstraps before the CR exists) never
exercises the self-registration fallback, and the narrowed spec has
nowhere to put the username/email /api/bootstrap needs anyway. Matches
DESIGN.md's §6 rewrite (bring-your-own is now the primary path, not an
equal alternative).
It's always the operator's own namespace by construction now (§6), never
anything else, so there was nothing for the field to vary -- key varies
instead (fixed 'token' when self-generated, whatever a human chose when
adopted from spec.credentialsSecretRef).
Traced the actual flow against terdut-server's real source before writing
any Stage 1 controller code, rather than trusting this section's own prior
description of it:
- internal/api/middleware.go's AuthMiddleware hard-rejects with 401 any
request carrying neither a Bearer token nor a session cookie, before
handleListServiceAccounts' own (more permissive) internal check ever
runs. So "on 403, self-lookup via GET /api/service-accounts?name=" --
this section's described fallback -- cannot work unauthenticated; an
earlier draft of this section assumed otherwise.
- That only actually matters in the rare case where this TerdutServer's
own controller loses the /api/bootstrap race... except Stage 1's own
setup (ROADMAP.md) guarantees it loses every time: terdut-server is
deployed via its existing chart, which runs its own bootstrap Job,
before the TerdutServer CR or its controller exist at all. The
self-registration flow was never going to complete for the one scenario
Stage 1 actually exercises.
Fix: spec.credentialsSecretRef (§4.1), bring-your-own -- a human mints an
instance-scoped service account once, manually, with their own admin
session, and hands the controller that Secret directly. This is now the
primary, expected path; self-registration on a genuinely fresh install
(where this controller might actually win the race) stays as the
fallback it was always meant to be, not the only path.
Also corrected: this section's opening paragraph still said "v1-blocking,
not v1-shippable" pending SERVICE-ACCOUNTS.md landing -- confirmed shipped
(internal/api/service_accounts.go, migration 014) since Stage 0's work on
this repo; stale framing removed.
Confirmed via the actual job log (run 857, job 1931), not just the exit
code: setup-envtest fetches envtest-v1.37.0-linux-amd64.tar.gz from
github.com in ~4s now, where it previously TLS-handshake-timed-out every
time. golangci-lint's own git-clone-to-github.com (the confound from the
first fix attempt) also went through fine in the same run.
Stage 0 is now fully green end to end: fmt, lint, test (envtest included).
Clean result, isolated from lint's own unrelated github.com flakiness:
post-MTU-fix (Ryuvia/charts#272), `make test` alone hits the exact same
"TLS handshake timeout" fetching envtest-v1.37.0-linux-amd64.tar.gz as
before the fix. Byte-for-byte identical error. The dind sidecar's MTU
mismatch was real (measured 1450 vs 1500 per the other session's report)
but it was not (solely) the cause of this specific failure.
Separately and incidentally: golangci-lint's own custom-gcl build also
does a plain `git clone https://github.com/...` and that is now failing
too (2/2, ~2.5min hang then generic exit 128) where it briefly succeeded
in an earlier pre-fix run -- noted in the comment but not chased further
here; worth someone's attention if it keeps recurring, since it'll block
`lint` regardless of the envtest question.
lint is currently blocked by an unrelated github.com git-clone failure
(golangci-lint's custom-gcl build), which means the last two runs never
reached setup-envtest -- the step the act-runner MTU fix (Ryuvia/charts#272)
was meant to affect. Narrowing to `make test` alone to get a clean signal;
will revert to `make fmt lint test` right after.
Host-mode (previous commit) fails earlier and differently: "go: command not
found" -- the runner host has no Go, so that path is dead.
Checked Ryuvia/charts' act-runner/templates/networkpolicy.yaml directly
rather than assuming: it's the only NetworkPolicy in the cluster, and it is
explicitly deny-ingress only -- its own comment states egress is
deliberately untouched, "CI pulls from registries and package indexes that
are not enumerable here" (issue #128). So my earlier claim that this needs
"allowlisting github.com on the runner's NetworkPolicy" was wrong: there is
no in-repo egress rule governing this at all. Whatever blocks github.com
from the dind bridge is outside anything Ryuvia/charts or Ryuvia/k8s
expresses in a Kubernetes object -- back to container-based (matching every
other Go job in this org) as the known-good shape, with `make test` left
red on the envtest fetch until that's actually found.
Confirmed the hard way (run 852, attempt 2): setup-envtest v0.25 fetches the
envtest kube-apiserver/etcd tarball from github.com's release CDN, not the
legacy GCS kubebuilder-tools bucket (that bucket 403s now for any object --
no fallback there for k8s 1.37 either). github.com is unreachable from this
job's container the same way terdut-server's ci.yaml already documents for
get.helm.sh -- TLS handshake timeout.
Dropping `container:` on this job is the same fix terdut-server's `chart` job
already uses for that exact class of problem (it reaches get.helm.sh only by
running on the host). Unproven for a Go job specifically -- no workflow in
this org has run Go outside a container before, so this also bets the runner
host has Go installed. If it fails on a missing `go` instead of the envtest
fetch, that bet was wrong and the real fix is allowlisting github.com's
release CDN on the runner's NetworkPolicy instead (Ryuvia/charts or
Ryuvia/k8s, outside this repo).
- .gitea/workflows/ci.yaml: fmt/lint/test, same no-actions/checkout-and-manual-clone
shape as terdut-server's ci.yaml, and the same reasoning for why (Node/ES2022
incompatibility on the runner image). No chart/security jobs yet -- nothing for
either to check until Stage 6 / real controller code exists.
- CLAUDE.md: Checks + Release sections, matching the sibling repos' convention from
the workspace-level CLAUDE.md ("each repo has its own CLAUDE.md... read it before
working in that repo"). Release is explicitly marked not-wired-yet rather than
copying terdut-server's, since there's no chart to release against until Stage 6.
- ROADMAP.md: moved the .release.conf bullet out of Stage 0 (it names a HELM_CHART
this repo doesn't have yet) -- it was already duplicated into Stage 6, which is
where it actually belongs.
Stage 0 done: `make fmt lint test` verified green locally. Real open question the CI
workflow's comments flag rather than assume past: whether storage.googleapis.com
(envtest's binary source) is reachable from this Gitea runner's container network the
way proxy.golang.org is -- terdut-server's own ci.yaml notes get.helm.sh/github.com are
not. Only running the workflow for real will confirm; the comment names the fallback
(move the job out of `container:`, like terdut-server's chart job) if it isn't.
kubebuilder init --domain ryuvia.com --repo git.ryuvia.com/niklas/terdut-operator
(--license none, no per-file header boilerplate -- terdut-server's source carries
none either). Go 1.26.0/controller-runtime v0.25.0/controller-tools v0.22.0, whatever
the current kubebuilder CLI (v4.16.0) scaffolds -- not pinned back to terdut-server's
go 1.25.9, since this is a separate module with its own toolchain.
Verified locally: build, vet, fmt all clean; `make lint` (golangci-lint, fetched into
bin/) 0 issues; `make test` (controller-gen + setup-envtest, fetched into bin/,
downloads real envtest binaries from storage.googleapis.com) passes.
Dropped kubebuilder's default .github/workflows/* -- this org runs on Gitea, not
GitHub; ci.yaml (next commit) is the only CI this repo gets.
- Add .gitignore (build artifacts, editor swapfiles, envtest testbin).
- Remove the stray .DESIGN.md.swp that was sitting untracked in the repo.
- Carries the DESIGN.md §5/§9/§13 edits from the secret-loss discussion:
fail-closed (not self-healed) TerdutAlertSource webhook Secret loss, and
the corrected RBAC section (the webhook Secret lives in the CR's tenant
namespace, not the operator's own namespace as an earlier draft claimed).
Stages the operator's implementation: TerdutServer stays bootstrap/credentials-only
(no Deployment/Service takeover) until Stage 5, so every earlier stage targets a
hand-deployed terdut-server in a disposable dev namespace instead of forcing the
chart-migration decision (§10) up front.
/api/bootstrap is single-shot per install (gated on COUNT(*) FROM
users, confirmed against internal/api/users.go and the chart's
bootstrap-job.yaml), not per identity — the two-identity bootstrap
plan and the delete-Secret-to-rotate runbook this section described
don't work against that. Rewrites §6 points 1/5/6 around a dedicated,
repeatable service-account credential instead (proposed server-side in
terdut-server's new SERVICE-ACCOUNTS.md), notes in §9 that Secret
mirroring is RBAC-sound but still hands out a server-admin-equivalent
credential per consenting namespace, and flags in §10 that chart-vs-
operator bootstrap ownership blocks §6 and needs deciding first.
Updates §13 to mark the service-account type as v1-blocking rather
than a someday improvement, and adds a version-discovery endpoint to
the same list (both this operator and terdut-tui currently detect
server capability by route-probing).
Niklas Ye