Pulls the demo's images on the host and loads them into the kind node,
since the node may not trust the TLS chain to git.ryuvia.com. Tags come
from the chart's appVersion, 01-server.yaml and 00-postgres.yaml, so a
bumped pin is picked up. A full run does the same first unless
PRELOAD_IMAGES=false; PULL_ALWAYS=true re-pulls tags the host has.
Images are saved for the host platform and loaded as an archive: `kind
load docker-image` imports with --all-platforms, which fails for a
multi-arch image under docker's containerd image store.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Cosmetic: make helm-package passes --version and --app-version from the
tag, so these fields decide nothing about what is published. They are kept
in step so the tree heading for v0.7.1 does not say 0.7.0, as in
the 0.7.0 placeholder commit and 295dc3a.
Claude-Session: https://claude.ai/code/session_016mBLURvJoMuUEr9cB2RpUN
Same reason as 03df2bd: the kind demo shows the current web UI, which now
draws boolean alert metrics as an on/off timeline. The demo has no
Prometheus and sets no allowedHosts, so that stays off here; v0.50.0's
migration 003 runs on the demo's fresh database at startup.
Claude-Session: https://claude.ai/code/session_016mBLURvJoMuUEr9cB2RpUN
Cosmetic: make helm-package passes --version and --app-version from the tag,
so these fields decide nothing about what is published. They are kept in
step so the tree does not state an older number than the release it heads
for, as in 295dc3a.
Claude-Session: https://claude.ai/code/session_016mBLURvJoMuUEr9cB2RpUN
Same reason as 93d2a00: the kind demo shows the current web UI, which now
includes the metrics graph. The demo has no Prometheus, so it sets no
allowedHosts and the graph stays off.
Claude-Session: https://claude.ai/code/session_016mBLURvJoMuUEr9cB2RpUN
terdut-server v0.49.0 graphs the metric behind each alert on the incident
page, but only queries hosts listed in TERDUT_PROMETHEUS_ALLOWED_HOSTS,
because the URL comes out of a webhook payload. An install this operator
creates had no way to set it short of spec.pod.extraEnv.
spec.prometheus takes allowedHosts, preroll and timeout and maps them to
TERDUT_PROMETHEUS_ALLOWED_HOSTS, _PREROLL and _TIMEOUT. Nothing is emitted
unless allowedHosts is non-empty, so existing TerdutServers get the same
Deployment as before, and unset durations are left to the server's defaults.
There is no CRD default for the host: which Prometheus to trust is the
deployer's call, not something this operator should assume.
The CRD in config/ is regenerated and the chart's copy carries the same
schema; the chart's optional TerdutServer passes the block through.
Claude-Session: https://claude.ai/code/session_016mBLURvJoMuUEr9cB2RpUN
Cosmetic: make helm-package passes --version and --app-version from the tag,
so these fields decide nothing about what is published. They are kept in
step so the tree does not state an older number than the release it heads
for, as in 3036abf.
Claude-Session: https://claude.ai/code/session_016mBLURvJoMuUEr9cB2RpUN
govulncheck fails on GO-2026-6809, a crypto/x509 flaw in the standard
library that is fixed in 1.26.10. Move the workflows' golang image off 1.26.9
the way terdut-server did, so the scan and the shipped binary agree.
Claude-Session: https://claude.ai/code/session_016mBLURvJoMuUEr9cB2RpUN
v0.43.0 predates v0.47.2, which lets the instance service account list a
team's dead man's switches. Without it GET /api/teams/{id}/deadman/switches
404s for the operator's key, so a TerdutTeam never becomes ready against the
demo's server. The demo is meant to track the current release.
Claude-Session: https://claude.ai/code/session_016mBLURvJoMuUEr9cB2RpUN
Cosmetic: make helm-package passes --version and --app-version from the tag,
so these fields decide nothing about what is published. They are kept in
step so the tree does not state an older number than the release it heads
for.
Claude-Session: https://claude.ai/code/session_016mBLURvJoMuUEr9cB2RpUN
govulncheck reports five vulnerabilities (GO-2026-6603, 6610, 6611, 6612,
6617) in x/net v0.58.0 that the code reaches through net/http's HTTP/2
client. v0.60.0 fixes them; the other x/ modules move with it.
Claude-Session: https://claude.ai/code/session_016mBLURvJoMuUEr9cB2RpUN
govulncheck in the security job reports ten standard-library
vulnerabilities (net/http, mime/multipart, crypto/tls), all fixed in
1.26.9. The workflows pinned golang:1.26.6-bookworm.
Claude-Session: https://claude.ai/code/session_016mBLURvJoMuUEr9cB2RpUN
Credentials: the TerdutServer controller generates <name>-operator-key in
the server's own namespace (owned by it) and hands it to the pods as
TERDUT_OPERATOR_KEY; the server creates its instance-scoped account from it
at every start. A replaced Secret rolls the pods. The bootstrap handshake,
the checkpoint Secret, per-team service accounts and credentials Secrets,
BootstrapStateLost and credentials.deletionPolicy are gone.
CRDs: TerdutServer, TerdutTeam and TerdutAlertSource. TerdutEscalationRule
and TerdutDeadmanSwitch become spec.escalation and spec.deadmanSwitches[]
on the team (matched by name, extras removed); team invites are removed.
A team is created under the identity <namespace>/<name> (external_id), so a
retry, a lost status or a deleted team heal by repeating the same call, and
a display name owned by another team is TeamNameTaken instead of an
adoption. The server resolves escalation usernames (UnknownUser condition).
OIDC claim names and trustEmail are spec fields.
Fixes: query values are URL-escaped; every delete treats 404 as success;
deleting a team no longer depends on allowedTeams consent; a switch or
integration deleted on the server is recreated; unnamed switches take the
CR's name.
Cleanup: scaffold e2e test, AGENTS.md, devcontainer, unused config/ pieces
and Client.Version() removed; DESIGN.md, README, ROADMAP and the demo
(run-demo.sh, manifests) rewritten for the new design. Secret RBAC stays
cluster-wide, now stated in DESIGN.md section 9.
Claude-Session: https://claude.ai/code/session_016mBLURvJoMuUEr9cB2RpUN
Cosmetic: make helm-package passes --version and --app-version from the
tag, so these two fields decide nothing about what gets published. Still
done, as with d502485 (0.4.0) and 88172ad (0.3.0) before it, because a
tree heading for v0.5.0 that still says 0.4.0 tells its reader
something false.
Co-authored-by: Claude <noreply@anthropic.com>
Deleting a TerdutServer removed the credential Secrets but never touched the
database, so a recreated one found a server that was already bootstrapped and
no key for it: /api/bootstrap answered 403 and the operator stopped at
BootstrapStateLost, whose message and DESIGN.md both said "delete and
recreate". That is how the terdut-demo install on the cluster got stuck on
2026-10-03: Helm's cleanupOnFail deleted its TerdutServer after a failed
upgrade, the recreate found the bootstrapped database, and it sat at Ready:
False for five days until the database was reset by hand. Recreating cannot
fix it, because the finalizer clears Secrets and the database is not its to
reset, so "a fresh create starts clean" was only ever true when the database
went with it.
spec.credentials.deletionPolicy is Retain by default: the finalizer keeps the
instance credential Secret (Delete removes it, as before). The bootstrap
checkpoint is always removed. Before calling /api/bootstrap, reconcile now
looks for the retained Secret and asks the server for the operator's own
service account with its token. Accepted: adopt it and skip bootstrap.
Rejected with 401/403: the Secret outlived a database reset, so ignore it and
bootstrap like a first install, which replaces it. Any other error retries.
terdut-server's own tests already call that endpoint with an instance-scoped
key, so the permission is not new.
BootstrapStateLost is still the answer when the server is bootstrapped and no
credential it accepts survives, but its message now names the Secret to
restore and says that recreating does not clear the database. DESIGN.md §6
says the same, and the chart passes the setting through as
terdutServer.credentials.deletionPolicy.
A retained Secret of a TerdutServer that is gone for good is an orphan to
delete by hand. It is inert: nothing adopts it unless the server accepts the
token.
Checked on the kind demo with a locally built image against the real
terdut-server v0.43.0: deleting the TerdutServer kept the Secret, recreating it
reached Ready with the same credential (identical hash) and both TerdutTeams
came back Ready with their original ids. The controller specs cover adoption,
a rejected token after a reset, the bootstrapped-and-rejected failure, and
both deletion policies.
Co-authored-by: Claude <noreply@anthropic.com>
The demo pinned v0.36.0, the floor for replicas: 2, and so showed none of the
web UI since: the queue and incident layouts, the rota and escalation
pages, the theme toggle, and the cluster chip, filter and page titles
(v0.42.0-v0.43.0). It pins v0.43.0 now; the comment keeps v0.36.0 as the
floor, which is what the replicas setting actually depends on.
fire-alerts.sh takes an optional CLUSTER, standing in for a Prometheus
external label plus `cluster` in Alertmanager's group_by (terdut-server's
README, "Several clusters, one team"). It goes on the alert's labels and
groupLabels, and into the group key and the fingerprint, so the same alert in
two clusters is two incidents and not one. Unset, the payload is exactly what
it was. run-demo.sh fires its alerts across prod-eu and prod-us, high-cpu in
both, so the queue has a chip and a filter to show.
run-demo.sh also failed on its second run, though it says it is safe to
re-run: it expected HTTP 409 when alice already exists, but a spent invite
is answered with 403 "invite link is not usable" before the username is ever
checked. It now tries to log alice in first and skips the signup if that works.
Checked on the kind cluster: the server rolled to v0.43.0, every CR became
Ready and Adopted (server, both teams, both escalation rules, both dead man's
switches, both alert sources), and /api/incidents/clusters,
/api/incidents?cluster=prod-us and the incident titles came back as expected.
No operator code changed, so this needs no operator release.
Co-authored-by: Claude <noreply@anthropic.com>
wait_for_ready waited for every demo object at once, including
terdutescalationrule-platform, which names alice as a level-1 target --
but alice does not exist yet at that point in main(): she is created by
redeem_platform_invite, which ran after wait_for_ready. terdut-server
resolves every named username at reconcile time, not just when an
escalation actually fires, so that CR could never reach Ready before
alice did, and main() had no step in between to create her.
Split into wait_for_objects (the shared loop, now taking its object list
as arguments) plus two callers: wait_for_teams_ready, covering just the
server and the two teams redeem_platform_invite/join_payments_team
need, run before alice exists; wait_for_remaining_ready, covering the
escalation rules, dead man's switches and alert sources, run after.
Co-authored-by: Claude <noreply@anthropic.com>
replicas: 1 and tag: v0.34.0 were both correct when written, but the CRD's
own default moved to 2 in v0.4.0 (same release this demo is meant to show
off), and v0.34.0 predates v0.36.0's advisory locks that make a second
replica safe instead of racing the first. Left as-is, the demo would have
been the one place in this repo demonstrating the exact unsafe combination
the CRD's own doc comment warns against: more than one replica against an
image that doesn't guard the sweeper/notifier/migration-runner singletons.
replicas is now stated explicitly as 2 rather than dropped to pick up the
default silently, matching every other field in this file's own habit of
spelling out what it depends on. tag moves to v0.36.0 specifically -- the
first version where the lock landed -- with the comment keeping v0.34.0's
original reasoning (the service-account race fix) alongside the new one,
since v0.36.0 still carries that fix forward.
Co-authored-by: Claude <noreply@anthropic.com>
Mirrors charts/terdut-server's own deployment.yaml change: v0.36.0 put
the sweeper, the notifier and the migration runner each behind a
Postgres advisory lock, and gave incident creation its own conflict
resolution, so the Recreate strategy and replicas-stays-at-1 guidance
this controller carried (explicitly tracking that chart's comment)
are no longer load-bearing.
spec.replicas' +kubebuilder:default moves 1 -> 2 (config/crd/bases and
the chart's CRD template regenerated via controller-gen and
kubebuilder's helm plugin respectively, then hand-verified identical
to the generator's own output rather than trusting a bulk regen --
the plugin's --output-dir charts writes a fresh charts/chart scaffold
rather than updating charts/terdut-operator in place, so only the
diff was taken, not the whole tree). terdutserver_deployment.go's
same-value fallback (reachable only for a TerdutServer stored before
this default existed) moves with it, and its Strategy changes from
Recreate to RollingUpdate with no explicit maxUnavailable/maxSurge --
the 25%/25% default rounds to 0/1 at replicas: 2, already
zero-downtime.
DESIGN.md's three places asserting multi-replica isn't a supported
topology (the illustrative spec.replicas YAML, spec.pod.affinity's
rationale, and the HPA deferred-feature note) are corrected to match;
the HPA note now gives its own standing reason (no scaling metric or
bounds decided yet) rather than a contradiction that no longer holds.
The chart's optional terdutServer.replicas sample value moves 1 -> 2
alongside it. image.tag must be v0.36.0 or newer for any of this to
hold -- stated in both the CRD field's doc comment and the chart
value's comment, not enforced in code, same stance the chart takes on
every other version-coupled assumption.
Co-authored-by: Claude <noreply@anthropic.com>
Required for this demo to actually exercise the fix for
niklas/terdut-operator#3 -- v0.33.2 still has the authorization gap this
demo hit live (callerMayManageServiceAccount had no branch letting an
instance-scoped account adopt a team-scoped account's key).
The actual fix for the human-onboarding gap niklas/terdut-server#23 found --
not a terdut-server change at all. A team-scoped credential is already
owner-equivalent for POST/GET/DELETE /api/teams/{teamID}/invites
(requireTeamOwner's synthetic-membership mechanism, ratified not
accidental per that repo's SERVICE-ACCOUNTS.md), and invite redemption
bypasses signup_mode entirely -- this TerdutTeam controller just never
grew a feature to use either fact.
New spec.invite{enabled, role (member|owner, default member), maxUses
(1-100, default 1)} and status.inviteSecretRef. The Secret lives in the
TerdutTeam's OWN namespace, not the operator's: unlike
status.credentialsSecretRef (a durable, high-privilege credential, kept
operator-side per DESIGN.md §6), an invite is bounded and limited-use,
meant for this namespace's own human operators to read and hand out --
same precedent as TerdutAlertSource's status.webhookURLSecretRef, same-
namespace and OwnerReference'd so deleting the TerdutTeam garbage-collects
it automatically.
internal/controller/terdutteam_invite.go: mints on first spec.invite.enabled,
refreshes a day ahead of terdut-server's fixed 7-day TTL (reading the
Secret's own stored expiresAt, no extra server round-trip per reconcile),
revokes server-side and deletes the Secret when flipped back to false. A
lost invite Secret is silently re-minted rather than treated as
unrecoverable the way TerdutAlertSource's webhook key is -- nothing
external holds a durable dependency on one specific invite link staying
stable, it's read once by one human and handed out.
New tdclient.Invite/CreateInvite/RevokeInvite. New envtest coverage: mint
into the team's own namespace, refresh-before-expiry, revoke-on-disable
(internal/controller/terdutteam_controller_test.go's new "spec.invite"
Describe block), plus the fake server growing invite support
(terdutserver_controller_test.go) -- its handleTeamSubPath dispatcher was
split further (deadman switches into their own handleDeadmanSubPath,
matching the existing handleIntegrationSubPath precedent) to stay under
golangci-lint's gocyclo threshold with the new route added.
examples/demo updated to prove this end to end: 02-team-platform.yaml
turns on spec.invite; run-demo.sh's bootstrap_login/join_demo_teams (the
psql signup_mode flip + a direct team_members INSERT) are replaced by
redeem_platform_invite (reads status.inviteSecretRef, a real POST
/api/signup with the invite token) and join_payments_team (POST
/api/teams/{teamID}/members using Payments' own credential and alice's
user id resolved via GET /api/users, deliberately not given its own
spec.invite, so the demo shows both onboarding paths this feature
unlocks) -- zero kubectl exec/psql calls remain anywhere in the script.
README.md's "First login" section rewritten to match; it no longer
documents the admin-token curl call that 403s against current
terdut-server (niklas/terdut-server#23).
Depends on niklas/terdut-server#24 (the callerMayManageServiceAccount fix
for terdut-operator#3) being released before this is deployed for real --
not required to build or test this change itself, since the envtest fake
never modeled that authorization gap to begin with.
One script, two modes (run-demo.sh / run-demo.sh --teardown), that takes a
fresh empty kind cluster all the way to a working demo: creates the
cluster if needed, helm-installs this chart, applies every CRD kind in
this directory, waits for all nine objects to go Ready, then does what
the README's own first-login section cannot (see niklas/terdut-server#23
and niklas/terdut-operator#3 -- no service-account credential this
operator holds can ever call /api/admin/settings or POST /api/users) by
reaching into the demo's own throwaway Postgres directly: flips
signup_mode to open, signs alice up for real over the ordinary signup
endpoint, and joins her to both Platform and Payments (open signup always
creates its own new team, never joins an existing one by name, so
without this she'd have a working login that can't see a single incident
this demo fires -- /api/incidents and /api/alerts are both scoped to the
caller's own team memberships). Finishes by port-forwarding the service
and firing fire-alerts.sh at both teams, so a fresh run already has
visible incidents waiting in the web UI.
Verified end to end against a real kind cluster, including a second,
genuinely-fresh run that hit niklas/terdut-operator#3 live (terdutteam-
platform wedged in the 403 retry loop that issue describes) -- confirmed
the script itself fails cleanly on that (clear FAILED message, correct
exit code, no orphaned port-forward) rather than hanging or leaving a
mess, which is the most this script can do about a bug in the operator
it's driving.
make helm-package passes --version and --app-version from the tag, so
these fields decide nothing about what is published -- but a tree
heading for v0.2.0 that still says 0.1.2 tells its reader something
false. Same as 738c210 and 0e3118d before it.
spec.pod (api/v1alpha1/terdutserver_types.go): annotations, nodeSelector,
tolerations, affinity, topologySpreadConstraints, resources, pod and
container securityContext, serviceAccountName, extraEnv/extraEnvFrom,
extraVolumes/extraVolumeMounts, imagePullSecrets, and an optional
disruptionBudget. All direct corev1 passthrough -- no wrapper types buy
anything for any of these, matching how CloudNativePG and the Zalando
postgres-operator both expose the same knobs, and matching this repo's
own SweeperSpec precedent ("wrap only when a round-trip through a
different type buys something"). affinity is pure user-supplied
passthrough, not a toggle-plus-generated-default the way a multi-replica
cluster operator's pod anti-affinity usually is: this operator never
auto-generates one, since spec.replicas above 1 isn't a supported
topology (the sweeper/notifier singleton constraint). Considered and
declined for this round: priorityClassName, pod labels beyond
annotations, and a HorizontalPodAutoscaler -- the last of those would
directly contradict the singleton constraint above.
disruptionBudget is the one field here that isn't a plain PodTemplateSpec
knob: when set, the controller now reconciles a PodDisruptionBudget
selecting the TerdutServer's own pods (new terdutserver_pdb.go); clearing
it deletes any it previously created. New RBAC marker on
poddisruptionbudgets to match.
Driven by a public-release pass: looking past this project's own use case
at what a mature, general-purpose operator CRD exposes here (researched
against Zalando postgres-operator and CloudNativePG specifically), not
just the fields this install happened to need.
Separately, and found while answering a question about exposing
TerdutServer through Istio instead of Gateway API: spec.networking's own
doc comment quietly promised a Gateway API HTTPRoute this operator would
build eventually ("a near-term follow-up, not deferred"). That promise is
wrong for a public release -- an operator managing someone's ingress
mechanism for them is a worse default than not touching it at all, and a
surprise HTTPRoute appearing once that follow-up eventually landed would
have been exactly backwards for an Istio (or plain-Ingress, or
intentionally-unexposed) install. Made the non-goal explicit and
permanent instead (DESIGN.md §1), removed the dead `gatewayListener`
field it was the only consumer of (zero runtime call sites anywhere --
setting it already had no effect, so this is a schema cleanup, not a
behavior change), and corrected ROADMAP.md's framing. hostname/servicePort
stay: both are live (TERDUT_PUBLIC_URL, container/Service port), this
operator just never acts on hostname for exposure. Added
examples/networking (Gateway API HTTPRoute, Istio VirtualService) showing
how to expose the plain ClusterIP Service the operator already creates --
outside the operator itself, as illustrations, not as something
examples/demo applies automatically.
No new terdut-server version requirement: both changes are CRD/controller-
only, nothing about the API this operator's bootstrap flow depends on
changed.
1. Renamed every object this demo creates (TerdutServer, Postgres
Secret/Deployment/Service) from terdut-demo[-postgres] to
terdut-operator-demo[-postgres]. The user applied this kit into the
already-live "terdut-demo" namespace -- the real operator exercise
from earlier in this repo's own history -- and this demo's own
TerdutServer/Postgres objects shared that exact name. The TerdutServer
apply was rejected outright (DatabaseSpec's own CEL rule: adding dsn
while the live object already had postgresClusterRef violates "exactly
one of" and the API server refused it), and the real Postgres Service
was never touched (confirmed live: still Zalando's own spilo selector,
endpoint still the real StatefulSet pod) -- but the Postgres Secret and
Deployment, having no such protection, were created as brand new,
extra, crash-looping objects sitting right next to the real ones.
Prefixing every name this demo creates means a repeat of this exact
mistake no longer collides with anything, documented directly in
README.md now.
2. The actual crash itself, independent of (1): capabilities.drop: ["ALL"]
(added responding to a PodSecurity "restricted" warning) took
CAP_CHOWN/CAP_FOWNER away from the root user postgres:17-alpine's own
entrypoint needs to chown/chmod the data directory before it drops
privileges itself -- confirmed in a real crashed pod's logs: `chmod:
/var/run/postgresql: Operation not permitted`. kubectl apply
--dry-run=server, which is as far as this got verified before, only
checks admission policy; it was never actually booted. Removed the
capability drop and verified for real this time: applied just
00-postgres.yaml alone into a disposable namespace, waited for the pod
to go Ready, read its logs ("database system is ready to accept
connections"), then deleted that namespace.
Exporter config logging may leak endpoint URLs in info logs
(otlptrace/otlptracegrpc/sdk, transitively through grpc's own otel
instrumentation -- all indirect in go.mod, nothing imports these by
name). govulncheck flagged it reachable through real call chains
(tdclient.Client.DeleteIntegration, cmd/main.go's own init), caught by
ci.yaml's security job while cutting v0.1.2 (run 897) -- same pattern as
terdut-server's own da48814 for grpc's CVE-2026-84445.
go.opentelemetry.io/otel, /metric, /sdk, /sdk/metric, /trace,
/exporters/otlp/otlptrace, /exporters/otlp/otlptrace/otlptracegrpc all
moved 1.44.0 -> 1.45.0 together, plus go-logr/logr's own patch bump and
proto/otlp + genproto that go mod tidy pulled along with them. Verified:
go build, full test suite (71.7% coverage unchanged), golangci-lint,
helm-lint, and govulncheck itself now reporting zero reachable
vulnerabilities.
make helm-package passes --version and --app-version from the tag, so
these fields decide nothing about what is published -- but a tree heading
for v0.1.2 that still says 0.1.1 tells its reader something false. Same
as 0e3118d before it.
Stale since this repo's actual first release (v0.1.1, 2026-10-01) already
did it -- Ryuvia/charts/terdut-operator already exists and already pins
v0.1.1. Caught while about to repeat the same wrong assumption for this
release.
This operator's own Deployment template crash-looped a few times against
a from-scratch postgres-operator cluster still doing initdb and Patroni
leader election -- exactly the gap examples/demo's own README just
documented for it. terdut-server's ping-retry budget on startup
(internal/db/db.go in that repo) is sized for a much shorter, different
race (NetworkPolicy propagation, a few seconds), not genuine first-time
cluster creation, so it exhausted and the process exited before ever
binding its HTTP port -- a startupProbe cannot fix that, since the crash
happens before there is anything to probe. Same root cause and same fix
as charts/terdut-server's own deployment.yaml template as of that repo's
v0.33.2.
waitForPostgresContainer reuses dbEnv unchanged: both of
resolveDatabaseEnv's two paths (DSN, postgresClusterRef) put
TERDUT_DB_DSN first, so it's already exactly what pg_isready needs, and
pg_isready needs no credentials, so dbEnv's optional PGPASSWORD riding
along too is harmless rather than load-bearing.
Covered by the existing envtest suite (asserts on Containers[0], the main
container, unaffected by adding InitContainers) -- `make test` passes
unchanged, 71.7% coverage on internal/controller. Updates examples/demo's
own README, which no longer needs to warn about this.
A self-contained demo kit: a TerdutServer against a throwaway, bare
Postgres (bring-your-own DSN -- simplest path to stand up from nothing,
ROADMAP.md Stage 1's own note), two TerdutTeams, and each team's own
TerdutEscalationRule/TerdutDeadmanSwitch/TerdutAlertSource, so every CRD
this operator manages is exercised together rather than in isolation the
way config/samples' one-of-each already does.
fire-alerts.sh sends terdut-server's own amPayload/amAlert shape (read
from internal/api/alertmanager.go in that repo, not guessed from its
docs) at whichever TerdutAlertSource's generated webhook Secret it reads
the key out of -- high-cpu/disk-full/pod-crash scenarios to open and
resolve incidents, and a heartbeat scenario matching each team's dead
man's switch matcher, so stopping it demonstrates the switch noticing
silence on its own.
Verified server-side (kubectl apply --dry-run=server -k examples/demo)
against this operator's own dev cluster, which already has these CRDs
installed: every object validates. The one warning that cluster's
"restricted" PodSecurity raises (postgres:17-alpine's entrypoint needs to
start as root before it drops privileges itself) is noted inline in
00-postgres.yaml rather than worked around -- not a real production
pattern, and this Postgres exists only to be thrown away with the rest of
the demo namespace.
README.md walks through: applying, watching status, why a few early
CrashLoopBackOff restarts on terdut-demo itself are expected (this
operator's Deployment template has no wait-for-postgres init container
yet, unlike charts/terdut-server's chart as of v0.33.2), reaching the web
UI (port-forward -- spec.networking.hostname is accepted but nothing
creates an HTTPRoute for it yet), turning on open signup with the
operator's own generated admin token since the bootstrap-created account
has no password, firing alerts, and tearing down.
Found while standing up terdut-demo (Ryuvia/charts#275), a second real
TerdutServer against the same Authentik provider as production: OIDCSpec
has no trustEmail override, so a demo install copying production's OIDC
config otherwise verbatim silently runs with the wrong default for it.
Not fixed here -- recorded in §13 as a real, found gap, not a decision,
same as the mid-life teamRef note already there.
No operator-mode section existed here before -- terdut-server's own
README.md documents the feature, but this repo's design doc never
mentioned it. Added as §6 point 7, confirmed against source
(internal/api/middleware.go's OperatorModeBlock, router.go's opMode
wrapper): it blocks human writes to exactly the resources this
operator's CRDs manage (team identity, OIDC-group binding, escalation,
dead man's switches, integrations), and nothing else -- team membership,
invites, and the on-call schedule/rota stay human-editable regardless,
confirmed from the router rather than assumed from the README's prose
alone.
Cosmetic: `make helm-package` passes --version/--app-version from the
tag (da48814's own comment), so this field decides nothing about what
gets published. Still done so the tree doesn't say 0.1.0 while heading
for a v0.1.1 release. Cites da48814 (the fix this version actually is).
v0.1.0's own trivy scan found it: google.golang.org/grpc v1.83.1 (bumped
in b4ccdb0 to clear an unrelated, earlier grpc CVE the same scan would
otherwise have flagged) has its own high-severity denial-of-service CVE,
fixed one patch later at v1.83.2. Pure bad timing between the two fixes,
not a different root cause. govulncheck and the trivy scan both ran
clean against this version locally before tagging.
Chart (charts/terdut-operator) generated via kubebuilder's own helm/v2-alpha
plugin from config/'s kustomize output -- CRDs + manager Deployment/RBAC
come from the same markers every other stage already generates, one source
of truth. Hand-added on top: the optional terdutServer values block
(DESIGN.md §10's "helm install and get a server" path, off by default) and
the release-skill plumbing -- .release.conf, release-vars/helm-lint/push/
helm-package/helm-push/release Makefile targets, .gitea/workflows/release.yaml
(test -> image/chart -> scan-image) -- mirroring terdut-server's own shape
(registry/namespace convention, multi-arch buildx push, trivy/govulncheck/
gitleaks scans). ci.yaml gains security and chart jobs to match.
Two real issues caught while wiring this, fixed before either shipped:
- Dockerfile's builder stage didn't pin --platform=$BUILDPLATFORM, which
would have made a multi-arch release build fail outright on this org's
runners (no binfmt registration) -- same fix terdut-server's own
Dockerfile already needed for the same reason.
- govulncheck found one real, reachable finding: google.golang.org/grpc
v1.82.1 (transitive via controller-runtime's otel exporter), fixed by
bumping to v1.83.1.
Full golden-path kind e2e pass, this time through `helm install` rather than
raw kustomize: TerdutServer (real terdut-server v0.33.0 image) -> TerdutTeam
-> one of each child kind, each confirmed Ready and then independently
confirmed against terdut-server's own API from inside the cluster (not just
the operator's own status). Deleted every CR in reverse order and confirmed
server-side cleanup the same independent way for all three child kinds, the
team, and the server. No new bugs found -- Stage 1's own kind pass already
caught what a real cluster catches that envtest can't.
Also dropped the kubebuilder helm plugin's default .github/workflows/
scaffold, same as Stage 0 already did for the main scaffold: this org runs
on Gitea, not GitHub.
Not done here, deliberately: an actual tagged release. release-preflight
found no terdut-operator/ entry under Ryuvia/charts yet to bump -- that
one-time wrapper bootstrap is a decision about deploying this operator for
real, not a side effect of finishing this stage.
make fmt lint test helm-lint build all clean.
Covers webhook Secret generation/ownership (DESIGN.md §4.5, §7), the
WebhookSecretLost fail-closed condition, and the kind-change
delete-and-recreate rotation path.
Idempotent-create here is deliberately neither adopt-on-409
(Team/service-account) nor list-and-match-by-name (TerdutDeadmanSwitch):
terdut-server shows the webhook key exactly once, at creation, and never
again, so no server-side lookup could ever recover it after a crash.
Instead the generated webhook Secret itself -- written immediately after
the POST, before status is ever touched -- is this CR's only durable
record that a create already succeeded; found with status.integrationID
still unset on a later reconcile, it's read back directly rather than
POSTing a second, orphaned integration. Found missing with
status.integrationID *set* instead, that's the already-designed
WebhookSecretLost case: fail closed, not self-healed, since the key is
genuinely gone and recreating it would rotate a live webhook URL with no
spec change to explain why.
Renaming (PATCH) never touches the key, so it's applied unconditionally
every reconcile, same as the escalation policy's whole-policy PUT. A
spec.kind change is the one case with no in-place update verb at all:
DELETE the old integration, delete the stale webhook Secret, then run the
same create path fresh -- fires a Warning event since this breaks whatever
still sends to the old URL.
Also: fakeTerdutServer grows POST/PATCH/DELETE .../integrations routes
behind a new handleIntegrationSubPath, split out of handleTeamSubPath to
stay under gocyclo's threshold; three goconst-flagged test literals
("does-not-exist", "unready") and one unparam-flagged test helper
parameter (bootstrapReadyTerdutServer's always-"default" namespace) get
shared/removed now that a fourth same-shaped caller made the repetition
concrete enough for the linter to flag.
DESIGN.md §13 gains one honest gap found while grounding this stage, not
introduced by it: no child CRD specially detects a mid-life teamRef
change; all three always resolve spec.teamRef fresh and trust the
already-stored server-side id remains valid there.
make fmt lint test build all clean; internal/controller envtest coverage
holds at 71.6%.
Both child CRDs resolve their own teamRef -> TerdutTeam.status via the new
shared resolveTeamAndClient helper (childref.go), never chaining up to
TerdutServer (DESIGN.md §5) -- TerdutTeam.status.serverEndpoint, added in
this same stage, is what makes that literally true.
TerdutEscalationRule: one PUT /api/teams/{id}/escalation per reconcile
(an upsert server-side, confirmed against source), resolving each "user"
target's username to a user_id via GET /api/users first and reporting
Ready: False, reason: UnknownUser if it doesn't resolve. No DELETE exists
for this resource, so its delete path PUTs an empty policy as the closest
available undo.
TerdutDeadmanSwitch: real create/update-in-place/delete, using
terdut-server v0.33.0's PUT (added specifically for this operator). No
unique-name constraint server-side, so idempotent-create here is
GET-list-and-match-by-name rather than adopt-on-409.
Extends tdclient with User/GetUserByUsername, the escalation request types
+ SetEscalation, and DeadmanSwitch + its CRUD methods. Also folds
ConditionTeamReady into the single shared ConditionReady constant, since
both were literally "Ready" and Stage 3 would otherwise have needed a
third same-valued constant.
internal/controller/terdutserver_controller_test.go's fakeTerdutServer
grows GET /api/users, PUT .../escalation, and the full dead man's switch
collection/item routes, replacing the old parseTeamPath/handleTeamByID
pair with a more general parseTeamSubPath/handleTeamSubPath dispatcher
that still covers every existing Stage 1/2 route unchanged.
make fmt lint test build all clean; envtest coverage for
internal/controller: 50.5% -> 71.7%.
Three real findings, same discipline as Stages 1/2:
- Dead man's switches gained PUT update-in-place in terdut-server
v0.33.0 (internal/api/teams.go's handleUpdateTeamDeadman, whose own
doc comment names terdut-operator as the reason it was added) --
§5's table still described delete-and-recreate, written before that
landed. Also: no unique-name constraint server-side at all, so this
resource's idempotent-create step is GET-list-and-match-by-name, not
adopt-on-409 the way Team/service-accounts work.
- §4.3's username->user_id resolution needs an endpoint: GET /api/users,
confirmed open to any authenticated caller (router.go's own "readable
by anyone signed in"), so the team-scoped credential already in hand
is enough -- no new server-side capability needed here, unlike
TEAM-LOOKUP.md's gap.
- §5's "a child never needs to chain up to TerdutServer" claim wasn't
actually true as written -- a child still needs the server's URL to
make any call, and the only way to get one was reading TerdutServer
directly. Fixed at the root: TerdutTeam.status now carries
serverEndpoint too (resolved once, by TerdutTeam's own controller,
same reconcile as teamID/credentialsSecretRef), so the claim holds
literally and child controllers need no terdutservers RBAC at all.
Implements ROADMAP.md Stage 2 against terdut-server's now-real
GET /api/teams?name= (TEAM-LOOKUP.md, landed in terdut-server just
before this commit) -- without it, the adopt-on-409 pattern this
controller depends on for team creation had no server-side lookup to
call, the same gap TerdutServer's own bootstrap flow hit and fixed in
Stage 1.
- api/v1alpha1: TerdutTeamSpec per DESIGN.md §4.2 (serverRef, displayName,
oidc). status.credentialsSecretRef drops namespace for key, matching
the fix already applied to TerdutServer's.
- internal/controller:
- terdutteam_controller.go: resolves serverRef (same-namespace by
default; cross-namespace gated by the target TerdutServer's
spec.allowedTeams, DESIGN.md §4.6), waits for that TerdutServer to be
Bootstrapped (no cross-controller RPC -- reads its
status.credentialsSecretRef directly, DESIGN.md §5), creates the team
and mints its team-scoped credential using the TerdutServer's
instance-scoped one, then applies rename/oidc-groups with the
team-scoped credential every reconcile (both are idempotent PUTs of
the whole resource -- applied unconditionally rather than diffed
against a stored last-applied value, same "cheap because it's small"
reasoning §5 already gives the escalation policy's whole-policy PUT).
- terdutteam_allowedteams.go: the §4.6 consent check in isolation from
any client, unit-tested directly against hand-built inputs.
- terdutteam_bootstrap.go: create-or-adopt-on-409 for the team itself
(via TEAM-LOOKUP.md) and for its team-scoped service account (via the
same GET-by-name+mint-new-key pattern Stage 1 already uses for the
instance account).
- secrets.go: extracted TerdutServer's write/read-credential-Secret
helpers into free functions, now shared by both controllers rather
than duplicated.
- Finalizer deletes the team server-side (owner-gated, needs the
team-scoped credential -- confirmed against source that an
instance-scoped one does not satisfy requireTeamOwner, same finding
as TEAM-LOOKUP.md's) and cleans up its credentials Secret. A team
created but never fully reconciled to Ready (no team-scoped
credential ever minted) is left orphaned server-side on delete -- a
known, documented limitation (terdut-server has no delete path that
doesn't require owner-equivalent access), not a silent gap.
- internal/tdclient: Team type, CreateTeam, GetTeamByName, RenameTeam,
DeleteTeam, SetTeamOIDCGroups, CreateTeamServiceAccount -- matching
terdut-server's real handlers' shapes field-for-field, same as Stage
1's client additions.
- Tests: envtest covering the happy path, both not-ready reasons
(ServerRefNotFound, WaitingForServer), cross-namespace allow/deny
(default-closed and explicit All), both adopt-on-409 paths (team
itself, team-scoped service account), and deletion. Extended the shared
fakeTerdutServer (Stage 1) with team endpoints rather than writing a
second, separately-drifting fake. 72.8%/30.6% coverage, 0 lint issues.
Verified locally: make fmt lint test build all clean.
§5's Team row now cites GET /api/teams?name= (terdut-server's
TEAM-LOOKUP.md, landed today) -- without it the idempotent-create general
rule's claim that every resource here has a real lookup to adopt-on-409
through wasn't actually true for Team specifically, confirmed by tracing
it before writing any TerdutTeam code, same as Stage 1's bootstrap flow.
Also: TerdutTeam.status.credentialsSecretRef drops namespace for key,
matching the TerdutServer fix from Stage 1 -- same reasoning, missed
there originally.
Ran a full kind end-to-end pass per ROADMAP.md's open item: real kind
cluster, real disposable Postgres, the real terdut-server v0.33.0 image,
the operator built into a real image and deployed as a real Pod (not
`go run` against the cluster -- that was tried first and correctly failed
on cluster DNS not resolving from outside the cluster network, which is
expected, not a bug).
Result: TerdutServer went Ready, the generated credentials Secret held a
real tdsa_-prefixed service-account key, and that key successfully
authenticated and exercised its real intended capability against the
actual server (GET/POST /api/teams -> 200/201) -- confirmed from
terdut-server's own access log, not just our side. Stage 1's actual goal
(ROADMAP.md) is proven, not just asserted.
Two real bugs surfaced that no envtest suite could have caught, since
envtest's client bypasses RBAC entirely:
- .dockerignore's `!**/*.go` doesn't work under podman (the scaffold's
own comment already named this exact gotcha, buildah/containers#6417,
and pointed at the fix) -- `docker build` was silently building from an
empty source tree ("package cmd/main.go is not in std") until this was
pinned down. Fixed by re-including cmd/api/internal by name, as that
comment suggested doing if this happened.
- The controller had no RBAC for events.k8s.io (the new events API
GetEventRecorder uses, unlike the deprecated GetEventRecorderFor) --
every Event emission failed server-side ("Server rejected event (will
not retry!)"), silently, since event-recording failure doesn't fail
reconciliation. Reconciliation itself was never affected, but DESIGN.md
§12's observability goal (every externally-visible action emits an
Event) silently wasn't being met in any real deployment. Added
+kubebuilder:rbac for events.k8s.io/events (create, patch); confirmed
fixed by restarting the operator and checking `kubectl describe
terdutserver` actually shows the Event afterward, not just that the log
line stopped.
Also noted, not fixed here (a different repo's bug): terdut-server's own
GET /api/me 500s for a service-account caller rather than a clean 4xx --
that endpoint assumes a human user in context. Worth a terdut-server
issue, not an operator concern.
Niklas Ye
niklas