# Demo Every CRD this operator reconciles, wired into one working install: one `TerdutServer`, two `TerdutTeam`s (Platform and Payments), each carrying its own escalation ladder and dead man's switches, and a `TerdutAlertSource` per team, plus a script that fires synthetic Alertmanager webhooks at it so you can watch real incidents appear, escalate and resolve. This is a demo kit, not a reference deployment: `00-postgres.yaml` runs Postgres with `emptyDir` storage and a password committed in this directory. Throw the whole namespace away when you're done. **Want this fully automated instead of walking through it by hand?** `./run-demo.sh` does everything below itself, against a fresh (or already-set-up) `kind` cluster — creates the cluster, installs the operator, applies every CR here, creates `alice` as the first user, and fires a few alerts. `./run-demo.sh --help` for the knobs, `./run-demo.sh --teardown` to tear it back down. The rest of this file is the manual walkthrough it automates. It pulls the demo's images on the host and loads them into the kind node first, since the node may not trust the TLS chain to `git.ryuvia.com`. To redo just that step (after bumping a pin, or for a pod stuck in `ImagePullBackOff`), run `./run-demo.sh --load-images`; add `PULL_ALWAYS=true` to re-pull tags the host already has. `PRELOAD_IMAGES=false` skips it in a full run. ## Prerequisites - The operator and its CRDs installed and running (`make install deploy IMG=...`, or `charts/terdut-operator` — see this repo's own README.md/DESIGN.md), pointed at a cluster you're fine creating throwaway resources in. A `kind` cluster is the easy choice. - `kubectl`, `jq`, `curl` on your path. **Apply this into a namespace of its own.** Every object name in this directory is prefixed `terdut-operator-demo` specifically so applying it by mistake into some other namespace that already has unrelated objects doesn't collide with them -- but that only helps if this directory's own objects don't collide with *each other* across two applies. Applying it twice into two different namespaces is fine; applying it a second time into a namespace that already has something else named `terdut-demo` (a real install from following `terdut-operator`'s own repo along, say) is exactly the mistake this prefix exists to avoid, and it only works if you don't override these names yourself. ## Apply it ```sh kubectl create namespace terdut-operator-demo kubectl apply -n terdut-operator-demo -k . ``` Listed and applied in dependency order (server → team → alert source), but you don't have to preserve that order yourself: every controller here re-queues and waits rather than failing when a ref isn't resolvable yet (`kubectl describe` shows `Reason: WaitingForServer` / `WaitingForTeam` while that settles). Platform stays at `Reason: UnknownUser` until `alice` exists: its escalation ladder names her. Watch it converge: ```sh kubectl get terdutservers,terdutteams,terdutalertsources \ -n terdut-operator-demo ``` The operator's own Deployment template now carries a `wait-for-postgres` init container (same fix as `charts/terdut-server`'s chart as of v0.33.2), so `terdut-operator-demo`'s pod should come up clean even against this brand-new Postgres doing its very first boot — no `CrashLoopBackOff` expected here. Once `terdut-operator-demo`'s own `Ready` condition is `True`, everything downstream of it should settle within a reconcile interval or two. ## See the web UI The operator never creates any external exposure for a `TerdutServer` -- that's a permanent non-goal (DESIGN.md §1), not a missing feature, so this demo just reaches it the simplest way there is: ```sh kubectl -n terdut-operator-demo port-forward svc/terdut-operator-demo 8080:8080 ``` and open http://localhost:8080. See `../networking` for worked examples of exposing it for real (Gateway API, Istio, or a plain `Ingress`) instead. ### First login The operator authenticates with a key of its own (the `TerdutServer`'s `-operator-key` Secret, handed to the server as `TERDUT_OPERATOR_KEY`) and never creates a user. A person gets in the way anyone does on a fresh terdut-server: `/api/bootstrap` creates the first user, an administrator, while no user exists yet: ```sh curl -sS -X POST http://localhost:8080/api/bootstrap -H 'Content-Type: application/json' \ -d '{"username":"alice","email":"alice@example.com","password":"a-long-demo-password"}' ``` The response carries an API key, shown once. An administrator can manage any team, so add `alice` to both teams with it (`POST /api/teams/{teamID}/members`; `kubectl get terdutteam -o jsonpath='{.status.teamID}'` gives the ids), or just use the UI's team pages. `run-demo.sh` does exactly this for you. ## Fire some alerts In another terminal, with the port-forward above still running: ```sh export NAMESPACE=terdut-operator-demo ./fire-alerts.sh platform high-cpu ./fire-alerts.sh platform disk-full ./fire-alerts.sh payments pod-crash # Watch it open an incident, escalate per the escalation ladders in # 02/03-team-*.yaml, and show up on the Platform/Payments team's own incident list. ./fire-alerts.sh platform high-cpu resolve ``` `fire-alerts.sh -h` (or any bad argument) prints the full scenario list. Each `(team, scenario)` pair is one stable fingerprint, so firing the same one twice updates the same alert (a real re-fire) and `resolve` closes exactly that one. Set `CLUSTER` to send the alert as if it came from one of several clusters: ```sh CLUSTER=prod-eu ./fire-alerts.sh platform high-cpu CLUSTER=prod-us ./fire-alerts.sh platform high-cpu # a second incident, not a join CLUSTER=prod-eu ./fire-alerts.sh platform high-cpu resolve ``` It stands in for a Prometheus external label plus `cluster` in Alertmanager's `group_by` (terdut-server's README, "Several clusters, one team"): the web UI then shows the cluster chip on each incident and a cluster filter in the queue. `CLUSTER` is part of the fingerprint, so resolve with the same value you fired with. `./run-demo.sh` fires its alerts across `prod-eu` and `prod-us`. ### Dead man's switches The `deadmanSwitches` in `02-team-platform.yaml` / `03-team-payments.yaml` expect a heartbeat alert on a 15-minute timeout: ```sh ./fire-alerts.sh platform heartbeat ``` Keep sending that (e.g. a `watch -n 60`) and nothing happens — that's the point. Stop sending it and, 15 minutes after the last one, terdut-server opens a `critical` incident on its own, with no webhook involved: proof the switch is watching for silence, not for a signal. ## Tear down ```sh kubectl delete namespace terdut-operator-demo ``` The operator's finalizers delete each team on the server (with its escalation, switches and integrations) before this namespace's objects actually disappear — give it a few seconds past the `kubectl delete` returning. A team with open incidents is not deleted until they are resolved.