terdut-operator/examples/demo/README.md

# Demo

Every CRD this operator reconciles, wired into one working install: one
`TerdutServer`, two `TerdutTeam`s (Platform and Payments), each carrying its
own escalation ladder and dead man's switches, and a `TerdutAlertSource` per
team, plus a script that fires synthetic Alertmanager webhooks at it so you can
watch real incidents appear, escalate and resolve.

This is a demo kit, not a reference deployment: `00-postgres.yaml` runs
Postgres with `emptyDir` storage and a password committed in this
directory. Throw the whole namespace away when you're done.

**Want this fully automated instead of walking through it by hand?**
`./run-demo.sh` does everything below itself, against a fresh (or
already-set-up) `kind` cluster — creates the cluster, installs the
operator, applies every CR here, creates `alice` as the first user, and fires a
few alerts. `./run-demo.sh --help` for the knobs, `./run-demo.sh
--teardown` to tear it back down. The rest of this file is the manual
walkthrough it automates.

It pulls the demo's images on the host and loads them into the kind node
first, since the node may not trust the TLS chain to `git.ryuvia.com`. To
redo just that step (after bumping a pin, or for a pod stuck in
`ImagePullBackOff`), run `./run-demo.sh --load-images`; add
`PULL_ALWAYS=true` to re-pull tags the host already has.
`PRELOAD_IMAGES=false` skips it in a full run.

## Prerequisites

- The operator and its CRDs installed and running (`make install
  deploy IMG=...`, or `charts/terdut-operator` — see this repo's own
  README.md/DESIGN.md), pointed at a cluster you're fine creating
  throwaway resources in. A `kind` cluster is the easy choice.
- `kubectl`, `jq`, `curl` on your path.

**Apply this into a namespace of its own.** Every object name in this
directory is prefixed `terdut-operator-demo` specifically so applying it
by mistake into some other namespace that already has unrelated objects
doesn't collide with them -- but that only helps if this directory's own
objects don't collide with *each other* across two applies. Applying it
twice into two different namespaces is fine; applying it a second time
into a namespace that already has something else named `terdut-demo` (a
real install from following `terdut-operator`'s own repo along, say) is
exactly the mistake this prefix exists to avoid, and it only works if you
don't override these names yourself.

## Apply it

```sh
kubectl create namespace terdut-operator-demo
kubectl apply -n terdut-operator-demo -k .
```

Listed and applied in dependency order (server → team → alert source), but
you don't have to preserve that order yourself: every controller here
re-queues and waits rather than failing when a ref isn't resolvable yet
(`kubectl describe` shows `Reason: WaitingForServer` / `WaitingForTeam` while
that settles). Platform stays at `Reason: UnknownUser` until `alice` exists:
its escalation ladder names her.

Watch it converge:

```sh
kubectl get terdutservers,terdutteams,terdutalertsources \
  -n terdut-operator-demo
```

The operator's own Deployment template now carries a `wait-for-postgres`
init container (same fix as `charts/terdut-server`'s chart as of v0.33.2),
so `terdut-operator-demo`'s pod should come up clean even against this brand-new
Postgres doing its very first boot — no `CrashLoopBackOff` expected here.

Once `terdut-operator-demo`'s own `Ready` condition is `True`, everything downstream
of it should settle within a reconcile interval or two.

## See the web UI

The operator never creates any external exposure for a `TerdutServer` --
that's a permanent non-goal (DESIGN.md §1), not a missing feature, so this
demo just reaches it the simplest way there is:

```sh
kubectl -n terdut-operator-demo port-forward svc/terdut-operator-demo 8080:8080
```

and open http://localhost:8080. See `../networking` for worked examples of
exposing it for real (Gateway API, Istio, or a plain `Ingress`) instead.

### First login

The operator authenticates with a key of its own (the `TerdutServer`'s
`<name>-operator-key` Secret, handed to the server as `TERDUT_OPERATOR_KEY`) and
never creates a user. A person gets in the way anyone does on a fresh
terdut-server: `/api/bootstrap` creates the first user, an administrator,
while no user exists yet:

```sh
curl -sS -X POST http://localhost:8080/api/bootstrap -H 'Content-Type: application/json' \
  -d '{"username":"alice","email":"alice@example.com","password":"a-long-demo-password"}'
```

The response carries an API key, shown once. An administrator can manage any
team, so add `alice` to both teams with it (`POST /api/teams/{teamID}/members`;
`kubectl get terdutteam -o jsonpath='{.status.teamID}'` gives the ids), or just
use the UI's team pages. `run-demo.sh` does exactly this for you.

## Fire some alerts

In another terminal, with the port-forward above still running:

```sh
export NAMESPACE=terdut-operator-demo

./fire-alerts.sh platform high-cpu
./fire-alerts.sh platform disk-full
./fire-alerts.sh payments pod-crash

# Watch it open an incident, escalate per the escalation ladders in
# 02/03-team-*.yaml, and show up on the Platform/Payments team's own incident list.

./fire-alerts.sh platform high-cpu resolve
```

`fire-alerts.sh -h` (or any bad argument) prints the full scenario list.
Each `(team, scenario)` pair is one stable fingerprint, so firing the same
one twice updates the same alert (a real re-fire) and `resolve` closes
exactly that one.

Set `CLUSTER` to send the alert as if it came from one of several clusters:

```sh
CLUSTER=prod-eu ./fire-alerts.sh platform high-cpu
CLUSTER=prod-us ./fire-alerts.sh platform high-cpu   # a second incident, not a join
CLUSTER=prod-eu ./fire-alerts.sh platform high-cpu resolve
```

It stands in for a Prometheus external label plus `cluster` in Alertmanager's
`group_by` (terdut-server's README, "Several clusters, one team"): the web UI
then shows the cluster chip on each incident and a cluster filter in the
queue. `CLUSTER` is part of the fingerprint, so resolve with the same value you
fired with. `./run-demo.sh` fires its alerts across `prod-eu` and `prod-us`.

### Dead man's switches

The `deadmanSwitches` in `02-team-platform.yaml` / `03-team-payments.yaml` expect a heartbeat
alert on a 15-minute timeout:

```sh
./fire-alerts.sh platform heartbeat
```

Keep sending that (e.g. a `watch -n 60`) and nothing happens — that's the
point. Stop sending it and, 15 minutes after the last one, terdut-server
opens a `critical` incident on its own, with no webhook involved: proof
the switch is watching for silence, not for a signal.

## Tear down

```sh
kubectl delete namespace terdut-operator-demo
```

The operator's finalizers delete each team on the server (with its
escalation, switches and integrations) before this namespace's objects
actually disappear — give it a few seconds past the `kubectl delete`
returning. A team with open incidents is not deleted until they are resolved.