List dead man's switches with their status on Team -> Switches

The page was a bare form: it did not say which switches existed or
whether they were alive. It now lists them, each with a Healthy, Dead or
Dormant badge, when its heartbeat was last heard and when it last opened
an incident (linked while that incident is open). A matcher that several
clusters satisfy is broken down per cluster, since a live cluster must
not hide a dead one. The form moved into a "New switch" sheet, and each
row has a Remove with a confirm.

That needed a switch to be a thing, so switches are rows now
(migration 009) with their own name, matcher, timeout and severity,
instead of one string with one team-wide timeout in deadman_configs.
Existing configuration is split into one row per matcher; a team whose
timeout was zero simply has none. The sweeper and the status endpoint
share one death rule (deadmanAlert.dead), so the page cannot disagree
with the pager. Incident group keys are unchanged, so incidents that
are open across the upgrade keep working.

The environment defaults (TERDUT_DEADMAN_*) are seeded into teams once
per install, recorded in settings, so a team that deletes its last
switch does not get it back on the next restart. Installs that already
had per-team rows are marked as seeded by the migration.

Removing a switch stops the watching but leaves an incident it already
opened open until someone resolves it.

API: GET/PUT /api/teams/{id}/deadman are replaced by
GET/POST /deadman/switches and DELETE /deadman/switches/{switchID}.
terdut-tui does not call them, so nothing to mirror there.
This commit is contained in:
Niklas Ye
2026-09-25 23:40:55 +02:00
parent 3ee8583f6f
commit f3918b863c
11 changed files with 868 additions and 313 deletions
+23 -14
View File
@@ -513,19 +513,27 @@ nothing unless something downstream notices it stop. That is what
`TERDUT_DEADMAN_MATCHERS` defaults to.
**Switches belong to a team**, which decides which of its own alerts are
heartbeats and how long a silence has to last. An owner sets them through
`PUT /api/teams/{teamID}/deadman`; a missed heartbeat opens an incident in the
team whose integration received it.
heartbeats and how long a silence has to last. Each **switch** is a row of its
own — a name, one matcher, a timeout and a severity — so switches in one team
can have different deadlines. An owner adds and removes them on **Team →
Switches**, which lists each with a status (**healthy**, **dead**, or
**dormant** until its first heartbeat), when it was last heard from, and when it
last opened an incident; a matcher that several clusters satisfy is broken down
per cluster. The API is `POST`/`DELETE /api/teams/{teamID}/deadman/switches`. A
missed heartbeat opens an incident in the team whose integration received it.
Removing a switch stops the watching; an incident it already opened stays open
until somebody resolves it.
The environment variables are the starting point, not the setting: at startup
every team **without** a configuration of its own is given one from them, and an
owner's later edit is never overwritten by a redeploy. A team created after
that starts watching nothing until its owner says otherwise — inheriting an
install-wide heartbeat would page a new team about a source it has never heard
of.
The environment variables are the starting point, not the setting: the **first**
time the server starts, every team is given a switch per default matcher from
them, once. After that a team's switches are its own — an owner's edit or
deletion is never put back by a redeploy. A team created later starts watching
nothing until its owner says otherwise — inheriting an install-wide heartbeat
would page a new team about a source it has never heard of.
A matcher is a set of exact label conditions, one of which must be the
`alertname`, in the same format the environment variable uses:
`alertname`, in the format the environment variable uses (one matcher per switch; the
variable takes several, separated by `;`):
```
alertname=Watchdog,cluster=prod; alertname=EdgeHeartbeat
@@ -718,8 +726,9 @@ administrator who is not in the team gets the same `404` as anybody else.
| `DELETE` | `/api/teams/{teamID}/invites/{inviteID}` | **owner** | Revoke a link before it expires |
| `GET` | `/api/teams/{teamID}/escalation` | member | The team's [escalation ladder](#escalation) `{repeat_count, fallback_topic, levels[]}`. Empty levels means the team has none |
| `PUT` | `/api/teams/{teamID}/escalation` | **owner** | Replace it wholesale. `400` for a level with no targets or no timeout — a rung that pages nobody is a silence with a number on it |
| `GET` | `/api/teams/{teamID}/deadman` | member | The team's [dead man's switch](#dead-mans-switch) configuration `{matchers, timeout_seconds, severity}` |
| `PUT` | `/api/teams/{teamID}/deadman` | **owner** | Replace it. `400` when no matcher names an `alertname`, because a switch that silently watches nothing is the failure this feature exists to prevent |
| `GET` | `/api/teams/{teamID}/deadman/switches` | member | The team's [dead man's switches](#dead-mans-switch), each `{id, name, matcher, timeout_seconds, severity, status, last_heartbeat_at, last_triggered_at, open_incident_id, sources[]}`. `status` is `healthy`, `dead` or `dormant`; `sources` has one entry per heartbeat fingerprint. Empty when the team watches nothing |
| `POST` | `/api/teams/{teamID}/deadman/switches` | **owner** | Add one: `{name?, matcher, timeout_seconds, severity?}`. `400` when the matcher names no `alertname` or holds several, or the timeout is not positive — a switch that silently watches nothing is the failure this feature exists to prevent |
| `DELETE` | `/api/teams/{teamID}/deadman/switches/{switchID}` | **owner** | Stop watching. An incident it opened stays open. `404` for a switch of another team |
### Notifications
@@ -964,8 +973,8 @@ What changes, and will need attention:
**Dead man's switches moved too.** `TERDUT_DEADMAN_MATCHERS`, `_TIMEOUT` and
`_SEVERITY` are no longer the setting; they are the default each existing team
is seeded with at startup, after which an owner edits them per team through
`PUT /api/teams/{teamID}/deadman` and a redeploy never overwrites that.
is seeded with at startup, after which an owner manages them per team through
`/api/teams/{teamID}/deadman/switches` and a redeploy never overwrites that.
Nothing else about an incident changes, and incidents never move between teams:
an alert belongs to whichever team's key it arrived on.