Give each team its own dead man's switches, and the UI a team to show
The rest of #4. Two halves that belong together because they are the same sentence from opposite ends: a team decides which of its alerts are heartbeats, and the UI has to be able to say which team it is talking about. Switches were three environment variables, which made them one setting for the whole install. That was the last piece of the alerting path a team could not control: it could take its own alerts on its own key and still not say which of them were heartbeats, or how long a silence had to last. They are a row per team now, edited by an owner through PUT /api/teams/{teamID}/deadman, and the sweeper runs each team against its own matchers, timeout and severity. The environment variables become the starting point rather than the setting. Every team without a configuration is seeded from them at startup, so an upgrade keeps watching exactly what it was watching, and SeedDeadmanConfigs never overwrites -- a redeploy must not put the environment's value back over an owner's edit. A team created later watches nothing until somebody says otherwise: inheriting an install-wide heartbeat would page a new team about a source it has never heard of, and a switch nobody chose is the kind that gets muted rather than fixed. A matcher string with no alertname in it is refused at the door instead of stored. Storing it would produce a switch that watches nothing silently, which is the exact failure the feature exists to prevent. NewRouter and Sweep lose their DeadmanConfig parameter -- there is no longer one answer to hand them. The type stays, because parsing a matcher string is still parsing a matcher string. The UI side: rows in the queue carry a team badge, the filter row gains a team chip per team, and "on call now" shows one card per team. All three appear only when the viewer is in more than one team -- otherwise they are the same word repeated down a list, which is noise rather than information, and the single-team install reads exactly as it did before teams existed. Verified against a live two-team server as well as in tests: the combined queue labelled by team, the team_id filter, a heartbeat that is a heartbeat in one team and an ordinary alert in another, and a new team's switches starting empty while the upgraded team keeps the environment's. Claude-Session: https://claude.ai/code/session_01RHPj4ggeFdEjKKfm4SHbD7
This commit is contained in:
@@ -161,7 +161,7 @@ somewhere to exec. The sidecar, the PVC and the `backupSidecar` values are all g
|
||||
| `TERDUT_DB_DSN` | — | **Required.** Postgres connection string, e.g. `postgres://terdut:secret@localhost:5432/terdut?sslmode=require` |
|
||||
| `TERDUT_ARCHIVE_AFTER` | `168h` (7d) | How long a resolved alert or incident stays in the default list before being auto-archived |
|
||||
| `TERDUT_STALE_AFTER` | `6h` | How long a firing alert may go without a refreshing webhook before it is treated as resolved — **must exceed your Alertmanager `repeat_interval`** |
|
||||
| `TERDUT_DEADMAN_MATCHERS` | `alertname=Watchdog` | Which alerts are [dead man's switches](#dead-mans-switch). `;` separates matchers, `,` the label conditions within one, `=` is exact equality. Every matcher must name an `alertname` |
|
||||
| `TERDUT_DEADMAN_MATCHERS` | `alertname=Watchdog` | The **default** matchers a team starts with — switches are per team now, and this seeds teams that have no configuration of their own. `;` separates matchers, `,` the label conditions within one, `=` is exact equality. Every matcher must name an `alertname` |
|
||||
| `TERDUT_DEADMAN_TIMEOUT` | `15m` | How long a heartbeat may go unheard before its switch is declared dead — **must be shorter than the `repeat_interval` of the route carrying it**. `0` disables dead man's switch handling |
|
||||
| `TERDUT_DEADMAN_SEVERITY` | `critical` | Severity a dead man's switch incident opens at |
|
||||
| `TERDUT_NTFY_URL` | — | ntfy server to publish push notifications to. Empty disables notifications entirely |
|
||||
@@ -378,11 +378,23 @@ kube-prometheus-stack already ships the alert for this. `Watchdog` is
|
||||
nothing unless something downstream notices it stop. That is what
|
||||
`TERDUT_DEADMAN_MATCHERS` defaults to.
|
||||
|
||||
**Switches belong to a team**, which decides which of its own alerts are
|
||||
heartbeats and how long a silence has to last. An owner sets them through
|
||||
`PUT /api/teams/{teamID}/deadman`; a missed heartbeat opens an incident in the
|
||||
team whose integration received it.
|
||||
|
||||
The environment variables are the starting point, not the setting: at startup
|
||||
every team **without** a configuration of its own is given one from them, and an
|
||||
owner's later edit is never overwritten by a redeploy. A team created after
|
||||
that starts watching nothing until its owner says otherwise — inheriting an
|
||||
install-wide heartbeat would page a new team about a source it has never heard
|
||||
of.
|
||||
|
||||
A matcher is a set of exact label conditions, one of which must be the
|
||||
`alertname`:
|
||||
`alertname`, in the same format the environment variable uses:
|
||||
|
||||
```
|
||||
TERDUT_DEADMAN_MATCHERS="alertname=Watchdog,cluster=prod; alertname=EdgeHeartbeat"
|
||||
alertname=Watchdog,cluster=prod; alertname=EdgeHeartbeat
|
||||
```
|
||||
|
||||
**The unit of monitoring is the fingerprint, not the alert name.** Two clusters
|
||||
@@ -523,6 +535,8 @@ incident. Move senders to a key and it goes away.
|
||||
| `GET` | `/api/teams/{teamID}/integrations` | member | List integrations. Never returns keys |
|
||||
| `POST` | `/api/teams/{teamID}/integrations` | **owner** | Mint an integration `{"name","kind"}` — key and URL shown once |
|
||||
| `DELETE` | `/api/teams/{teamID}/integrations/{integrationID}` | **owner** | Revoke an integration |
|
||||
| `GET` | `/api/teams/{teamID}/deadman` | member | The team's [dead man's switch](#dead-mans-switch) configuration `{matchers, timeout_seconds, severity}` |
|
||||
| `PUT` | `/api/teams/{teamID}/deadman` | **owner** | Replace it. `400` when no matcher names an `alertname`, because a switch that silently watches nothing is the failure this feature exists to prevent |
|
||||
|
||||
### Notifications
|
||||
|
||||
@@ -764,6 +778,11 @@ What changes, and will need attention:
|
||||
fingerprint, the same Alertmanager groupKey, and put somebody on call on the
|
||||
same date.
|
||||
|
||||
**Dead man's switches moved too.** `TERDUT_DEADMAN_MATCHERS`, `_TIMEOUT` and
|
||||
`_SEVERITY` are no longer the setting; they are the default each existing team
|
||||
is seeded with at startup, after which an owner edits them per team through
|
||||
`PUT /api/teams/{teamID}/deadman` and a redeploy never overwrites that.
|
||||
|
||||
Nothing else about an incident changes, and incidents never move between teams:
|
||||
an alert belongs to whichever team's key it arrived on.
|
||||
|
||||
|
||||
Reference in New Issue
Block a user