diff --git a/README.md b/README.md index 944b680..6d4173d 100644 --- a/README.md +++ b/README.md @@ -100,6 +100,9 @@ Set `backupSidecar.enabled=false` if you back the volume up some other way. | `TERDUT_DB_PATH` | `terdut.db` | Path to the SQLite database file | | `TERDUT_ARCHIVE_AFTER` | `168h` (7d) | How long a resolved alert or incident stays in the default list before being auto-archived | | `TERDUT_STALE_AFTER` | `6h` | How long a firing alert may go without a refreshing webhook before it is treated as resolved — **must exceed your Alertmanager `repeat_interval`** | +| `TERDUT_DEADMAN_MATCHERS` | `alertname=Watchdog` | Which alerts are [dead man's switches](#dead-mans-switch). `;` separates matchers, `,` the label conditions within one, `=` is exact equality. Every matcher must name an `alertname` | +| `TERDUT_DEADMAN_TIMEOUT` | `15m` | How long a heartbeat may go unheard before its switch is declared dead — **must be shorter than the `repeat_interval` of the route carrying it**. `0` disables dead man's switch handling | +| `TERDUT_DEADMAN_SEVERITY` | `critical` | Severity a dead man's switch incident opens at | | `TERDUT_NTFY_URL` | — | ntfy server to publish push notifications to. Empty disables notifications entirely | | `TERDUT_NTFY_TOKEN` | — | Bearer token for an access-controlled ntfy | | `TERDUT_NTFY_FALLBACK_TOPIC` | — | Topic used when nobody is on call | @@ -108,7 +111,11 @@ Set `backupSidecar.enabled=false` if you back the volume up some other way. Durations use Go syntax (`30m`, `12h`, `168h`). An unparseable value falls back to the default. -In the Helm chart the two sweeper durations are set via `sweeper.staleAfter` and `sweeper.archiveAfter`, and notifications via the `notify.*` values. +Note that `TERDUT_STALE_AFTER` and `TERDUT_DEADMAN_TIMEOUT` point in opposite directions. Staleness +is a generous grace period around a `repeat_interval` you do not control; a dead man's switch is a +deadline you set deliberately, and the heartbeat's route is configured to beat faster than it. + +In the Helm chart the two sweeper durations are set via `sweeper.staleAfter` and `sweeper.archiveAfter`, dead man's switches via the `deadman.*` values, and notifications via the `notify.*` values. --- @@ -129,6 +136,30 @@ route: The webhook endpoint requires no authentication. +If you use the [dead man's switch](#dead-mans-switch) — and the default configuration does — give +the heartbeat a route of its own, because the deadline is only as tight as the interval feeding it: + +```yaml +route: + receiver: terdut + repeat_interval: 4h + routes: + - matchers: [ 'alertname = "Watchdog"' ] + receiver: terdut + group_wait: 0s + group_interval: 1m + repeat_interval: 1m +``` + +That delivers a heartbeat every **2 minutes**, not every minute. Alertmanager only reconsiders a +group every `group_interval`, and at exactly one elapsed interval `repeat_interval` has not *quite* +passed, so the send slips to the next tick — equal values give 2×. Two minutes against the 15 minute +default is seven heartbeats per window, which is the point; use `group_interval: 30s` if you want +the numbers to mean what they say. + +kube-prometheus-stack users get the `Watchdog` alert (`expr: vector(1)`) for free; it just needs +routing to terdut rather than to `null`. + --- ## Alerts and incidents @@ -179,6 +210,10 @@ alert that never stopped firing. that group opens a *new* incident rather than reopening this one. If the alert underneath never stops firing, the incident stays closed — that is what resolving by hand asserts. +- **On recovery**, for a [dead man's switch](#dead-mans-switch) incident whose + heartbeat started arriving again (`"resolution_source": "recovered"`). These + incidents have no member alerts, so the automatic cascade above cannot reach + them. To quieten an incident you expect to come back, snooze it instead (`POST /api/incidents/{id}/snooze`). A snooze hides the incident from the default @@ -267,6 +302,72 @@ to distinguish them from a real Alertmanager resolve (`"alertmanager"`). An expiry cascades: once it leaves an incident with nothing firing under it, the incident resolves too, in the same sweep. +### Dead man's switch + +Everything above assumes alerts arrive. If Prometheus stops evaluating, or +Alertmanager cannot reach this server, nothing arrives — and silence looks +exactly like everything being fine. A dead man's switch inverts the handling for +one designated alert so that silence is the signal: + +- **receiving** it opens no incident, and +- the **absence** of it does. + +kube-prometheus-stack already ships the alert for this. `Watchdog` is +`expr: vector(1)`, so it fires permanently and is re-sent forever; it is worth +nothing unless something downstream notices it stop. That is what +`TERDUT_DEADMAN_MATCHERS` defaults to. + +A matcher is a set of exact label conditions, one of which must be the +`alertname`: + +``` +TERDUT_DEADMAN_MATCHERS="alertname=Watchdog,cluster=prod; alertname=EdgeHeartbeat" +``` + +**The unit of monitoring is the fingerprint, not the alert name.** Two clusters +sending the same `Watchdog` are two independent switches, so a healthy one can +never mask a dead one. + +#### The lifecycle + +A switch is **dormant** until its first heartbeat arrives. A configured matcher +that has never been heard from opens nothing, so a fresh deploy or a restored +database does not page. It also means a matcher that never matches anything is +silently inert — check the startup log line, which lists the matchers that +survived parsing. + +Once armed, the sweeper declares it **dead** when either the heartbeat has not +been refreshed within `TERDUT_DEADMAN_TIMEOUT`, or Alertmanager explicitly +resolved it — the sender saying the heartbeat stopped needs no further waiting. +That opens an incident at `TERDUT_DEADMAN_SEVERITY`, assigned and paged like any +other, and marks the heartbeat alert `"resolution_source": "deadman"` so the +alert list stops claiming a dead switch is firing. + +It **recovers** when the heartbeat starts arriving again: the incident resolves +with `"resolution_source": "recovered"` and the all-clear goes to whoever was +paged. + +Resolving the incident by hand sticks, the same way it does for an alert-backed +one. While the switch stays silent nothing new opens — so a decommissioned +source is a one-time page rather than a nag. The switch **re-arms** on the next +heartbeat: come back and die again, and that is a new incident. + +#### Two things to know + +`TERDUT_DEADMAN_TIMEOUT` must be **shorter** than the `repeat_interval` of the +route carrying the heartbeat, which is the exact opposite of +`TERDUT_STALE_AFTER`. Inheriting a default `repeat_interval` of 4h gives you a +switch that takes four hours to notice anything, so give the heartbeat +[its own route](#alertmanager-configuration). Matched alerts are exempt from +stale-alert expiry — a heartbeat answers to its own timeout and nothing else. + +A dead man's switch incident has **no member alerts**: +`GET /api/incidents/{id}/alerts` returns an empty list. There is no alert +describing the problem, because the problem is that no alert arrived. What +happened is on the timeline instead, as a `deadman_silent` event carrying the age +of the last heartbeat, and the heartbeat's labels are on the incident's +`group_labels`. + --- ## API reference @@ -345,7 +446,7 @@ only by their author. The rest of the timeline is a record of what happened. | `assigned_to_id` / `assigned_to` | | *optional* — user id, username | | `snoozed_until` | timestamp | *optional* — a value in the past reads as not snoozed | | `resolved_at` | timestamp | *optional* | -| `resolution_source` | string | *optional* — `"alerts"` or `"manual"` | +| `resolution_source` | string | *optional* — `"alerts"`, `"manual"` or `"recovered"` | | `archived_at` | timestamp | *optional* | | `alerts` | array | Only on `GET /api/incidents/{id}` | @@ -366,9 +467,9 @@ name: degrade unknown values to "resolved, reason unknown". Types written today: `triggered`, `alert_added`, `alert_resolved`, `acknowledged`, `unacknowledged`, `assigned`, `snoozed`, `unsnoozed`, `resolved`, -`note`, `notified`, `notify_failed`. On an `assigned` event `user_id` is the -**assignee**, not the actor. New types may be added; render unknown ones -generically rather than dropping them. +`note`, `notified`, `notify_failed`, `deadman_silent`. On an `assigned` event +`user_id` is the **assignee**, not the actor. New types may be added; render +unknown ones generically rather than dropping them. On `notified` and `notify_failed`, `detail` carries the notification kind (`triggered` | `reminder` | `resolved`), and on a failure the reason after it. @@ -390,7 +491,8 @@ Archived alerts are hidden from `GET /api/alerts` unless `?archived=true` is passed; alert archiving is automatic housekeeping by the sweeper, not a user action. Resolved alerts carry `resolution_source`: `"alertmanager"` for a real resolved webhook, `"expiry"` when the sweeper inferred it (see -[Stale alert expiry](#stale-alert-expiry)). +[Stale alert expiry](#stale-alert-expiry)), `"deadman"` for a heartbeat declared +dead (see [Dead man's switch](#dead-mans-switch)). #### The alert object @@ -411,7 +513,7 @@ when unset, so clients must treat them as nullable. | `generator_url` | string | Link back to the originating Prometheus | | `received_at` | timestamp | When the server last accepted a webhook for this alert — see below | | `incident_id` | integer | *optional* — the most recent incident this alert belongs to | -| `resolution_source` | string | *optional* — `"alertmanager"` or `"expiry"` | +| `resolution_source` | string | *optional* — `"alertmanager"`, `"expiry"` or `"deadman"` | | `archived_at` | timestamp | *optional* — set while archived | ##### `received_at` is a liveness heartbeat @@ -466,6 +568,13 @@ which happened. Clients may rely on this: worthwhile, since `"expiry"` can also mean the alert is still firing and the notification path broke. +- **`"deadman"` — a heartbeat was declared dead** (see + [Dead man's switch](#dead-mans-switch)). Like `"expiry"`, an inference from + silence rather than an observed end, so `ends_at` is approximate — but a much + tighter one, bounded by `TERDUT_DEADMAN_TIMEOUT`. It is also the one resolution + a re-fire under the same `starts_at` can undo, since the switch coming back is + exactly the evidence that the inference was wrong. + Treat the value as an open set and tolerate ones you do not recognise — new sources may be added, and unknown values should degrade to "resolved, reason unknown" rather than being rejected. @@ -522,6 +631,26 @@ Nothing about the two documented alert contracts changes: `received_at` is still advanced on every accepted webhook, and `resolution_source` still means what it did. +## Upgrading to dead man's switches + +Dead man's switch handling is **on by default**, watching `alertname=Watchdog` +with a 15 minute timeout. If you already route `Watchdog` to this server, the +behaviour of that alert changes on upgrade, in both directions: + +- it stops opening incidents when it arrives, and +- it starts opening one when it stops arriving. + +**Check your `repeat_interval` before upgrading.** The switch pages whenever a +heartbeat has not been refreshed within `TERDUT_DEADMAN_TIMEOUT`, so a `Watchdog` +route inheriting a 4h or 12h `repeat_interval` will page constantly against the +15 minute default. Either give the heartbeat +[its own fast route](#alertmanager-configuration) — the point of the feature — or +set `TERDUT_DEADMAN_TIMEOUT` above your current `repeat_interval` until you have. +`TERDUT_DEADMAN_TIMEOUT=0` turns the whole thing off. + +There is no migration and no schema change. An existing open incident from a +`Watchdog` that arrived under the old behaviour is unaffected; resolve it by hand. + --- ## Development diff --git a/charts/terdut-server/Chart.yaml b/charts/terdut-server/Chart.yaml index 8cb38e6..67b7823 100644 --- a/charts/terdut-server/Chart.yaml +++ b/charts/terdut-server/Chart.yaml @@ -2,5 +2,5 @@ apiVersion: v2 name: terdut-server description: A Helm chart for Terminal Duty — on-call alert management server type: application -version: 0.6.0 +version: 0.9.0 appVersion: "latest" diff --git a/charts/terdut-server/templates/deployment.yaml b/charts/terdut-server/templates/deployment.yaml index f237029..e3b6d62 100644 --- a/charts/terdut-server/templates/deployment.yaml +++ b/charts/terdut-server/templates/deployment.yaml @@ -73,6 +73,12 @@ spec: value: "{{ .Values.sweeper.staleAfter }}" - name: TERDUT_ARCHIVE_AFTER value: "{{ .Values.sweeper.archiveAfter }}" + - name: TERDUT_DEADMAN_MATCHERS + value: "{{ .Values.deadman.matchers }}" + - name: TERDUT_DEADMAN_TIMEOUT + value: "{{ .Values.deadman.timeout }}" + - name: TERDUT_DEADMAN_SEVERITY + value: "{{ .Values.deadman.severity }}" {{- if .Values.notify.ntfyUrl }} - name: TERDUT_NTFY_URL value: "{{ .Values.notify.ntfyUrl }}" diff --git a/charts/terdut-server/values.yaml b/charts/terdut-server/values.yaml index 18e6e1d..5bc62a4 100644 --- a/charts/terdut-server/values.yaml +++ b/charts/terdut-server/values.yaml @@ -26,6 +26,44 @@ sweeper: # How long a resolved alert stays in the default list before auto-archiving. archiveAfter: 168h +# Alerts treated as dead man's switches: receiving one opens no incident, and +# the absence of one does. The Watchdog alert kube-prometheus-stack ships is +# exactly this — an always-firing alert whose only value is something noticing +# when it stops. +deadman: + # Which alerts to treat as heartbeats. ";" separates matchers, "," separates + # the label conditions within one, "=" is exact equality. Every matcher must + # name an alertname: + # alertname=Watchdog,cluster=prod; alertname=EdgeHeartbeat + # Each distinct label set is watched independently, so two clusters sending + # the same alertname are two switches and a live one cannot mask a dead one. + matchers: "alertname=Watchdog" + # How long a heartbeat may go unheard before its switch is declared dead. + # + # This must be SHORTER than the Alertmanager repeat_interval of the route + # carrying the heartbeat — the opposite of sweeper.staleAfter. The default + # repeat_interval of 4h (12h in many setups) makes for a useless dead man's + # switch, so give the heartbeat a route of its own: + # + # - matchers: [ 'alertname = "Watchdog"' ] + # receiver: terdut + # group_wait: 0s + # group_interval: 1m + # repeat_interval: 1m + # + # That delivers every 2m rather than every 1m: a group is only reconsidered + # each group_interval, and at exactly one elapsed interval repeat_interval has + # not quite passed, so equal values give 2x. Fine against 15m; use + # group_interval: 30s if you want a true 1m. + # + # Set to 0 to disable dead man's switch handling entirely. + timeout: 15m + # Severity a dead man's switch incident opens at. These incidents have no + # member alerts to derive one from, and the heartbeat's own severity label is + # meaningless — Watchdog ships as "none". Only "critical" maps to the ntfy + # priority that overrides a phone's quiet hours. + severity: critical + notify: # ntfy server that push notifications are published to, e.g. # http://ntfy.ntfy.svc.cluster.local. Empty disables notifications entirely. diff --git a/cmd/terdut/main.go b/cmd/terdut/main.go index d0c7223..2726af6 100644 --- a/cmd/terdut/main.go +++ b/cmd/terdut/main.go @@ -36,7 +36,9 @@ func main() { RepeatEvery: cfg.NotifyRepeat, } - router := api.NewRouter(database, notify) + deadman := api.ParseDeadmanConfig(cfg.DeadmanMatchers, cfg.DeadmanTimeout, cfg.DeadmanSeverity) + + router := api.NewRouter(database, notify, deadman) srv := &http.Server{ Addr: cfg.Addr, @@ -49,7 +51,7 @@ func main() { ctx, stop := signal.NotifyContext(context.Background(), syscall.SIGINT, syscall.SIGTERM) defer stop() - go api.StartArchiver(ctx, database, cfg.ArchiveAfter, cfg.StaleAfter) + go api.StartArchiver(ctx, database, cfg.ArchiveAfter, cfg.StaleAfter, deadman, notify) go api.StartNotifier(ctx, database, notify) go func() { diff --git a/internal/api/alertmanager.go b/internal/api/alertmanager.go index 9729fd9..4339a7b 100644 --- a/internal/api/alertmanager.go +++ b/internal/api/alertmanager.go @@ -14,6 +14,12 @@ import ( const ( resolutionAlertmanager = "alertmanager" resolutionExpiry = "expiry" + + // resolutionDeadman marks a heartbeat the dead man's switch sweeper declared + // dead. Distinct from expiry because it is load-bearing, not just + // descriptive: it is the one resolution the ingest upsert will let a + // same-instance re-fire undo, so a switch that comes back can be heard. + resolutionDeadman = "deadman" ) // amPayload mirrors the Alertmanager webhook v4 payload. @@ -56,9 +62,15 @@ type ingested struct { // justResolved marks the firing → resolved edge, worth a timeline entry. justResolved bool + + // deadman marks a heartbeat: an alert whose arrival means everything is + // fine. It is stored like any other alert — received_at is the heartbeat — + // but it never reaches an incident. Its absence is what opens one, which + // sweepDeadman decides later and elsewhere. + deadman bool } -func handleAlertmanagerWebhook(db *sql.DB, notify NotifyConfig) http.HandlerFunc { +func handleAlertmanagerWebhook(db *sql.DB, notify NotifyConfig, deadman DeadmanConfig) http.HandlerFunc { return func(w http.ResponseWriter, r *http.Request) { var payload amPayload if err := decodeJSON(r, &payload); err != nil { @@ -69,7 +81,7 @@ func handleAlertmanagerWebhook(db *sql.DB, notify NotifyConfig) http.HandlerFunc // Alertmanager retries anything that is not 2xx, and a retry of a payload // we failed to store is more useful than an error it cannot act on — so // failures are logged, not surfaced. - if err := ingest(r.Context(), db, notify, payload); err != nil { + if err := ingest(r.Context(), db, notify, deadman, payload); err != nil { log.Printf("webhook ingest (group %q): %v", payload.GroupKey, err) } @@ -80,14 +92,14 @@ func handleAlertmanagerWebhook(db *sql.DB, notify NotifyConfig) http.HandlerFunc // ingest stores a payload's alerts and reconciles the incident for its group. // The whole payload is one transaction: an incident that opened but whose alerts // failed to link would be a work item nobody could act on. -func ingest(ctx context.Context, db *sql.DB, notify NotifyConfig, payload amPayload) error { +func ingest(ctx context.Context, db *sql.DB, notify NotifyConfig, deadman DeadmanConfig, payload amPayload) error { tx, err := db.BeginTx(ctx, nil) if err != nil { return err } defer tx.Rollback() //nolint:errcheck - accepted, err := upsertAlerts(ctx, tx, payload.Alerts) + accepted, err := upsertAlerts(ctx, tx, deadman, payload.Alerts) if err != nil { return err } @@ -103,7 +115,7 @@ func ingest(ctx context.Context, db *sql.DB, notify NotifyConfig, payload amPayl if incidentID != 0 { touched[incidentID] = true for _, a := range accepted { - if !a.firing { + if !a.firing || a.deadman { continue } if err := linkAlert(ctx, tx, incidentID, a.id); err != nil { @@ -113,7 +125,7 @@ func ingest(ctx context.Context, db *sql.DB, notify NotifyConfig, payload amPayl } for _, a := range accepted { - if !a.justResolved { + if !a.justResolved || a.deadman { continue } id, err := openIncidentForAlert(ctx, tx, a.id) @@ -144,7 +156,7 @@ func ingest(ctx context.Context, db *sql.DB, notify NotifyConfig, payload amPayl // upsertAlerts stores each alert of a payload and reports what changed. Payloads // the ordering guard rejected are left out entirely. -func upsertAlerts(ctx context.Context, tx *sql.Tx, alerts []amAlert) ([]ingested, error) { +func upsertAlerts(ctx context.Context, tx *sql.Tx, deadman DeadmanConfig, alerts []amAlert) ([]ingested, error) { now := time.Now().Unix() accepted := make([]ingested, 0, len(alerts)) @@ -187,7 +199,15 @@ func upsertAlerts(ctx context.Context, tx *sql.Tx, alerts []amAlert) ([]ingested // older than the stored one. Alertmanager retries failed notifications, // so a stale firing retry can arrive after the resolved one; it carries // the same startsAt, whereas a genuine re-fire carries a newer one. - // Within a single instance, resolution is terminal. + // Within a single instance, resolution is terminal — with one exception. + // + // A resolution this server synthesised for a dead man's switch is not + // Alertmanager's word that the instance ended; it is our inference from + // silence. The heartbeat that proves us wrong carries the unchanged + // startsAt of an alert that never stopped firing, so without the + // exemption a switch could go dead exactly once and never be heard from + // again. Scoped to 'deadman' so no resolution anybody else wrote can be + // undone by a stale retry. if _, err := tx.ExecContext(ctx, ` INSERT INTO alerts (fingerprint, name, status, labels, annotations, starts_at, ends_at, @@ -211,7 +231,8 @@ func upsertAlerts(ctx context.Context, tx *sql.Tx, alerts []amAlert) ([]ingested THEN NULL ELSE alerts.archived_at END WHERE excluded.starts_at > alerts.starts_at OR (excluded.starts_at = alerts.starts_at - AND NOT (alerts.status = 'resolved' AND excluded.status = 'firing'))`, + AND (alerts.resolution_source = '`+resolutionDeadman+`' + OR NOT (alerts.status = 'resolved' AND excluded.status = 'firing')))`, a.Fingerprint, name, a.Status, string(labelsJSON), string(annotationsJSON), a.StartsAt.Unix(), endsAtUnix, @@ -244,6 +265,7 @@ func upsertAlerts(ctx context.Context, tx *sql.Tx, alerts []amAlert) ([]ingested firing: firing, newOccurrence: firing && (!existed || a.StartsAt.Unix() > prevStartsAt || prevStatus == "resolved"), justResolved: !firing && existed && prevStatus == "firing", + deadman: deadman.isDeadman(a.Labels), }) } @@ -258,10 +280,17 @@ func upsertAlerts(ctx context.Context, tx *sql.Tx, alerts []amAlert) ([]ingested // something actually started firing. Without that, a manually resolved incident // would reappear on the next repeat_interval re-send of an alert that never // stopped, and manual resolution would be meaningless. +// +// Heartbeats do not count as anything here. A group of nothing but dead man's +// switch alerts opens no incident at all, and a mixed group gets an incident for +// its real alerts only. func incidentForGroup(ctx context.Context, tx *sql.Tx, notify NotifyConfig, payload amPayload, accepted []ingested) (int64, error) { var firstName string anyFiring, anyNew := false, false for _, a := range accepted { + if a.deadman { + continue + } if a.firing { if !anyFiring { firstName = a.name @@ -297,13 +326,20 @@ func incidentForGroup(ctx context.Context, tx *sql.Tx, notify NotifyConfig, payl if !anyNew { return 0, nil } - return openIncident(ctx, tx, notify, groupKey, payload.GroupLabels, firstName) + return openIncident(ctx, tx, notify, groupKey, + incidentTitle(payload.GroupLabels, firstName), payload.GroupLabels, nil) } -// openIncident creates an incident for a group and assigns it to whoever is on -// call today, which is the point at which the schedule stops being decorative. -func openIncident(ctx context.Context, tx *sql.Tx, notify NotifyConfig, groupKey string, groupLabels map[string]string, fallbackName string) (int64, error) { - onCall, err := currentOnCall(ctx, tx) +// openIncident creates an incident and assigns it to whoever is on call today, +// which is the point at which the schedule stops being decorative. +// +// The one place an incident is born, for both of the things that can raise one: +// the webhook, inside its transaction, and the dead man's switch sweeper, inside +// its own. Hence the querier rather than a *sql.Tx. A nil severity leaves the +// column for refreshSeverity to fill from the member alerts; the sweeper passes +// one because its incidents have no members to derive it from. +func openIncident(ctx context.Context, q querier, notify NotifyConfig, groupKey, title string, groupLabels map[string]string, severity *string) (int64, error) { + onCall, err := currentOnCall(ctx, q) if err != nil { return 0, err } @@ -313,10 +349,10 @@ func openIncident(ctx context.Context, tx *sql.Tx, notify NotifyConfig, groupKey labelsJSON = []byte("{}") } - res, err := tx.ExecContext(ctx, ` - INSERT INTO incidents (group_key, title, group_labels, status, triggered_at, assigned_to) - VALUES (?, ?, ?, 'triggered', ?, ?)`, - groupKey, incidentTitle(groupLabels, fallbackName), string(labelsJSON), + res, err := q.ExecContext(ctx, ` + INSERT INTO incidents (group_key, title, group_labels, status, severity, triggered_at, assigned_to) + VALUES (?, ?, ?, 'triggered', ?, ?, ?)`, + groupKey, title, string(labelsJSON), severity, time.Now().Unix(), onCall) if err != nil { return 0, err @@ -326,20 +362,20 @@ func openIncident(ctx context.Context, tx *sql.Tx, notify NotifyConfig, groupKey return 0, err } - if err := logEvent(ctx, tx, id, evTriggered, nil, nil, nil); err != nil { + if err := logEvent(ctx, q, id, evTriggered, nil, nil, nil); err != nil { return 0, err } if onCall != nil { // On an "assigned" event user_id is the assignee, not the actor. - if err := logEvent(ctx, tx, id, evAssigned, onCall, nil, nil); err != nil { + if err := logEvent(ctx, q, id, evAssigned, onCall, nil, nil); err != nil { return 0, err } } - // Queue the page, but do not send it here: this runs inside the webhook's - // transaction on a single-connection pool, so an HTTP call would hold up - // every other request. The notifier picks the row up within a tick. - if err := enqueueOpened(ctx, tx, notify, id, onCall); err != nil { + // Queue the page, but do not send it here: this runs inside a transaction on + // a single-connection pool, so an HTTP call would hold up every other + // request. The notifier picks the row up within a tick. + if err := enqueueOpened(ctx, q, notify, id, onCall); err != nil { return 0, err } return id, nil diff --git a/internal/api/api_test.go b/internal/api/api_test.go index 1204898..b1c2b6c 100644 --- a/internal/api/api_test.go +++ b/internal/api/api_test.go @@ -20,19 +20,31 @@ import ( // tests can age rows directly — the sweeper's inputs are wall-clock timestamps. type ts struct { *httptest.Server - key string - db *sql.DB - notify api.NotifyConfig + key string + db *sql.DB + notify api.NotifyConfig + deadman api.DeadmanConfig } // newTS builds a server over a fresh in-memory database. Notifications are off // unless a NotifyConfig is passed, so tests that predate them are unaffected. +// Dead man's switches are off too — see newDeadmanTS. func newTS(t *testing.T, notify ...api.NotifyConfig) *ts { t.Helper() var cfg api.NotifyConfig if len(notify) > 0 { cfg = notify[0] } + return newDeadmanTS(t, api.DeadmanConfig{}, cfg) +} + +// newDeadmanTS is newTS with dead man's switch handling configured. +func newDeadmanTS(t *testing.T, deadman api.DeadmanConfig, notify ...api.NotifyConfig) *ts { + t.Helper() + var cfg api.NotifyConfig + if len(notify) > 0 { + cfg = notify[0] + } database, err := db.Open(":memory:") if err != nil { @@ -41,7 +53,7 @@ func newTS(t *testing.T, notify ...api.NotifyConfig) *ts { if err := db.Migrate(database); err != nil { t.Fatalf("migrate: %v", err) } - srv := httptest.NewServer(api.NewRouter(database, cfg)) + srv := httptest.NewServer(api.NewRouter(database, cfg, deadman)) t.Cleanup(func() { srv.Close(); database.Close() }) body, _ := json.Marshal(map[string]string{"username": "admin", "email": "admin@test.com"}) @@ -57,7 +69,7 @@ func newTS(t *testing.T, notify ...api.NotifyConfig) *ts { json.NewDecoder(resp.Body).Decode(&result) key := result["api_key"].(map[string]any)["key"].(string) - return &ts{Server: srv, key: key, db: database, notify: cfg} + return &ts{Server: srv, key: key, db: database, notify: cfg, deadman: deadman} } // exec runs a statement against the test database. @@ -493,7 +505,7 @@ func TestArchive_AlertListFilter(t *testing.T) { } // 2. Let the sweeper archive it: ends_at is already well past archiveAfter. - api.Sweep(context.Background(), s.db, time.Hour, 6*time.Hour) + api.Sweep(context.Background(), s.db, time.Hour, 6*time.Hour, s.deadman, s.notify) // 3. Default list excludes it. decode(t, s.req(t, http.MethodGet, "/api/alerts", nil), &alerts) @@ -534,7 +546,7 @@ func postAlert(t *testing.T, s *ts, fingerprint, status, startsAt, endsAt string func sweep(t *testing.T, s *ts, staleAfter time.Duration) { t.Helper() - api.Sweep(context.Background(), s.db, noArchive, staleAfter) + api.Sweep(context.Background(), s.db, noArchive, staleAfter, s.deadman, s.notify) } // A firing alert Alertmanager stopped refreshing is resolved via the diff --git a/internal/api/archiver.go b/internal/api/archiver.go index bc2fb75..3248d64 100644 --- a/internal/api/archiver.go +++ b/internal/api/archiver.go @@ -19,28 +19,34 @@ const ( // StartArchiver runs the alert sweeper until ctx is cancelled, starting with an // immediate pass so a restart reconciles state right away. -func StartArchiver(ctx context.Context, db *sql.DB, archiveAfter, staleAfter time.Duration) { +func StartArchiver(ctx context.Context, db *sql.DB, archiveAfter, staleAfter time.Duration, deadman DeadmanConfig, notify NotifyConfig) { ticker := time.NewTicker(sweepInterval) defer ticker.Stop() - Sweep(ctx, db, archiveAfter, staleAfter) + Sweep(ctx, db, archiveAfter, staleAfter, deadman, notify) for { select { case <-ticker.C: - Sweep(ctx, db, archiveAfter, staleAfter) + Sweep(ctx, db, archiveAfter, staleAfter, deadman, notify) case <-ctx.Done(): return } } } -// Sweep runs a single pass, in dependency order: expire stale firing alerts, -// close the incidents that leaves with nothing firing, then archive whatever has -// been settled long enough. Running them in one pass means an alert can go stale -// and its incident can close and archive without waiting three ticks. +// Sweep runs a single pass, in dependency order: reconcile the dead man's +// switches, expire stale firing alerts, close the incidents that leaves with +// nothing firing, then archive whatever has been settled long enough. Running +// them in one pass means an alert can go stale and its incident can close and +// archive without waiting three ticks. +// +// The switches go first because they hand expireStale the alerts it must not +// touch: a heartbeat answers to its own, much tighter, timeout, and the generic +// staleness rules would otherwise resolve it as 'expiry' long before that. // Exported so tests can drive a pass without waiting on the ticker. -func Sweep(ctx context.Context, db *sql.DB, archiveAfter, staleAfter time.Duration) { - expireStale(ctx, db, staleAfter) +func Sweep(ctx context.Context, db *sql.DB, archiveAfter, staleAfter time.Duration, deadman DeadmanConfig, notify NotifyConfig) { + heartbeats := sweepDeadman(ctx, db, deadman, notify) + expireStale(ctx, db, staleAfter, heartbeats) resolveSettledIncidents(ctx, db) archiveResolved(ctx, db, archiveAfter) archiveResolvedIncidents(ctx, db, archiveAfter) @@ -60,16 +66,26 @@ func Sweep(ctx context.Context, db *sql.DB, archiveAfter, staleAfter time.Durati // notifications every repeat_interval, making received_at a liveness // heartbeat — provided staleAfter exceeds that interval. // +// Alerts in skip are left alone: they are dead man's switch heartbeats, whose +// liveness sweepDeadman has already judged against a timeout of its own. +// // The matching rows are collected before the update rather than updated in bulk, // because each one owes its incident a timeline entry. -func expireStale(ctx context.Context, db *sql.DB, staleAfter time.Duration) { +func expireStale(ctx context.Context, db *sql.DB, staleAfter time.Duration, skip map[int64]bool) { now := time.Now() - ids, err := staleAlertIDs(ctx, db, now, staleAfter) + found, err := staleAlertIDs(ctx, db, now, staleAfter) if err != nil { log.Printf("sweeper: find stale: %v", err) return } + + ids := make([]int64, 0, len(found)) + for _, id := range found { + if !skip[id] { + ids = append(ids, id) + } + } if len(ids) == 0 { return } diff --git a/internal/api/deadman.go b/internal/api/deadman.go new file mode 100644 index 0000000..d54447c --- /dev/null +++ b/internal/api/deadman.go @@ -0,0 +1,378 @@ +package api + +import ( + "context" + "database/sql" + "encoding/json" + "log" + "sort" + "strings" + "time" +) + +// deadmanGroupPrefix namespaces the incidents this file opens. Alertmanager +// group keys always contain braces, so this can never collide with one, and the +// partial unique index on open group_key (see 008_incidents.sql) gives one open +// incident per switch for free. +const deadmanGroupPrefix = "deadman:" + +// DeadmanMatcher selects the alerts that are heartbeats rather than problems. +// Every condition has to match, and Name — the alertname label — is mandatory: +// it is what lets the sweeper find candidate rows through alerts_name_idx +// instead of JSON-extracting labels from every row in the table. +type DeadmanMatcher struct { + Name string + Labels map[string]string +} + +// String renders the matcher the way it was configured, which is also how it +// reads in an incident title. +func (m DeadmanMatcher) String() string { + if len(m.Labels) == 0 { + return m.Name + } + parts := make([]string, 0, len(m.Labels)) + for k, v := range m.Labels { + parts = append(parts, k+"="+v) + } + sort.Strings(parts) + return m.Name + " (" + strings.Join(parts, ", ") + ")" +} + +// matches reports whether an alert's labels satisfy every condition. +func (m DeadmanMatcher) matches(labels map[string]string) bool { + if labels["alertname"] != m.Name { + return false + } + for k, v := range m.Labels { + if labels[k] != v { + return false + } + } + return true +} + +// DeadmanConfig inverts the handling of the alerts it matches: receiving one +// opens nothing, and the absence of one opens an incident. +// +// The unit of monitoring is the fingerprint, not the matcher — two clusters +// sending the same heartbeat alertname are two independent switches, so one +// healthy cluster cannot mask a dead one. +type DeadmanConfig struct { + Matchers []DeadmanMatcher + + // Timeout is how long a matched alert may go without a refreshing webhook + // before it is declared dead. It must be shorter than Alertmanager's + // repeat_interval for the heartbeat's route, which is what refreshes it. + // Zero disables dead man's switch handling entirely. + Timeout time.Duration + + // Severity is the severity every dead man's switch incident opens at. These + // incidents have no member alerts to derive one from, and the heartbeat's + // own severity label is meaningless — Watchdog ships as "none". + Severity string +} + +// enabled reports whether there is anything to watch. +func (c DeadmanConfig) enabled() bool { return c.Timeout > 0 && len(c.Matchers) > 0 } + +// match returns the first matcher an alert satisfies. +func (c DeadmanConfig) match(labels map[string]string) (DeadmanMatcher, bool) { + if !c.enabled() { + return DeadmanMatcher{}, false + } + for _, m := range c.Matchers { + if m.matches(labels) { + return m, true + } + } + return DeadmanMatcher{}, false +} + +// isDeadman is match without the matcher, for the ingest path. +func (c DeadmanConfig) isDeadman(labels map[string]string) bool { + _, ok := c.match(labels) + return ok +} + +// names lists the distinct alertnames worth loading from the database. +func (c DeadmanConfig) names() []string { + seen := map[string]bool{} + out := make([]string, 0, len(c.Matchers)) + for _, m := range c.Matchers { + if !seen[m.Name] { + seen[m.Name] = true + out = append(out, m.Name) + } + } + return out +} + +// ParseDeadmanConfig reads the matcher list from its configured form: +// ";" separates matchers, "," separates the conditions within one, and "=" is +// exact label equality — `alertname=Watchdog,cluster=prod; alertname=Heartbeat`. +// +// A malformed or alertname-less entry is dropped rather than fatal, following +// config.duration's rule that one bad tuning knob should not take the server +// down. Silence would be worse here than elsewhere, though — a typo that +// disarms the switch is exactly the failure this feature exists to catch — so +// the matchers that survived are logged. +func ParseDeadmanConfig(matchers string, timeout time.Duration, severity string) DeadmanConfig { + cfg := DeadmanConfig{Timeout: timeout, Severity: severity} + + for _, entry := range strings.Split(matchers, ";") { + entry = strings.TrimSpace(entry) + if entry == "" { + continue + } + + m := DeadmanMatcher{Labels: map[string]string{}} + malformed := false + for _, cond := range strings.Split(entry, ",") { + k, v, ok := strings.Cut(cond, "=") + k, v = strings.TrimSpace(k), strings.TrimSpace(v) + if !ok || k == "" || v == "" { + log.Printf("deadman: ignoring matcher %q: %q is not label=value", entry, strings.TrimSpace(cond)) + malformed = true + break + } + if k == "alertname" { + m.Name = v + continue + } + m.Labels[k] = v + } + if malformed { + continue + } + if m.Name == "" { + log.Printf("deadman: ignoring matcher %q: no alertname condition", entry) + continue + } + cfg.Matchers = append(cfg.Matchers, m) + } + + switch { + case timeout <= 0: + log.Print("deadman: disabled (timeout is zero)") + case len(cfg.Matchers) == 0: + log.Print("deadman: disabled (no usable matchers)") + default: + rendered := make([]string, 0, len(cfg.Matchers)) + for _, m := range cfg.Matchers { + rendered = append(rendered, m.String()) + } + log.Printf("deadman: watching %s, timeout %s, severity %s", + strings.Join(rendered, "; "), timeout, severity) + } + return cfg +} + +// deadmanAlert is one switch: the alert row carrying its last heartbeat. +type deadmanAlert struct { + id int64 + fingerprint string + labels map[string]string + matcher DeadmanMatcher + resolved bool + receivedAt int64 +} + +// groupKey is the switch's identity as an incident. Per fingerprint, so each +// source is tracked on its own. +func (a deadmanAlert) groupKey() string { return deadmanGroupPrefix + a.fingerprint } + +// sweepDeadman is the whole point of the feature: it opens an incident for every +// switch that has stopped chirping, and closes one whose switch came back. +// +// It returns the ids of the alerts it owns, because the generic staleness +// expiry must leave them alone — staleAfter and ends_at would otherwise resolve +// a heartbeat long before its own, much tighter, timeout ever fired. +func sweepDeadman(ctx context.Context, db *sql.DB, cfg DeadmanConfig, notify NotifyConfig) map[int64]bool { + owned := map[int64]bool{} + if !cfg.enabled() { + return owned + } + + switches, err := deadmanAlerts(ctx, db, cfg) + if err != nil { + log.Printf("deadman: load switches: %v", err) + return owned + } + + now := time.Now() + cutoff := now.Add(-cfg.Timeout).Unix() + + for _, sw := range switches { + owned[sw.id] = true + + // An explicit resolved from Alertmanager is a stronger death signal than + // mere absence: the sender is telling us the heartbeat stopped, so there + // is nothing left to wait out. + if sw.resolved || sw.receivedAt < cutoff { + if err := deadmanDied(ctx, db, cfg, notify, sw, now); err != nil { + log.Printf("deadman: open incident for %s: %v", sw.matcher.Name, err) + } + continue + } + if err := deadmanRecovered(ctx, db, sw); err != nil { + log.Printf("deadman: resolve incident for %s: %v", sw.matcher.Name, err) + } + } + return owned +} + +// deadmanAlerts loads every alert row that a matcher claims. The candidate query +// is narrowed by alertname so it rides alerts_name_idx; the rest of the matching +// happens in Go, which keeps one implementation of the rules. The rows are read +// in full before the caller writes, because the pool holds a single connection. +func deadmanAlerts(ctx context.Context, db *sql.DB, cfg DeadmanConfig) ([]deadmanAlert, error) { + names := cfg.names() + args := make([]any, 0, len(names)) + for _, n := range names { + args = append(args, n) + } + + rows, err := db.QueryContext(ctx, ` + SELECT id, fingerprint, labels, status, received_at + FROM alerts + WHERE name IN (`+placeholders(len(names))+`) + AND archived_at IS NULL`, args...) + if err != nil { + return nil, err + } + defer rows.Close() + + var out []deadmanAlert + for rows.Next() { + var a deadmanAlert + var labelsJSON, status string + if err := rows.Scan(&a.id, &a.fingerprint, &labelsJSON, &status, &a.receivedAt); err != nil { + return nil, err + } + json.Unmarshal([]byte(labelsJSON), &a.labels) //nolint:errcheck + + m, ok := cfg.match(a.labels) + if !ok { + continue + } + a.matcher = m + a.resolved = status == "resolved" + out = append(out, a) + } + return out, rows.Err() +} + +// deadmanDied raises the incident for a switch that has gone quiet. +// +// Two conditions gate it, and both matter. There must be no open incident for +// the switch already — the partial unique index enforces that anyway, but a +// second one would be a wasted page. And the heartbeat must have been seen since +// the last incident was raised, which is the re-arm rule: resolving a dead man's +// switch incident sticks, exactly as resolving an alert-backed one does (see +// incidentForGroup), and a source that is gone for good is a one-time page +// rather than a nag. Only a heartbeat that comes back and dies again earns a new +// incident. +func deadmanDied(ctx context.Context, db *sql.DB, cfg DeadmanConfig, notify NotifyConfig, sw deadmanAlert, now time.Time) error { + var lastTriggered, open int64 + if err := db.QueryRowContext(ctx, ` + SELECT COALESCE(MAX(triggered_at), 0), + COALESCE(SUM(resolved_at IS NULL), 0) + FROM incidents WHERE group_key = ?`, + sw.groupKey()).Scan(&lastTriggered, &open); err != nil { + return err + } + if open > 0 || sw.receivedAt <= lastTriggered { + return nil + } + + tx, err := db.BeginTx(ctx, nil) + if err != nil { + return err + } + defer tx.Rollback() //nolint:errcheck + + // A heartbeat nobody has heard from is not firing, and saying otherwise in + // the alert list would be a lie. An Alertmanager-sourced resolution keeps its + // own source: it told us the truth first. + if !sw.resolved { + if _, err := tx.ExecContext(ctx, ` + UPDATE alerts + SET status = 'resolved', + resolution_source = ?, + ends_at = COALESCE(ends_at, unixepoch()) + WHERE id = ? AND status = 'firing'`, resolutionDeadman, sw.id); err != nil { + return err + } + } + + severity := cfg.Severity + var sev *string + if severity != "" { + sev = &severity + } + + incidentID, err := openIncident(ctx, tx, notify, sw.groupKey(), + "No heartbeat from "+sw.matcher.String(), sw.labels, sev) + if err != nil { + return err + } + + alertID := sw.id + detail := "last heartbeat " + humanDuration(now.Sub(time.Unix(sw.receivedAt, 0))) + " ago" + if err := logEvent(ctx, tx, incidentID, evDeadmanSilent, nil, &alertID, &detail); err != nil { + return err + } + + if err := tx.Commit(); err != nil { + return err + } + log.Printf("deadman: %s went silent, opened incident %d", sw.matcher.String(), incidentID) + return nil +} + +// deadmanRecovered closes the incident for a switch that started chirping again. +// +// It cannot go through resolveIfSettled: a dead man's switch incident has no +// member alerts (linking the heartbeat would have the settled-incident cascade +// close it on the very same sweep that opened it), so the alert-driven cascade +// ignores it entirely and recovery is the only automatic way out. +func deadmanRecovered(ctx context.Context, db *sql.DB, sw deadmanAlert) error { + var incidentID int64 + switch err := db.QueryRowContext(ctx, ` + SELECT id FROM incidents + WHERE group_key = ? AND resolved_at IS NULL`, sw.groupKey()).Scan(&incidentID); { + case err == sql.ErrNoRows: + return nil + case err != nil: + return err + } + + tx, err := db.BeginTx(ctx, nil) + if err != nil { + return err + } + defer tx.Rollback() //nolint:errcheck + + if _, err := tx.ExecContext(ctx, ` + UPDATE incidents + SET status = 'resolved', resolved_at = ?, resolution_source = ? + WHERE id = ? AND resolved_at IS NULL`, + time.Now().Unix(), incidentResolutionRecovered, incidentID); err != nil { + return err + } + if err := logEvent(ctx, tx, incidentID, evResolved, nil, nil, nil); err != nil { + return err + } + // The all-clear goes to whoever was paged, which enqueueResolved works out + // from the incident's own notification history. + if err := enqueueResolved(ctx, tx, incidentID); err != nil { + return err + } + + if err := tx.Commit(); err != nil { + return err + } + log.Printf("deadman: %s is back, resolved incident %d", sw.matcher.String(), incidentID) + return nil +} diff --git a/internal/api/deadman_test.go b/internal/api/deadman_test.go new file mode 100644 index 0000000..727b43a --- /dev/null +++ b/internal/api/deadman_test.go @@ -0,0 +1,468 @@ +package api_test + +import ( + "net/http" + "testing" + "time" + + "github.com/yeniklas/terdut-server/internal/api" +) + +// --------------------------------------------------------------------------- +// Harness +// --------------------------------------------------------------------------- + +// watchdogGroupKey is what Alertmanager sends for a Watchdog grouped by +// alertname, which is how the deployed route is configured. +const watchdogGroupKey = `{}:{alertname="Watchdog"}` + +// deadmanCfg watches Watchdog with a timeout short enough to reason about and +// long enough that a fresh heartbeat is never accidentally stale. +func deadmanCfg() api.DeadmanConfig { + return api.ParseDeadmanConfig("alertname=Watchdog", time.Hour, "critical") +} + +// deadmanTS is notifyTS with dead man's switch handling on: notifications +// enabled against a fake ntfy, the admin on call today with a topic. +func deadmanTS(t *testing.T, cfg api.DeadmanConfig) (*ts, *fakeNtfy) { + t.Helper() + f := newFakeNtfy(t) + s := newDeadmanTS(t, cfg, api.NotifyConfig{ + BaseURL: f.URL, + PublicURL: "https://terdut.example.com", + }) + + putOnCall(t, s, 1) + setTopic(t, s, 1, "terdut-admin") + return s, f +} + +// heartbeat posts one Watchdog webhook. Its startsAt never changes: a dead man's +// switch alert fires once and is re-sent unchanged forever, which is precisely +// what makes its absence meaningful. +func heartbeat(t *testing.T, s *ts, fingerprint string, labels map[string]string) { + t.Helper() + postWebhook(t, s, []map[string]any{ + amAlert(fingerprint, "Watchdog", "firing", "2026-05-20T10:00:00Z", zeroTime, labels), + }, watchdogGroupKey) +} + +// silence back-dates a heartbeat's received_at, which is the only clock the +// sweeper reads. There is no fake clock in this package. +func silence(t *testing.T, s *ts, fingerprint string, ago time.Duration) { + t.Helper() + s.exec(t, "UPDATE alerts SET received_at = ? WHERE fingerprint = ?", + time.Now().Add(-ago).Unix(), fingerprint) +} + +// ageIncidents back-dates every incident. The re-arm rule compares a heartbeat +// against the last incident raised for its switch, so a test that wants a second +// episode has to put the first one in the past — there is no fake clock here. +func ageIncidents(t *testing.T, s *ts, ago time.Duration) { + t.Helper() + past := time.Now().Add(-ago).Unix() + s.exec(t, `UPDATE incidents + SET triggered_at = ?, + resolved_at = CASE WHEN resolved_at IS NULL THEN NULL ELSE ? END`, + past, past) +} + +// incidentByGroup reads the incident for a group key, resolved ones included. +func incidentByGroup(t *testing.T, s *ts, groupKey string) (id int64, status, severity string, source *string) { + t.Helper() + err := s.db.QueryRow(` + SELECT id, status, COALESCE(severity, ''), resolution_source + FROM incidents WHERE group_key = ? ORDER BY id DESC LIMIT 1`, + groupKey).Scan(&id, &status, &severity, &source) + if err != nil { + t.Fatalf("read incident for group %s: %v", groupKey, err) + } + return id, status, severity, source +} + +// --------------------------------------------------------------------------- +// Receiving a heartbeat +// --------------------------------------------------------------------------- + +// The whole inversion: arrival of a dead man's switch alert is good news, and +// good news is not an incident. +func TestDeadman_HeartbeatOpensNoIncident(t *testing.T) { + s, _ := deadmanTS(t, deadmanCfg()) + + heartbeat(t, s, "fp-watchdog", nil) + + if got := s.countIncidents(t); got != 0 { + t.Fatalf("expected a heartbeat to open no incident, got %d", got) + } + if got := s.countNotifications(t, ""); got != 0 { + t.Errorf("expected no notification for a heartbeat, got %d", got) + } + if status, _, _ := s.alertRow(t, "fp-watchdog"); status != "firing" { + t.Errorf("expected the heartbeat to be stored firing, got %q", status) + } +} + +// A heartbeat routed into a group alongside real alerts must not join their +// incident: it is not a symptom of anything. +func TestDeadman_MixedGroupExcludesHeartbeat(t *testing.T) { + s, _ := deadmanTS(t, deadmanCfg()) + + postWebhook(t, s, []map[string]any{ + amAlert("fp-mixed-wd", "Watchdog", "firing", "2026-05-20T10:00:00Z", zeroTime, nil), + amAlert("fp-mixed-disk", "DiskFull", "firing", "2026-05-20T10:00:00Z", zeroTime, + map[string]string{"severity": "critical"}), + }, `{}:{namespace="prod"}`) + + if got := s.countIncidents(t); got != 1 { + t.Fatalf("expected 1 incident for the real alert, got %d", got) + } + + var alerts []map[string]any + decode(t, s.req(t, http.MethodGet, "/api/incidents/1/alerts", nil), &alerts) + if len(alerts) != 1 { + t.Fatalf("expected 1 member alert, got %d", len(alerts)) + } + if name := alerts[0]["name"]; name != "DiskFull" { + t.Errorf("expected only the real alert linked, got %v", name) + } +} + +// A matcher scoped by label only claims the alerts it names, so a heartbeat from +// somewhere else stays an ordinary alert. +func TestDeadman_LabelScopedMatcherIgnoresOthers(t *testing.T) { + s, _ := deadmanTS(t, api.ParseDeadmanConfig("alertname=Watchdog,cluster=prod", time.Hour, "critical")) + + heartbeat(t, s, "fp-dev", map[string]string{"cluster": "dev"}) + + if got := s.countIncidents(t); got != 1 { + t.Fatalf("expected an unmatched Watchdog to behave like any other alert, got %d incidents", got) + } +} + +// --------------------------------------------------------------------------- +// Silence +// --------------------------------------------------------------------------- + +func TestDeadman_SilenceOpensIncident(t *testing.T) { + s, f := deadmanTS(t, deadmanCfg()) + + heartbeat(t, s, "fp-watchdog", nil) + silence(t, s, "fp-watchdog", 2*time.Hour) + sweep(t, s, noArchive) + + if got := s.countIncidents(t); got != 1 { + t.Fatalf("expected silence to open 1 incident, got %d", got) + } + id, status, severity, _ := incidentByGroup(t, s, "deadman:fp-watchdog") + if status != "triggered" { + t.Errorf("expected a triggered incident, got %q", status) + } + if severity != "critical" { + t.Errorf("expected the configured severity, got %q", severity) + } + + // The alert list must not keep claiming a dead heartbeat is firing. + alertStatus, source, _ := s.alertRow(t, "fp-watchdog") + if alertStatus != "resolved" || source == nil || *source != "deadman" { + t.Errorf("expected the heartbeat resolved as deadman, got %q / %v", alertStatus, source) + } + + // Nobody was told anything by an alert here, so the page has to come from + // the switch itself. + s.sweepNotify(t) + msgs := f.messages() + if len(msgs) != 1 { + t.Fatalf("expected 1 page, got %d", len(msgs)) + } + if msgs[0].Topic != "terdut-admin" { + t.Errorf("expected the on-call topic, got %q", msgs[0].Topic) + } + if msgs[0].Priority != 5 { + t.Errorf("expected a critical page to override quiet hours (priority 5), got %d", msgs[0].Priority) + } + + // The timeline says why, with the age of the last heartbeat. + types := eventTypes(timeline(t, s, int(id))) + found := false + for _, ty := range types { + if ty == "deadman_silent" { + found = true + } + } + if !found { + t.Errorf("expected a deadman_silent event, got %v", types) + } +} + +// The generic staleness sweep must keep its hands off heartbeats: they answer to +// their own, much tighter, timeout, and an 'expiry' resolution here would be +// both wrong and unrecoverable. +func TestDeadman_GenericExpiryLeavesHeartbeatAlone(t *testing.T) { + s, _ := deadmanTS(t, deadmanCfg()) + + heartbeat(t, s, "fp-watchdog", nil) + silence(t, s, "fp-watchdog", 5*time.Minute) + + // staleAfter far tighter than the dead man's switch timeout. + sweep(t, s, time.Minute) + + status, source, _ := s.alertRow(t, "fp-watchdog") + if status != "firing" || source != nil { + t.Errorf("expected a live heartbeat left alone, got %q / %v", status, source) + } + if got := s.countIncidents(t); got != 0 { + t.Errorf("expected no incident for a heartbeat that is still fresh, got %d", got) + } +} + +// An explicit resolved from Alertmanager is the sender telling us the heartbeat +// stopped. There is nothing left to wait out. +func TestDeadman_AlertmanagerResolvedIsImmediateDeath(t *testing.T) { + s, _ := deadmanTS(t, deadmanCfg()) + + heartbeat(t, s, "fp-watchdog", nil) + postWebhook(t, s, []map[string]any{ + amAlert("fp-watchdog", "Watchdog", "resolved", "2026-05-20T10:00:00Z", zeroTime, nil), + }, watchdogGroupKey) + + // No ageing: received_at is seconds old, well inside the timeout. + sweep(t, s, noArchive) + + if got := s.countIncidents(t); got != 1 { + t.Fatalf("expected a resolved heartbeat to open an incident at once, got %d", got) + } + // Alertmanager told the truth first, so its resolution source stands. + if _, source, _ := s.alertRow(t, "fp-watchdog"); source == nil || *source != "alertmanager" { + t.Errorf("expected the Alertmanager resolution source kept, got %v", source) + } +} + +// Each label set is its own switch, so one healthy source cannot mask a dead one. +func TestDeadman_TracksEachFingerprintSeparately(t *testing.T) { + s, _ := deadmanTS(t, deadmanCfg()) + + heartbeat(t, s, "fp-a", map[string]string{"cluster": "a"}) + heartbeat(t, s, "fp-b", map[string]string{"cluster": "b"}) + silence(t, s, "fp-b", 2*time.Hour) + sweep(t, s, noArchive) + + if got := s.countIncidents(t); got != 1 { + t.Fatalf("expected only the silent switch to page, got %d incidents", got) + } + if _, status, _, _ := incidentByGroup(t, s, "deadman:fp-b"); status != "triggered" { + t.Errorf("expected the incident to belong to the silent switch, got %q", status) + } + if status, _, _ := s.alertRow(t, "fp-a"); status != "firing" { + t.Errorf("expected the live switch untouched, got %q", status) + } +} + +// A switch nothing has ever been heard from is dormant. A fresh deploy, a +// restored database or a typo'd alertname must not page. +func TestDeadman_UnheardOfSwitchIsDormant(t *testing.T) { + s, _ := deadmanTS(t, api.ParseDeadmanConfig("alertname=NeverSent", time.Hour, "critical")) + + sweep(t, s, noArchive) + + if got := s.countIncidents(t); got != 0 { + t.Fatalf("expected a switch that never chirped to be dormant, got %d incidents", got) + } +} + +// --------------------------------------------------------------------------- +// Recovery and re-arming +// --------------------------------------------------------------------------- + +// The returning heartbeat carries the unchanged startsAt of an alert that never +// stopped firing, so this also covers the ingest guard exemption: without it the +// upsert would discard the payload and the switch could die exactly once. +func TestDeadman_RecoveryResolvesIncident(t *testing.T) { + s, _ := deadmanTS(t, deadmanCfg()) + + heartbeat(t, s, "fp-watchdog", nil) + silence(t, s, "fp-watchdog", 2*time.Hour) + sweep(t, s, noArchive) + + heartbeat(t, s, "fp-watchdog", nil) + if status, source, _ := s.alertRow(t, "fp-watchdog"); status != "firing" || source != nil { + t.Fatalf("expected the returning heartbeat to be accepted, got %q / %v", status, source) + } + + sweep(t, s, noArchive) + + _, status, _, source := incidentByGroup(t, s, "deadman:fp-watchdog") + if status != "resolved" { + t.Errorf("expected recovery to close the incident, got %q", status) + } + if source == nil || *source != "recovered" { + t.Errorf("expected resolution_source recovered, got %v", source) + } + if got := s.countNotifications(t, "resolved"); got != 1 { + t.Errorf("expected 1 all-clear, got %d", got) + } +} + +// Resolving a dead man's switch incident sticks, exactly as it does for an +// alert-backed one. A source that is gone for good is a one-time page. +func TestDeadman_ManualResolveSticksWhileSilent(t *testing.T) { + s, _ := deadmanTS(t, deadmanCfg()) + + heartbeat(t, s, "fp-watchdog", nil) + silence(t, s, "fp-watchdog", 2*time.Hour) + sweep(t, s, noArchive) + + s.req(t, http.MethodPost, "/api/incidents/1/resolve", nil).Body.Close() + + // Still silent, several sweeps later. + sweep(t, s, noArchive) + sweep(t, s, noArchive) + + if got := s.countIncidents(t); got != 1 { + t.Fatalf("expected a manually resolved incident to stay closed, got %d", got) + } +} + +// ...but the switch re-arms, so a heartbeat that comes back and dies again is a +// new incident rather than silence forever. +func TestDeadman_ReArmsAfterHeartbeatReturns(t *testing.T) { + s, _ := deadmanTS(t, deadmanCfg()) + + heartbeat(t, s, "fp-watchdog", nil) + silence(t, s, "fp-watchdog", 2*time.Hour) + sweep(t, s, noArchive) + s.req(t, http.MethodPost, "/api/incidents/1/resolve", nil).Body.Close() + + // That episode is yesterday's news; the heartbeat now returns after it. + ageIncidents(t, s, 10*time.Hour) + + heartbeat(t, s, "fp-watchdog", nil) + sweep(t, s, noArchive) + if got := s.countIncidents(t); got != 1 { + t.Fatalf("expected the live switch to open nothing, got %d incidents", got) + } + + silence(t, s, "fp-watchdog", 2*time.Hour) + sweep(t, s, noArchive) + + if got := s.countIncidents(t); got != 2 { + t.Fatalf("expected a second death to open a second incident, got %d", got) + } +} + +// A dead man's switch incident has no member alerts — linking the heartbeat +// would have the settled-incident cascade close it on the very sweep that opened +// it — so the cascade has to leave it alone. +func TestDeadman_SettledCascadeLeavesIncidentOpen(t *testing.T) { + s, _ := deadmanTS(t, deadmanCfg()) + + heartbeat(t, s, "fp-watchdog", nil) + silence(t, s, "fp-watchdog", 2*time.Hour) + sweep(t, s, noArchive) + sweep(t, s, noArchive) + + if _, status, _, _ := incidentByGroup(t, s, "deadman:fp-watchdog"); status != "triggered" { + t.Fatalf("expected the incident to stay open until the switch recovers, got %q", status) + } +} + +// --------------------------------------------------------------------------- +// Configuration +// --------------------------------------------------------------------------- + +func TestParseDeadmanConfig(t *testing.T) { + tests := []struct { + name string + matchers string + timeout time.Duration + want []api.DeadmanMatcher + enabled bool + }{ + { + name: "single alertname", + matchers: "alertname=Watchdog", + timeout: time.Hour, + want: []api.DeadmanMatcher{{Name: "Watchdog", Labels: map[string]string{}}}, + enabled: true, + }, + { + name: "several matchers with extra labels and whitespace", + matchers: " alertname=Watchdog, cluster=prod ; alertname=EdgeHeartbeat ", + timeout: time.Hour, + want: []api.DeadmanMatcher{ + {Name: "Watchdog", Labels: map[string]string{"cluster": "prod"}}, + {Name: "EdgeHeartbeat", Labels: map[string]string{}}, + }, + enabled: true, + }, + { + // Mandatory: it is what keeps the sweeper's candidate query on an index. + name: "matcher without alertname is dropped", + matchers: "cluster=prod; alertname=Watchdog", + timeout: time.Hour, + want: []api.DeadmanMatcher{{Name: "Watchdog", Labels: map[string]string{}}}, + enabled: true, + }, + { + name: "malformed condition drops only its matcher", + matchers: "alertname=Watchdog,garbage; alertname=Other", + timeout: time.Hour, + want: []api.DeadmanMatcher{{Name: "Other", Labels: map[string]string{}}}, + enabled: true, + }, + { + name: "zero timeout disables", + matchers: "alertname=Watchdog", + timeout: 0, + want: []api.DeadmanMatcher{{Name: "Watchdog", Labels: map[string]string{}}}, + enabled: false, + }, + { + name: "no usable matchers disables", + matchers: "", + timeout: time.Hour, + want: nil, + enabled: false, + }, + } + + for _, tc := range tests { + t.Run(tc.name, func(t *testing.T) { + got := api.ParseDeadmanConfig(tc.matchers, tc.timeout, "critical") + if len(got.Matchers) != len(tc.want) { + t.Fatalf("got %d matchers %v, want %d", len(got.Matchers), got.Matchers, len(tc.want)) + } + for i, w := range tc.want { + if got.Matchers[i].Name != w.Name { + t.Errorf("matcher %d: name %q, want %q", i, got.Matchers[i].Name, w.Name) + } + if len(got.Matchers[i].Labels) != len(w.Labels) { + t.Errorf("matcher %d: labels %v, want %v", i, got.Matchers[i].Labels, w.Labels) + continue + } + for k, v := range w.Labels { + if got.Matchers[i].Labels[k] != v { + t.Errorf("matcher %d: label %s=%q, want %q", i, k, got.Matchers[i].Labels[k], v) + } + } + } + }) + } +} + +// A zero config is off, which is what keeps the feature opt-in for anything +// building a router without one. +func TestDeadman_DisabledConfigIsInert(t *testing.T) { + s, _ := deadmanTS(t, api.DeadmanConfig{}) + + heartbeat(t, s, "fp-watchdog", nil) + silence(t, s, "fp-watchdog", 48*time.Hour) + sweep(t, s, time.Hour) + + // Ordinary alert handling: an incident from the arrival, not the absence. + if got := s.countIncidents(t); got != 1 { + t.Fatalf("expected plain alert handling with deadman off, got %d incidents", got) + } + if _, source, _ := s.alertRow(t, "fp-watchdog"); source == nil || *source != "expiry" { + t.Errorf("expected the generic sweeper to own the alert, got %v", source) + } +} diff --git a/internal/api/incident_store.go b/internal/api/incident_store.go index bcc69aa..fef7ed4 100644 --- a/internal/api/incident_store.go +++ b/internal/api/incident_store.go @@ -16,6 +16,11 @@ import ( const ( incidentResolutionAlerts = "alerts" incidentResolutionManual = "manual" + + // incidentResolutionRecovered closes a dead man's switch incident whose + // heartbeat started arriving again. It cannot be "alerts": these incidents + // have no member alerts for the cascade to work from. + incidentResolutionRecovered = "recovered" ) // Incident timeline event types. Stored as free text so adding one later is not @@ -31,6 +36,7 @@ const ( evUnsnoozed = "unsnoozed" evResolved = "resolved" evNote = "note" + evDeadmanSilent = "deadman_silent" ) // severityLabel is the Alertmanager label an incident's severity is derived from. diff --git a/internal/api/incidents_test.go b/internal/api/incidents_test.go index d0e0e2c..30971fb 100644 --- a/internal/api/incidents_test.go +++ b/internal/api/incidents_test.go @@ -613,7 +613,7 @@ func TestSweeper_ArchivesResolvedIncidents(t *testing.T) { s.exec(t, "UPDATE incidents SET resolved_at = ? WHERE id = 1", time.Now().Add(-30*24*time.Hour).Unix()) - api.Sweep(context.Background(), s.db, 7*24*time.Hour, 6*time.Hour) + api.Sweep(context.Background(), s.db, 7*24*time.Hour, 6*time.Hour, s.deadman, s.notify) if inc := getIncident(t, s, 1); inc["archived_at"] == nil { t.Error("expected the sweeper to archive a long-resolved incident") diff --git a/internal/api/notify_test.go b/internal/api/notify_test.go index 91062b6..a2fdb9e 100644 --- a/internal/api/notify_test.go +++ b/internal/api/notify_test.go @@ -438,7 +438,7 @@ func TestNotify_SweepPurgesExpiredAckTokens(t *testing.T) { s.sweepNotify(t) s.exec(t, "UPDATE incident_ack_tokens SET expires_at = ?", time.Now().Add(-time.Minute).Unix()) - api.Sweep(context.Background(), s.db, 168*time.Hour, 6*time.Hour) + api.Sweep(context.Background(), s.db, 168*time.Hour, 6*time.Hour, s.deadman, s.notify) var n int if err := s.db.QueryRow("SELECT COUNT(*) FROM incident_ack_tokens").Scan(&n); err != nil { diff --git a/internal/api/router.go b/internal/api/router.go index fcb4fd1..fc9341a 100644 --- a/internal/api/router.go +++ b/internal/api/router.go @@ -8,10 +8,11 @@ import ( "github.com/go-chi/chi/v5/middleware" ) -// NewRouter builds the HTTP surface. notify is passed through to the webhook, -// the only handler that has to decide where a new incident's page goes; a zero -// value disables notifications. -func NewRouter(db *sql.DB, notify NotifyConfig) http.Handler { +// NewRouter builds the HTTP surface. notify and deadman are passed through to +// the webhook, the only handler that has to decide where a new incident's page +// goes and which arriving alerts are heartbeats rather than problems. A zero +// notify disables notifications; a zero deadman disables dead man's switches. +func NewRouter(db *sql.DB, notify NotifyConfig, deadman DeadmanConfig) http.Handler { r := chi.NewRouter() r.Use(middleware.Logger) r.Use(middleware.Recoverer) @@ -25,7 +26,7 @@ func NewRouter(db *sql.DB, notify NotifyConfig) http.Handler { // the scoped token in its path rather than an API key, and has to stay // reachable from outside the cluster for the button to work. r.Post("/api/bootstrap", handleBootstrap(db)) - r.Post("/api/alertmanager/webhook", handleAlertmanagerWebhook(db, notify)) + r.Post("/api/alertmanager/webhook", handleAlertmanagerWebhook(db, notify, deadman)) r.Post("/api/notify/ack/{token}", handleNotifyAck(db)) // All other /api routes require a valid API key. diff --git a/internal/config/config.go b/internal/config/config.go index 954d0a8..590fdc4 100644 --- a/internal/config/config.go +++ b/internal/config/config.go @@ -15,6 +15,25 @@ type Config struct { // repeat_interval (default 4h), which is what refreshes the alert. StaleAfter time.Duration + // DeadmanMatchers selects the alerts that are heartbeats rather than + // problems: receiving one opens no incident, and the absence of one does. + // + // ";" separates matchers, "," the label conditions within one, "=" is exact + // equality — `alertname=Watchdog,cluster=prod; alertname=Heartbeat`. Every + // matcher must name an alertname. See api.ParseDeadmanConfig. + DeadmanMatchers string + + // DeadmanTimeout is how long a heartbeat may go unheard before its switch is + // declared dead. It must be *shorter* than the Alertmanager repeat_interval + // of the route carrying the heartbeat — the opposite of StaleAfter, and the + // reason a dead man's switch usually wants a route of its own. Zero disables + // dead man's switch handling entirely. + DeadmanTimeout time.Duration + + // DeadmanSeverity is the severity a dead man's switch incident opens at. + // These incidents have no member alerts to derive one from. + DeadmanSeverity string + // NtfyURL is the ntfy server push notifications are published to. Empty // disables notifications entirely. NtfyURL string @@ -44,12 +63,24 @@ func Load() Config { if dbPath == "" { dbPath = "terdut.db" } + deadmanMatchers := os.Getenv("TERDUT_DEADMAN_MATCHERS") + if deadmanMatchers == "" { + deadmanMatchers = "alertname=Watchdog" + } + deadmanSeverity := os.Getenv("TERDUT_DEADMAN_SEVERITY") + if deadmanSeverity == "" { + deadmanSeverity = "critical" + } return Config{ Addr: addr, DBPath: dbPath, ArchiveAfter: duration("TERDUT_ARCHIVE_AFTER", 7*24*time.Hour), StaleAfter: duration("TERDUT_STALE_AFTER", 6*time.Hour), + DeadmanMatchers: deadmanMatchers, + DeadmanTimeout: duration("TERDUT_DEADMAN_TIMEOUT", 15*time.Minute), + DeadmanSeverity: deadmanSeverity, + NtfyURL: os.Getenv("TERDUT_NTFY_URL"), NtfyToken: os.Getenv("TERDUT_NTFY_TOKEN"), NtfyFallbackTopic: os.Getenv("TERDUT_NTFY_FALLBACK_TOPIC"),