Notice when the Watchdog alert stops arriving

Everything this server does assumes alerts arrive. If Prometheus stops
evaluating, or Alertmanager cannot reach us, nothing arrives — and
silence is indistinguishable from everything being fine. The cluster has
shipped the alert for exactly this case all along: Watchdog is
expr: vector(1), so it fires permanently and is re-sent forever, and it
is worth nothing unless something downstream notices it stop. Nothing
did. It arrived, opened no incident because a repeat_interval re-send is
not a new occurrence, and when the monitoring stack died the sweeper
quietly expired it and paged nobody.

So the handling is inverted for a configurable set of alerts: receiving
one opens no incident, and the absence of one does. TERDUT_DEADMAN_MATCHERS
selects them as label matchers, defaulting to alertname=Watchdog.

The unit of monitoring is the fingerprint rather than the alert name. Two
clusters sending the same Watchdog are two independent switches, so a
healthy one can never mask a dead one. Every matcher must name an
alertname, which keeps the sweeper's candidate query on alerts_name_idx
instead of JSON-extracting labels from every row, and leaves matching
with a single implementation.

A switch is dormant until its first heartbeat: a matcher nothing has ever
sent opens nothing, so a fresh deploy or a restored database does not
page. Resolving the incident by hand sticks, exactly as it does for an
alert-backed one, so a decommissioned source is a one-time page rather
than a nag; the switch re-arms only when the heartbeat comes back, and
dying again is a new incident.

The incident has no member alerts on purpose. Linking the heartbeat would
have the settled-incident cascade close it on the very sweep that opened
it, and there is no alert describing the problem anyway — the problem is
that no alert arrived. What happened is on the timeline instead, and
recovery is the only automatic way out.

One narrow exemption in the ingest guard makes recovery possible at all.
A heartbeat we declared dead is marked resolved, and the one that proves
us wrong carries the unchanged startsAt of an alert that never stopped
firing — so "resolution is terminal within an instance" would discard it
forever and a switch could die exactly once. The exemption is scoped to
resolution_source = 'deadman', which is the only resolution this server
infers from silence on a timeout of its own, so nothing another writer
set can be undone by a stale retry. Matched alerts are also held back
from the generic staleness expiry, which would otherwise resolve a
heartbeat as 'expiry' long before its own tighter deadline.

The timeout points the opposite way to TERDUT_STALE_AFTER: staleness is a
generous grace period around a repeat_interval you do not control, while
this is a deadline you set deliberately and configure the heartbeat's
route to beat. Inheriting a 4h or 12h repeat_interval gives a dead man's
switch with a twelve hour fuse, so the README spells out the route the
heartbeat needs.

internal/api/alertmanager.go

@@ -14,6 +14,12 @@ import (
 const (
 	resolutionAlertmanager = "alertmanager"
 	resolutionExpiry       = "expiry"
+
+	// resolutionDeadman marks a heartbeat the dead man's switch sweeper declared
+	// dead. Distinct from expiry because it is load-bearing, not just
+	// descriptive: it is the one resolution the ingest upsert will let a
+	// same-instance re-fire undo, so a switch that comes back can be heard.
+	resolutionDeadman = "deadman"
 )
 
 // amPayload mirrors the Alertmanager webhook v4 payload.
@@ -56,9 +62,15 @@ type ingested struct {
 
 	// justResolved marks the firing → resolved edge, worth a timeline entry.
 	justResolved bool
+
+	// deadman marks a heartbeat: an alert whose arrival means everything is
+	// fine. It is stored like any other alert — received_at is the heartbeat —
+	// but it never reaches an incident. Its absence is what opens one, which
+	// sweepDeadman decides later and elsewhere.
+	deadman bool
 }
 
-func handleAlertmanagerWebhook(db *sql.DB, notify NotifyConfig) http.HandlerFunc {
+func handleAlertmanagerWebhook(db *sql.DB, notify NotifyConfig, deadman DeadmanConfig) http.HandlerFunc {
 	return func(w http.ResponseWriter, r *http.Request) {
 		var payload amPayload
 		if err := decodeJSON(r, &payload); err != nil {
@@ -69,7 +81,7 @@ func handleAlertmanagerWebhook(db *sql.DB, notify NotifyConfig) http.HandlerFunc
 		// Alertmanager retries anything that is not 2xx, and a retry of a payload
 		// we failed to store is more useful than an error it cannot act on — so
 		// failures are logged, not surfaced.
-		if err := ingest(r.Context(), db, notify, payload); err != nil {
+		if err := ingest(r.Context(), db, notify, deadman, payload); err != nil {
 			log.Printf("webhook ingest (group %q): %v", payload.GroupKey, err)
 		}
 
@@ -80,14 +92,14 @@ func handleAlertmanagerWebhook(db *sql.DB, notify NotifyConfig) http.HandlerFunc
 // ingest stores a payload's alerts and reconciles the incident for its group.
 // The whole payload is one transaction: an incident that opened but whose alerts
 // failed to link would be a work item nobody could act on.
-func ingest(ctx context.Context, db *sql.DB, notify NotifyConfig, payload amPayload) error {
+func ingest(ctx context.Context, db *sql.DB, notify NotifyConfig, deadman DeadmanConfig, payload amPayload) error {
 	tx, err := db.BeginTx(ctx, nil)
 	if err != nil {
 		return err
 	}
 	defer tx.Rollback() //nolint:errcheck
 
-	accepted, err := upsertAlerts(ctx, tx, payload.Alerts)
+	accepted, err := upsertAlerts(ctx, tx, deadman, payload.Alerts)
 	if err != nil {
 		return err
 	}
@@ -103,7 +115,7 @@ func ingest(ctx context.Context, db *sql.DB, notify NotifyConfig, payload amPayl
 	if incidentID != 0 {
 		touched[incidentID] = true
 		for _, a := range accepted {
-			if !a.firing {
+			if !a.firing || a.deadman {
 				continue
 			}
 			if err := linkAlert(ctx, tx, incidentID, a.id); err != nil {
@@ -113,7 +125,7 @@ func ingest(ctx context.Context, db *sql.DB, notify NotifyConfig, payload amPayl
 	}
 
 	for _, a := range accepted {
-		if !a.justResolved {
+		if !a.justResolved || a.deadman {
 			continue
 		}
 		id, err := openIncidentForAlert(ctx, tx, a.id)
@@ -144,7 +156,7 @@ func ingest(ctx context.Context, db *sql.DB, notify NotifyConfig, payload amPayl
 
 // upsertAlerts stores each alert of a payload and reports what changed. Payloads
 // the ordering guard rejected are left out entirely.
-func upsertAlerts(ctx context.Context, tx *sql.Tx, alerts []amAlert) ([]ingested, error) {
+func upsertAlerts(ctx context.Context, tx *sql.Tx, deadman DeadmanConfig, alerts []amAlert) ([]ingested, error) {
 	now := time.Now().Unix()
 	accepted := make([]ingested, 0, len(alerts))
 
@@ -187,7 +199,15 @@ func upsertAlerts(ctx context.Context, tx *sql.Tx, alerts []amAlert) ([]ingested
 		// older than the stored one. Alertmanager retries failed notifications,
 		// so a stale firing retry can arrive after the resolved one; it carries
 		// the same startsAt, whereas a genuine re-fire carries a newer one.
-		// Within a single instance, resolution is terminal.
+		// Within a single instance, resolution is terminal — with one exception.
+		//
+		// A resolution this server synthesised for a dead man's switch is not
+		// Alertmanager's word that the instance ended; it is our inference from
+		// silence. The heartbeat that proves us wrong carries the unchanged
+		// startsAt of an alert that never stopped firing, so without the
+		// exemption a switch could go dead exactly once and never be heard from
+		// again. Scoped to 'deadman' so no resolution anybody else wrote can be
+		// undone by a stale retry.
 		if _, err := tx.ExecContext(ctx, `
 			INSERT INTO alerts
 				(fingerprint, name, status, labels, annotations, starts_at, ends_at,
@@ -211,7 +231,8 @@ func upsertAlerts(ctx context.Context, tx *sql.Tx, alerts []amAlert) ([]ingested
 				                        THEN NULL ELSE alerts.archived_at END
 			WHERE excluded.starts_at > alerts.starts_at
 			   OR (excluded.starts_at = alerts.starts_at
-			       AND NOT (alerts.status = 'resolved' AND excluded.status = 'firing'))`,
+			       AND (alerts.resolution_source = '`+resolutionDeadman+`'
+			            OR NOT (alerts.status = 'resolved' AND excluded.status = 'firing')))`,
 			a.Fingerprint, name, a.Status,
 			string(labelsJSON), string(annotationsJSON),
 			a.StartsAt.Unix(), endsAtUnix,
@@ -244,6 +265,7 @@ func upsertAlerts(ctx context.Context, tx *sql.Tx, alerts []amAlert) ([]ingested
 			firing:        firing,
 			newOccurrence: firing && (!existed || a.StartsAt.Unix() > prevStartsAt || prevStatus == "resolved"),
 			justResolved:  !firing && existed && prevStatus == "firing",
+			deadman:       deadman.isDeadman(a.Labels),
 		})
 	}
 
@@ -258,10 +280,17 @@ func upsertAlerts(ctx context.Context, tx *sql.Tx, alerts []amAlert) ([]ingested
 // something actually started firing. Without that, a manually resolved incident
 // would reappear on the next repeat_interval re-send of an alert that never
 // stopped, and manual resolution would be meaningless.
+//
+// Heartbeats do not count as anything here. A group of nothing but dead man's
+// switch alerts opens no incident at all, and a mixed group gets an incident for
+// its real alerts only.
 func incidentForGroup(ctx context.Context, tx *sql.Tx, notify NotifyConfig, payload amPayload, accepted []ingested) (int64, error) {
 	var firstName string
 	anyFiring, anyNew := false, false
 	for _, a := range accepted {
+		if a.deadman {
+			continue
+		}
 		if a.firing {
 			if !anyFiring {
 				firstName = a.name
@@ -297,13 +326,20 @@ func incidentForGroup(ctx context.Context, tx *sql.Tx, notify NotifyConfig, payl
 	if !anyNew {
 		return 0, nil
 	}
-	return openIncident(ctx, tx, notify, groupKey, payload.GroupLabels, firstName)
+	return openIncident(ctx, tx, notify, groupKey,
+		incidentTitle(payload.GroupLabels, firstName), payload.GroupLabels, nil)
 }
 
-// openIncident creates an incident for a group and assigns it to whoever is on
-// call today, which is the point at which the schedule stops being decorative.
-func openIncident(ctx context.Context, tx *sql.Tx, notify NotifyConfig, groupKey string, groupLabels map[string]string, fallbackName string) (int64, error) {
-	onCall, err := currentOnCall(ctx, tx)
+// openIncident creates an incident and assigns it to whoever is on call today,
+// which is the point at which the schedule stops being decorative.
+//
+// The one place an incident is born, for both of the things that can raise one:
+// the webhook, inside its transaction, and the dead man's switch sweeper, inside
+// its own. Hence the querier rather than a *sql.Tx. A nil severity leaves the
+// column for refreshSeverity to fill from the member alerts; the sweeper passes
+// one because its incidents have no members to derive it from.
+func openIncident(ctx context.Context, q querier, notify NotifyConfig, groupKey, title string, groupLabels map[string]string, severity *string) (int64, error) {
+	onCall, err := currentOnCall(ctx, q)
 	if err != nil {
 		return 0, err
 	}
@@ -313,10 +349,10 @@ func openIncident(ctx context.Context, tx *sql.Tx, notify NotifyConfig, groupKey
 		labelsJSON = []byte("{}")
 	}
 
-	res, err := tx.ExecContext(ctx, `
-		INSERT INTO incidents (group_key, title, group_labels, status, triggered_at, assigned_to)
-		VALUES (?, ?, ?, 'triggered', ?, ?)`,
-		groupKey, incidentTitle(groupLabels, fallbackName), string(labelsJSON),
+	res, err := q.ExecContext(ctx, `
+		INSERT INTO incidents (group_key, title, group_labels, status, severity, triggered_at, assigned_to)
+		VALUES (?, ?, ?, 'triggered', ?, ?, ?)`,
+		groupKey, title, string(labelsJSON), severity,
 		time.Now().Unix(), onCall)
 	if err != nil {
 		return 0, err
@@ -326,20 +362,20 @@ func openIncident(ctx context.Context, tx *sql.Tx, notify NotifyConfig, groupKey
 		return 0, err
 	}
 
-	if err := logEvent(ctx, tx, id, evTriggered, nil, nil, nil); err != nil {
+	if err := logEvent(ctx, q, id, evTriggered, nil, nil, nil); err != nil {
 		return 0, err
 	}
 	if onCall != nil {
 		// On an "assigned" event user_id is the assignee, not the actor.
-		if err := logEvent(ctx, tx, id, evAssigned, onCall, nil, nil); err != nil {
+		if err := logEvent(ctx, q, id, evAssigned, onCall, nil, nil); err != nil {
 			return 0, err
 		}
 	}
 
-	// Queue the page, but do not send it here: this runs inside the webhook's
-	// transaction on a single-connection pool, so an HTTP call would hold up
-	// every other request. The notifier picks the row up within a tick.
-	if err := enqueueOpened(ctx, tx, notify, id, onCall); err != nil {
+	// Queue the page, but do not send it here: this runs inside a transaction on
+	// a single-connection pool, so an HTTP call would hold up every other
+	// request. The notifier picks the row up within a tick.
+	if err := enqueueOpened(ctx, q, notify, id, onCall); err != nil {
 		return 0, err
 	}
 	return id, nil

internal/api/api_test.go

@@ -20,19 +20,31 @@ import (
 // tests can age rows directly — the sweeper's inputs are wall-clock timestamps.
 type ts struct {
 	*httptest.Server
-	key    string
-	db     *sql.DB
-	notify api.NotifyConfig
+	key     string
+	db      *sql.DB
+	notify  api.NotifyConfig
+	deadman api.DeadmanConfig
 }
 
 // newTS builds a server over a fresh in-memory database. Notifications are off
 // unless a NotifyConfig is passed, so tests that predate them are unaffected.
+// Dead man's switches are off too — see newDeadmanTS.
 func newTS(t *testing.T, notify ...api.NotifyConfig) *ts {
 	t.Helper()
 	var cfg api.NotifyConfig
 	if len(notify) > 0 {
 		cfg = notify[0]
 	}
+	return newDeadmanTS(t, api.DeadmanConfig{}, cfg)
+}
+
+// newDeadmanTS is newTS with dead man's switch handling configured.
+func newDeadmanTS(t *testing.T, deadman api.DeadmanConfig, notify ...api.NotifyConfig) *ts {
+	t.Helper()
+	var cfg api.NotifyConfig
+	if len(notify) > 0 {
+		cfg = notify[0]
+	}
 
 	database, err := db.Open(":memory:")
 	if err != nil {
@@ -41,7 +53,7 @@ func newTS(t *testing.T, notify ...api.NotifyConfig) *ts {
 	if err := db.Migrate(database); err != nil {
 		t.Fatalf("migrate: %v", err)
 	}
-	srv := httptest.NewServer(api.NewRouter(database, cfg))
+	srv := httptest.NewServer(api.NewRouter(database, cfg, deadman))
 	t.Cleanup(func() { srv.Close(); database.Close() })
 
 	body, _ := json.Marshal(map[string]string{"username": "admin", "email": "admin@test.com"})
@@ -57,7 +69,7 @@ func newTS(t *testing.T, notify ...api.NotifyConfig) *ts {
 	json.NewDecoder(resp.Body).Decode(&result)
 	key := result["api_key"].(map[string]any)["key"].(string)
 
-	return &ts{Server: srv, key: key, db: database, notify: cfg}
+	return &ts{Server: srv, key: key, db: database, notify: cfg, deadman: deadman}
 }
 
 // exec runs a statement against the test database.
@@ -493,7 +505,7 @@ func TestArchive_AlertListFilter(t *testing.T) {
 	}
 
 	// 2. Let the sweeper archive it: ends_at is already well past archiveAfter.
-	api.Sweep(context.Background(), s.db, time.Hour, 6*time.Hour)
+	api.Sweep(context.Background(), s.db, time.Hour, 6*time.Hour, s.deadman, s.notify)
 
 	// 3. Default list excludes it.
 	decode(t, s.req(t, http.MethodGet, "/api/alerts", nil), &alerts)
@@ -534,7 +546,7 @@ func postAlert(t *testing.T, s *ts, fingerprint, status, startsAt, endsAt string
 
 func sweep(t *testing.T, s *ts, staleAfter time.Duration) {
 	t.Helper()
-	api.Sweep(context.Background(), s.db, noArchive, staleAfter)
+	api.Sweep(context.Background(), s.db, noArchive, staleAfter, s.deadman, s.notify)
 }
 
 // A firing alert Alertmanager stopped refreshing is resolved via the

internal/api/archiver.go

@@ -19,28 +19,34 @@ const (
 
 // StartArchiver runs the alert sweeper until ctx is cancelled, starting with an
 // immediate pass so a restart reconciles state right away.
-func StartArchiver(ctx context.Context, db *sql.DB, archiveAfter, staleAfter time.Duration) {
+func StartArchiver(ctx context.Context, db *sql.DB, archiveAfter, staleAfter time.Duration, deadman DeadmanConfig, notify NotifyConfig) {
 	ticker := time.NewTicker(sweepInterval)
 	defer ticker.Stop()
 
-	Sweep(ctx, db, archiveAfter, staleAfter)
+	Sweep(ctx, db, archiveAfter, staleAfter, deadman, notify)
 	for {
 		select {
 		case <-ticker.C:
-			Sweep(ctx, db, archiveAfter, staleAfter)
+			Sweep(ctx, db, archiveAfter, staleAfter, deadman, notify)
 		case <-ctx.Done():
 			return
 		}
 	}
 }
 
-// Sweep runs a single pass, in dependency order: expire stale firing alerts,
-// close the incidents that leaves with nothing firing, then archive whatever has
-// been settled long enough. Running them in one pass means an alert can go stale
-// and its incident can close and archive without waiting three ticks.
+// Sweep runs a single pass, in dependency order: reconcile the dead man's
+// switches, expire stale firing alerts, close the incidents that leaves with
+// nothing firing, then archive whatever has been settled long enough. Running
+// them in one pass means an alert can go stale and its incident can close and
+// archive without waiting three ticks.
+//
+// The switches go first because they hand expireStale the alerts it must not
+// touch: a heartbeat answers to its own, much tighter, timeout, and the generic
+// staleness rules would otherwise resolve it as 'expiry' long before that.
 // Exported so tests can drive a pass without waiting on the ticker.
-func Sweep(ctx context.Context, db *sql.DB, archiveAfter, staleAfter time.Duration) {
-	expireStale(ctx, db, staleAfter)
+func Sweep(ctx context.Context, db *sql.DB, archiveAfter, staleAfter time.Duration, deadman DeadmanConfig, notify NotifyConfig) {
+	heartbeats := sweepDeadman(ctx, db, deadman, notify)
+	expireStale(ctx, db, staleAfter, heartbeats)
 	resolveSettledIncidents(ctx, db)
 	archiveResolved(ctx, db, archiveAfter)
 	archiveResolvedIncidents(ctx, db, archiveAfter)
@@ -60,16 +66,26 @@ func Sweep(ctx context.Context, db *sql.DB, archiveAfter, staleAfter time.Durati
 //     notifications every repeat_interval, making received_at a liveness
 //     heartbeat — provided staleAfter exceeds that interval.
 //
+// Alerts in skip are left alone: they are dead man's switch heartbeats, whose
+// liveness sweepDeadman has already judged against a timeout of its own.
+//
 // The matching rows are collected before the update rather than updated in bulk,
 // because each one owes its incident a timeline entry.
-func expireStale(ctx context.Context, db *sql.DB, staleAfter time.Duration) {
+func expireStale(ctx context.Context, db *sql.DB, staleAfter time.Duration, skip map[int64]bool) {
 	now := time.Now()
 
-	ids, err := staleAlertIDs(ctx, db, now, staleAfter)
+	found, err := staleAlertIDs(ctx, db, now, staleAfter)
 	if err != nil {
 		log.Printf("sweeper: find stale: %v", err)
 		return
 	}
+
+	ids := make([]int64, 0, len(found))
+	for _, id := range found {
+		if !skip[id] {
+			ids = append(ids, id)
+		}
+	}
 	if len(ids) == 0 {
 		return
 	}

internal/api/deadman.go

@@ -0,0 +1,378 @@
+package api
+
+import (
+	"context"
+	"database/sql"
+	"encoding/json"
+	"log"
+	"sort"
+	"strings"
+	"time"
+)
+
+// deadmanGroupPrefix namespaces the incidents this file opens. Alertmanager
+// group keys always contain braces, so this can never collide with one, and the
+// partial unique index on open group_key (see 008_incidents.sql) gives one open
+// incident per switch for free.
+const deadmanGroupPrefix = "deadman:"
+
+// DeadmanMatcher selects the alerts that are heartbeats rather than problems.
+// Every condition has to match, and Name — the alertname label — is mandatory:
+// it is what lets the sweeper find candidate rows through alerts_name_idx
+// instead of JSON-extracting labels from every row in the table.
+type DeadmanMatcher struct {
+	Name   string
+	Labels map[string]string
+}
+
+// String renders the matcher the way it was configured, which is also how it
+// reads in an incident title.
+func (m DeadmanMatcher) String() string {
+	if len(m.Labels) == 0 {
+		return m.Name
+	}
+	parts := make([]string, 0, len(m.Labels))
+	for k, v := range m.Labels {
+		parts = append(parts, k+"="+v)
+	}
+	sort.Strings(parts)
+	return m.Name + " (" + strings.Join(parts, ", ") + ")"
+}
+
+// matches reports whether an alert's labels satisfy every condition.
+func (m DeadmanMatcher) matches(labels map[string]string) bool {
+	if labels["alertname"] != m.Name {
+		return false
+	}
+	for k, v := range m.Labels {
+		if labels[k] != v {
+			return false
+		}
+	}
+	return true
+}
+
+// DeadmanConfig inverts the handling of the alerts it matches: receiving one
+// opens nothing, and the absence of one opens an incident.
+//
+// The unit of monitoring is the fingerprint, not the matcher — two clusters
+// sending the same heartbeat alertname are two independent switches, so one
+// healthy cluster cannot mask a dead one.
+type DeadmanConfig struct {
+	Matchers []DeadmanMatcher
+
+	// Timeout is how long a matched alert may go without a refreshing webhook
+	// before it is declared dead. It must be shorter than Alertmanager's
+	// repeat_interval for the heartbeat's route, which is what refreshes it.
+	// Zero disables dead man's switch handling entirely.
+	Timeout time.Duration
+
+	// Severity is the severity every dead man's switch incident opens at. These
+	// incidents have no member alerts to derive one from, and the heartbeat's
+	// own severity label is meaningless — Watchdog ships as "none".
+	Severity string
+}
+
+// enabled reports whether there is anything to watch.
+func (c DeadmanConfig) enabled() bool { return c.Timeout > 0 && len(c.Matchers) > 0 }
+
+// match returns the first matcher an alert satisfies.
+func (c DeadmanConfig) match(labels map[string]string) (DeadmanMatcher, bool) {
+	if !c.enabled() {
+		return DeadmanMatcher{}, false
+	}
+	for _, m := range c.Matchers {
+		if m.matches(labels) {
+			return m, true
+		}
+	}
+	return DeadmanMatcher{}, false
+}
+
+// isDeadman is match without the matcher, for the ingest path.
+func (c DeadmanConfig) isDeadman(labels map[string]string) bool {
+	_, ok := c.match(labels)
+	return ok
+}
+
+// names lists the distinct alertnames worth loading from the database.
+func (c DeadmanConfig) names() []string {
+	seen := map[string]bool{}
+	out := make([]string, 0, len(c.Matchers))
+	for _, m := range c.Matchers {
+		if !seen[m.Name] {
+			seen[m.Name] = true
+			out = append(out, m.Name)
+		}
+	}
+	return out
+}
+
+// ParseDeadmanConfig reads the matcher list from its configured form:
+// ";" separates matchers, "," separates the conditions within one, and "=" is
+// exact label equality — `alertname=Watchdog,cluster=prod; alertname=Heartbeat`.
+//
+// A malformed or alertname-less entry is dropped rather than fatal, following
+// config.duration's rule that one bad tuning knob should not take the server
+// down. Silence would be worse here than elsewhere, though — a typo that
+// disarms the switch is exactly the failure this feature exists to catch — so
+// the matchers that survived are logged.
+func ParseDeadmanConfig(matchers string, timeout time.Duration, severity string) DeadmanConfig {
+	cfg := DeadmanConfig{Timeout: timeout, Severity: severity}
+
+	for _, entry := range strings.Split(matchers, ";") {
+		entry = strings.TrimSpace(entry)
+		if entry == "" {
+			continue
+		}
+
+		m := DeadmanMatcher{Labels: map[string]string{}}
+		malformed := false
+		for _, cond := range strings.Split(entry, ",") {
+			k, v, ok := strings.Cut(cond, "=")
+			k, v = strings.TrimSpace(k), strings.TrimSpace(v)
+			if !ok || k == "" || v == "" {
+				log.Printf("deadman: ignoring matcher %q: %q is not label=value", entry, strings.TrimSpace(cond))
+				malformed = true
+				break
+			}
+			if k == "alertname" {
+				m.Name = v
+				continue
+			}
+			m.Labels[k] = v
+		}
+		if malformed {
+			continue
+		}
+		if m.Name == "" {
+			log.Printf("deadman: ignoring matcher %q: no alertname condition", entry)
+			continue
+		}
+		cfg.Matchers = append(cfg.Matchers, m)
+	}
+
+	switch {
+	case timeout <= 0:
+		log.Print("deadman: disabled (timeout is zero)")
+	case len(cfg.Matchers) == 0:
+		log.Print("deadman: disabled (no usable matchers)")
+	default:
+		rendered := make([]string, 0, len(cfg.Matchers))
+		for _, m := range cfg.Matchers {
+			rendered = append(rendered, m.String())
+		}
+		log.Printf("deadman: watching %s, timeout %s, severity %s",
+			strings.Join(rendered, "; "), timeout, severity)
+	}
+	return cfg
+}
+
+// deadmanAlert is one switch: the alert row carrying its last heartbeat.
+type deadmanAlert struct {
+	id          int64
+	fingerprint string
+	labels      map[string]string
+	matcher     DeadmanMatcher
+	resolved    bool
+	receivedAt  int64
+}
+
+// groupKey is the switch's identity as an incident. Per fingerprint, so each
+// source is tracked on its own.
+func (a deadmanAlert) groupKey() string { return deadmanGroupPrefix + a.fingerprint }
+
+// sweepDeadman is the whole point of the feature: it opens an incident for every
+// switch that has stopped chirping, and closes one whose switch came back.
+//
+// It returns the ids of the alerts it owns, because the generic staleness
+// expiry must leave them alone — staleAfter and ends_at would otherwise resolve
+// a heartbeat long before its own, much tighter, timeout ever fired.
+func sweepDeadman(ctx context.Context, db *sql.DB, cfg DeadmanConfig, notify NotifyConfig) map[int64]bool {
+	owned := map[int64]bool{}
+	if !cfg.enabled() {
+		return owned
+	}
+
+	switches, err := deadmanAlerts(ctx, db, cfg)
+	if err != nil {
+		log.Printf("deadman: load switches: %v", err)
+		return owned
+	}
+
+	now := time.Now()
+	cutoff := now.Add(-cfg.Timeout).Unix()
+
+	for _, sw := range switches {
+		owned[sw.id] = true
+
+		// An explicit resolved from Alertmanager is a stronger death signal than
+		// mere absence: the sender is telling us the heartbeat stopped, so there
+		// is nothing left to wait out.
+		if sw.resolved || sw.receivedAt < cutoff {
+			if err := deadmanDied(ctx, db, cfg, notify, sw, now); err != nil {
+				log.Printf("deadman: open incident for %s: %v", sw.matcher.Name, err)
+			}
+			continue
+		}
+		if err := deadmanRecovered(ctx, db, sw); err != nil {
+			log.Printf("deadman: resolve incident for %s: %v", sw.matcher.Name, err)
+		}
+	}
+	return owned
+}
+
+// deadmanAlerts loads every alert row that a matcher claims. The candidate query
+// is narrowed by alertname so it rides alerts_name_idx; the rest of the matching
+// happens in Go, which keeps one implementation of the rules. The rows are read
+// in full before the caller writes, because the pool holds a single connection.
+func deadmanAlerts(ctx context.Context, db *sql.DB, cfg DeadmanConfig) ([]deadmanAlert, error) {
+	names := cfg.names()
+	args := make([]any, 0, len(names))
+	for _, n := range names {
+		args = append(args, n)
+	}
+
+	rows, err := db.QueryContext(ctx, `
+		SELECT id, fingerprint, labels, status, received_at
+		FROM   alerts
+		WHERE  name IN (`+placeholders(len(names))+`)
+		  AND  archived_at IS NULL`, args...)
+	if err != nil {
+		return nil, err
+	}
+	defer rows.Close()
+
+	var out []deadmanAlert
+	for rows.Next() {
+		var a deadmanAlert
+		var labelsJSON, status string
+		if err := rows.Scan(&a.id, &a.fingerprint, &labelsJSON, &status, &a.receivedAt); err != nil {
+			return nil, err
+		}
+		json.Unmarshal([]byte(labelsJSON), &a.labels) //nolint:errcheck
+
+		m, ok := cfg.match(a.labels)
+		if !ok {
+			continue
+		}
+		a.matcher = m
+		a.resolved = status == "resolved"
+		out = append(out, a)
+	}
+	return out, rows.Err()
+}
+
+// deadmanDied raises the incident for a switch that has gone quiet.
+//
+// Two conditions gate it, and both matter. There must be no open incident for
+// the switch already — the partial unique index enforces that anyway, but a
+// second one would be a wasted page. And the heartbeat must have been seen since
+// the last incident was raised, which is the re-arm rule: resolving a dead man's
+// switch incident sticks, exactly as resolving an alert-backed one does (see
+// incidentForGroup), and a source that is gone for good is a one-time page
+// rather than a nag. Only a heartbeat that comes back and dies again earns a new
+// incident.
+func deadmanDied(ctx context.Context, db *sql.DB, cfg DeadmanConfig, notify NotifyConfig, sw deadmanAlert, now time.Time) error {
+	var lastTriggered, open int64
+	if err := db.QueryRowContext(ctx, `
+		SELECT COALESCE(MAX(triggered_at), 0),
+		       COALESCE(SUM(resolved_at IS NULL), 0)
+		FROM   incidents WHERE group_key = ?`,
+		sw.groupKey()).Scan(&lastTriggered, &open); err != nil {
+		return err
+	}
+	if open > 0 || sw.receivedAt <= lastTriggered {
+		return nil
+	}
+
+	tx, err := db.BeginTx(ctx, nil)
+	if err != nil {
+		return err
+	}
+	defer tx.Rollback() //nolint:errcheck
+
+	// A heartbeat nobody has heard from is not firing, and saying otherwise in
+	// the alert list would be a lie. An Alertmanager-sourced resolution keeps its
+	// own source: it told us the truth first.
+	if !sw.resolved {
+		if _, err := tx.ExecContext(ctx, `
+			UPDATE alerts
+			   SET status            = 'resolved',
+			       resolution_source = ?,
+			       ends_at           = COALESCE(ends_at, unixepoch())
+			 WHERE id = ? AND status = 'firing'`, resolutionDeadman, sw.id); err != nil {
+			return err
+		}
+	}
+
+	severity := cfg.Severity
+	var sev *string
+	if severity != "" {
+		sev = &severity
+	}
+
+	incidentID, err := openIncident(ctx, tx, notify, sw.groupKey(),
+		"No heartbeat from "+sw.matcher.String(), sw.labels, sev)
+	if err != nil {
+		return err
+	}
+
+	alertID := sw.id
+	detail := "last heartbeat " + humanDuration(now.Sub(time.Unix(sw.receivedAt, 0))) + " ago"
+	if err := logEvent(ctx, tx, incidentID, evDeadmanSilent, nil, &alertID, &detail); err != nil {
+		return err
+	}
+
+	if err := tx.Commit(); err != nil {
+		return err
+	}
+	log.Printf("deadman: %s went silent, opened incident %d", sw.matcher.String(), incidentID)
+	return nil
+}
+
+// deadmanRecovered closes the incident for a switch that started chirping again.
+//
+// It cannot go through resolveIfSettled: a dead man's switch incident has no
+// member alerts (linking the heartbeat would have the settled-incident cascade
+// close it on the very same sweep that opened it), so the alert-driven cascade
+// ignores it entirely and recovery is the only automatic way out.
+func deadmanRecovered(ctx context.Context, db *sql.DB, sw deadmanAlert) error {
+	var incidentID int64
+	switch err := db.QueryRowContext(ctx, `
+		SELECT id FROM incidents
+		 WHERE group_key = ? AND resolved_at IS NULL`, sw.groupKey()).Scan(&incidentID); {
+	case err == sql.ErrNoRows:
+		return nil
+	case err != nil:
+		return err
+	}
+
+	tx, err := db.BeginTx(ctx, nil)
+	if err != nil {
+		return err
+	}
+	defer tx.Rollback() //nolint:errcheck
+
+	if _, err := tx.ExecContext(ctx, `
+		UPDATE incidents
+		   SET status = 'resolved', resolved_at = ?, resolution_source = ?
+		 WHERE id = ? AND resolved_at IS NULL`,
+		time.Now().Unix(), incidentResolutionRecovered, incidentID); err != nil {
+		return err
+	}
+	if err := logEvent(ctx, tx, incidentID, evResolved, nil, nil, nil); err != nil {
+		return err
+	}
+	// The all-clear goes to whoever was paged, which enqueueResolved works out
+	// from the incident's own notification history.
+	if err := enqueueResolved(ctx, tx, incidentID); err != nil {
+		return err
+	}
+
+	if err := tx.Commit(); err != nil {
+		return err
+	}
+	log.Printf("deadman: %s is back, resolved incident %d", sw.matcher.String(), incidentID)
+	return nil
+}

internal/api/deadman_test.go

@@ -0,0 +1,468 @@
+package api_test
+
+import (
+	"net/http"
+	"testing"
+	"time"
+
+	"github.com/yeniklas/terdut-server/internal/api"
+)
+
+// ---------------------------------------------------------------------------
+// Harness
+// ---------------------------------------------------------------------------
+
+// watchdogGroupKey is what Alertmanager sends for a Watchdog grouped by
+// alertname, which is how the deployed route is configured.
+const watchdogGroupKey = `{}:{alertname="Watchdog"}`
+
+// deadmanCfg watches Watchdog with a timeout short enough to reason about and
+// long enough that a fresh heartbeat is never accidentally stale.
+func deadmanCfg() api.DeadmanConfig {
+	return api.ParseDeadmanConfig("alertname=Watchdog", time.Hour, "critical")
+}
+
+// deadmanTS is notifyTS with dead man's switch handling on: notifications
+// enabled against a fake ntfy, the admin on call today with a topic.
+func deadmanTS(t *testing.T, cfg api.DeadmanConfig) (*ts, *fakeNtfy) {
+	t.Helper()
+	f := newFakeNtfy(t)
+	s := newDeadmanTS(t, cfg, api.NotifyConfig{
+		BaseURL:   f.URL,
+		PublicURL: "https://terdut.example.com",
+	})
+
+	putOnCall(t, s, 1)
+	setTopic(t, s, 1, "terdut-admin")
+	return s, f
+}
+
+// heartbeat posts one Watchdog webhook. Its startsAt never changes: a dead man's
+// switch alert fires once and is re-sent unchanged forever, which is precisely
+// what makes its absence meaningful.
+func heartbeat(t *testing.T, s *ts, fingerprint string, labels map[string]string) {
+	t.Helper()
+	postWebhook(t, s, []map[string]any{
+		amAlert(fingerprint, "Watchdog", "firing", "2026-05-20T10:00:00Z", zeroTime, labels),
+	}, watchdogGroupKey)
+}
+
+// silence back-dates a heartbeat's received_at, which is the only clock the
+// sweeper reads. There is no fake clock in this package.
+func silence(t *testing.T, s *ts, fingerprint string, ago time.Duration) {
+	t.Helper()
+	s.exec(t, "UPDATE alerts SET received_at = ? WHERE fingerprint = ?",
+		time.Now().Add(-ago).Unix(), fingerprint)
+}
+
+// ageIncidents back-dates every incident. The re-arm rule compares a heartbeat
+// against the last incident raised for its switch, so a test that wants a second
+// episode has to put the first one in the past — there is no fake clock here.
+func ageIncidents(t *testing.T, s *ts, ago time.Duration) {
+	t.Helper()
+	past := time.Now().Add(-ago).Unix()
+	s.exec(t, `UPDATE incidents
+	              SET triggered_at = ?,
+	                  resolved_at  = CASE WHEN resolved_at IS NULL THEN NULL ELSE ? END`,
+		past, past)
+}
+
+// incidentByGroup reads the incident for a group key, resolved ones included.
+func incidentByGroup(t *testing.T, s *ts, groupKey string) (id int64, status, severity string, source *string) {
+	t.Helper()
+	err := s.db.QueryRow(`
+		SELECT id, status, COALESCE(severity, ''), resolution_source
+		FROM   incidents WHERE group_key = ? ORDER BY id DESC LIMIT 1`,
+		groupKey).Scan(&id, &status, &severity, &source)
+	if err != nil {
+		t.Fatalf("read incident for group %s: %v", groupKey, err)
+	}
+	return id, status, severity, source
+}
+
+// ---------------------------------------------------------------------------
+// Receiving a heartbeat
+// ---------------------------------------------------------------------------
+
+// The whole inversion: arrival of a dead man's switch alert is good news, and
+// good news is not an incident.
+func TestDeadman_HeartbeatOpensNoIncident(t *testing.T) {
+	s, _ := deadmanTS(t, deadmanCfg())
+
+	heartbeat(t, s, "fp-watchdog", nil)
+
+	if got := s.countIncidents(t); got != 0 {
+		t.Fatalf("expected a heartbeat to open no incident, got %d", got)
+	}
+	if got := s.countNotifications(t, ""); got != 0 {
+		t.Errorf("expected no notification for a heartbeat, got %d", got)
+	}
+	if status, _, _ := s.alertRow(t, "fp-watchdog"); status != "firing" {
+		t.Errorf("expected the heartbeat to be stored firing, got %q", status)
+	}
+}
+
+// A heartbeat routed into a group alongside real alerts must not join their
+// incident: it is not a symptom of anything.
+func TestDeadman_MixedGroupExcludesHeartbeat(t *testing.T) {
+	s, _ := deadmanTS(t, deadmanCfg())
+
+	postWebhook(t, s, []map[string]any{
+		amAlert("fp-mixed-wd", "Watchdog", "firing", "2026-05-20T10:00:00Z", zeroTime, nil),
+		amAlert("fp-mixed-disk", "DiskFull", "firing", "2026-05-20T10:00:00Z", zeroTime,
+			map[string]string{"severity": "critical"}),
+	}, `{}:{namespace="prod"}`)
+
+	if got := s.countIncidents(t); got != 1 {
+		t.Fatalf("expected 1 incident for the real alert, got %d", got)
+	}
+
+	var alerts []map[string]any
+	decode(t, s.req(t, http.MethodGet, "/api/incidents/1/alerts", nil), &alerts)
+	if len(alerts) != 1 {
+		t.Fatalf("expected 1 member alert, got %d", len(alerts))
+	}
+	if name := alerts[0]["name"]; name != "DiskFull" {
+		t.Errorf("expected only the real alert linked, got %v", name)
+	}
+}
+
+// A matcher scoped by label only claims the alerts it names, so a heartbeat from
+// somewhere else stays an ordinary alert.
+func TestDeadman_LabelScopedMatcherIgnoresOthers(t *testing.T) {
+	s, _ := deadmanTS(t, api.ParseDeadmanConfig("alertname=Watchdog,cluster=prod", time.Hour, "critical"))
+
+	heartbeat(t, s, "fp-dev", map[string]string{"cluster": "dev"})
+
+	if got := s.countIncidents(t); got != 1 {
+		t.Fatalf("expected an unmatched Watchdog to behave like any other alert, got %d incidents", got)
+	}
+}
+
+// ---------------------------------------------------------------------------
+// Silence
+// ---------------------------------------------------------------------------
+
+func TestDeadman_SilenceOpensIncident(t *testing.T) {
+	s, f := deadmanTS(t, deadmanCfg())
+
+	heartbeat(t, s, "fp-watchdog", nil)
+	silence(t, s, "fp-watchdog", 2*time.Hour)
+	sweep(t, s, noArchive)
+
+	if got := s.countIncidents(t); got != 1 {
+		t.Fatalf("expected silence to open 1 incident, got %d", got)
+	}
+	id, status, severity, _ := incidentByGroup(t, s, "deadman:fp-watchdog")
+	if status != "triggered" {
+		t.Errorf("expected a triggered incident, got %q", status)
+	}
+	if severity != "critical" {
+		t.Errorf("expected the configured severity, got %q", severity)
+	}
+
+	// The alert list must not keep claiming a dead heartbeat is firing.
+	alertStatus, source, _ := s.alertRow(t, "fp-watchdog")
+	if alertStatus != "resolved" || source == nil || *source != "deadman" {
+		t.Errorf("expected the heartbeat resolved as deadman, got %q / %v", alertStatus, source)
+	}
+
+	// Nobody was told anything by an alert here, so the page has to come from
+	// the switch itself.
+	s.sweepNotify(t)
+	msgs := f.messages()
+	if len(msgs) != 1 {
+		t.Fatalf("expected 1 page, got %d", len(msgs))
+	}
+	if msgs[0].Topic != "terdut-admin" {
+		t.Errorf("expected the on-call topic, got %q", msgs[0].Topic)
+	}
+	if msgs[0].Priority != 5 {
+		t.Errorf("expected a critical page to override quiet hours (priority 5), got %d", msgs[0].Priority)
+	}
+
+	// The timeline says why, with the age of the last heartbeat.
+	types := eventTypes(timeline(t, s, int(id)))
+	found := false
+	for _, ty := range types {
+		if ty == "deadman_silent" {
+			found = true
+		}
+	}
+	if !found {
+		t.Errorf("expected a deadman_silent event, got %v", types)
+	}
+}
+
+// The generic staleness sweep must keep its hands off heartbeats: they answer to
+// their own, much tighter, timeout, and an 'expiry' resolution here would be
+// both wrong and unrecoverable.
+func TestDeadman_GenericExpiryLeavesHeartbeatAlone(t *testing.T) {
+	s, _ := deadmanTS(t, deadmanCfg())
+
+	heartbeat(t, s, "fp-watchdog", nil)
+	silence(t, s, "fp-watchdog", 5*time.Minute)
+
+	// staleAfter far tighter than the dead man's switch timeout.
+	sweep(t, s, time.Minute)
+
+	status, source, _ := s.alertRow(t, "fp-watchdog")
+	if status != "firing" || source != nil {
+		t.Errorf("expected a live heartbeat left alone, got %q / %v", status, source)
+	}
+	if got := s.countIncidents(t); got != 0 {
+		t.Errorf("expected no incident for a heartbeat that is still fresh, got %d", got)
+	}
+}
+
+// An explicit resolved from Alertmanager is the sender telling us the heartbeat
+// stopped. There is nothing left to wait out.
+func TestDeadman_AlertmanagerResolvedIsImmediateDeath(t *testing.T) {
+	s, _ := deadmanTS(t, deadmanCfg())
+
+	heartbeat(t, s, "fp-watchdog", nil)
+	postWebhook(t, s, []map[string]any{
+		amAlert("fp-watchdog", "Watchdog", "resolved", "2026-05-20T10:00:00Z", zeroTime, nil),
+	}, watchdogGroupKey)
+
+	// No ageing: received_at is seconds old, well inside the timeout.
+	sweep(t, s, noArchive)
+
+	if got := s.countIncidents(t); got != 1 {
+		t.Fatalf("expected a resolved heartbeat to open an incident at once, got %d", got)
+	}
+	// Alertmanager told the truth first, so its resolution source stands.
+	if _, source, _ := s.alertRow(t, "fp-watchdog"); source == nil || *source != "alertmanager" {
+		t.Errorf("expected the Alertmanager resolution source kept, got %v", source)
+	}
+}
+
+// Each label set is its own switch, so one healthy source cannot mask a dead one.
+func TestDeadman_TracksEachFingerprintSeparately(t *testing.T) {
+	s, _ := deadmanTS(t, deadmanCfg())
+
+	heartbeat(t, s, "fp-a", map[string]string{"cluster": "a"})
+	heartbeat(t, s, "fp-b", map[string]string{"cluster": "b"})
+	silence(t, s, "fp-b", 2*time.Hour)
+	sweep(t, s, noArchive)
+
+	if got := s.countIncidents(t); got != 1 {
+		t.Fatalf("expected only the silent switch to page, got %d incidents", got)
+	}
+	if _, status, _, _ := incidentByGroup(t, s, "deadman:fp-b"); status != "triggered" {
+		t.Errorf("expected the incident to belong to the silent switch, got %q", status)
+	}
+	if status, _, _ := s.alertRow(t, "fp-a"); status != "firing" {
+		t.Errorf("expected the live switch untouched, got %q", status)
+	}
+}
+
+// A switch nothing has ever been heard from is dormant. A fresh deploy, a
+// restored database or a typo'd alertname must not page.
+func TestDeadman_UnheardOfSwitchIsDormant(t *testing.T) {
+	s, _ := deadmanTS(t, api.ParseDeadmanConfig("alertname=NeverSent", time.Hour, "critical"))
+
+	sweep(t, s, noArchive)
+
+	if got := s.countIncidents(t); got != 0 {
+		t.Fatalf("expected a switch that never chirped to be dormant, got %d incidents", got)
+	}
+}
+
+// ---------------------------------------------------------------------------
+// Recovery and re-arming
+// ---------------------------------------------------------------------------
+
+// The returning heartbeat carries the unchanged startsAt of an alert that never
+// stopped firing, so this also covers the ingest guard exemption: without it the
+// upsert would discard the payload and the switch could die exactly once.
+func TestDeadman_RecoveryResolvesIncident(t *testing.T) {
+	s, _ := deadmanTS(t, deadmanCfg())
+
+	heartbeat(t, s, "fp-watchdog", nil)
+	silence(t, s, "fp-watchdog", 2*time.Hour)
+	sweep(t, s, noArchive)
+
+	heartbeat(t, s, "fp-watchdog", nil)
+	if status, source, _ := s.alertRow(t, "fp-watchdog"); status != "firing" || source != nil {
+		t.Fatalf("expected the returning heartbeat to be accepted, got %q / %v", status, source)
+	}
+
+	sweep(t, s, noArchive)
+
+	_, status, _, source := incidentByGroup(t, s, "deadman:fp-watchdog")
+	if status != "resolved" {
+		t.Errorf("expected recovery to close the incident, got %q", status)
+	}
+	if source == nil || *source != "recovered" {
+		t.Errorf("expected resolution_source recovered, got %v", source)
+	}
+	if got := s.countNotifications(t, "resolved"); got != 1 {
+		t.Errorf("expected 1 all-clear, got %d", got)
+	}
+}
+
+// Resolving a dead man's switch incident sticks, exactly as it does for an
+// alert-backed one. A source that is gone for good is a one-time page.
+func TestDeadman_ManualResolveSticksWhileSilent(t *testing.T) {
+	s, _ := deadmanTS(t, deadmanCfg())
+
+	heartbeat(t, s, "fp-watchdog", nil)
+	silence(t, s, "fp-watchdog", 2*time.Hour)
+	sweep(t, s, noArchive)
+
+	s.req(t, http.MethodPost, "/api/incidents/1/resolve", nil).Body.Close()
+
+	// Still silent, several sweeps later.
+	sweep(t, s, noArchive)
+	sweep(t, s, noArchive)
+
+	if got := s.countIncidents(t); got != 1 {
+		t.Fatalf("expected a manually resolved incident to stay closed, got %d", got)
+	}
+}
+
+// ...but the switch re-arms, so a heartbeat that comes back and dies again is a
+// new incident rather than silence forever.
+func TestDeadman_ReArmsAfterHeartbeatReturns(t *testing.T) {
+	s, _ := deadmanTS(t, deadmanCfg())
+
+	heartbeat(t, s, "fp-watchdog", nil)
+	silence(t, s, "fp-watchdog", 2*time.Hour)
+	sweep(t, s, noArchive)
+	s.req(t, http.MethodPost, "/api/incidents/1/resolve", nil).Body.Close()
+
+	// That episode is yesterday's news; the heartbeat now returns after it.
+	ageIncidents(t, s, 10*time.Hour)
+
+	heartbeat(t, s, "fp-watchdog", nil)
+	sweep(t, s, noArchive)
+	if got := s.countIncidents(t); got != 1 {
+		t.Fatalf("expected the live switch to open nothing, got %d incidents", got)
+	}
+
+	silence(t, s, "fp-watchdog", 2*time.Hour)
+	sweep(t, s, noArchive)
+
+	if got := s.countIncidents(t); got != 2 {
+		t.Fatalf("expected a second death to open a second incident, got %d", got)
+	}
+}
+
+// A dead man's switch incident has no member alerts — linking the heartbeat
+// would have the settled-incident cascade close it on the very sweep that opened
+// it — so the cascade has to leave it alone.
+func TestDeadman_SettledCascadeLeavesIncidentOpen(t *testing.T) {
+	s, _ := deadmanTS(t, deadmanCfg())
+
+	heartbeat(t, s, "fp-watchdog", nil)
+	silence(t, s, "fp-watchdog", 2*time.Hour)
+	sweep(t, s, noArchive)
+	sweep(t, s, noArchive)
+
+	if _, status, _, _ := incidentByGroup(t, s, "deadman:fp-watchdog"); status != "triggered" {
+		t.Fatalf("expected the incident to stay open until the switch recovers, got %q", status)
+	}
+}
+
+// ---------------------------------------------------------------------------
+// Configuration
+// ---------------------------------------------------------------------------
+
+func TestParseDeadmanConfig(t *testing.T) {
+	tests := []struct {
+		name     string
+		matchers string
+		timeout  time.Duration
+		want     []api.DeadmanMatcher
+		enabled  bool
+	}{
+		{
+			name:     "single alertname",
+			matchers: "alertname=Watchdog",
+			timeout:  time.Hour,
+			want:     []api.DeadmanMatcher{{Name: "Watchdog", Labels: map[string]string{}}},
+			enabled:  true,
+		},
+		{
+			name:     "several matchers with extra labels and whitespace",
+			matchers: " alertname=Watchdog, cluster=prod ; alertname=EdgeHeartbeat ",
+			timeout:  time.Hour,
+			want: []api.DeadmanMatcher{
+				{Name: "Watchdog", Labels: map[string]string{"cluster": "prod"}},
+				{Name: "EdgeHeartbeat", Labels: map[string]string{}},
+			},
+			enabled: true,
+		},
+		{
+			// Mandatory: it is what keeps the sweeper's candidate query on an index.
+			name:     "matcher without alertname is dropped",
+			matchers: "cluster=prod; alertname=Watchdog",
+			timeout:  time.Hour,
+			want:     []api.DeadmanMatcher{{Name: "Watchdog", Labels: map[string]string{}}},
+			enabled:  true,
+		},
+		{
+			name:     "malformed condition drops only its matcher",
+			matchers: "alertname=Watchdog,garbage; alertname=Other",
+			timeout:  time.Hour,
+			want:     []api.DeadmanMatcher{{Name: "Other", Labels: map[string]string{}}},
+			enabled:  true,
+		},
+		{
+			name:     "zero timeout disables",
+			matchers: "alertname=Watchdog",
+			timeout:  0,
+			want:     []api.DeadmanMatcher{{Name: "Watchdog", Labels: map[string]string{}}},
+			enabled:  false,
+		},
+		{
+			name:     "no usable matchers disables",
+			matchers: "",
+			timeout:  time.Hour,
+			want:     nil,
+			enabled:  false,
+		},
+	}
+
+	for _, tc := range tests {
+		t.Run(tc.name, func(t *testing.T) {
+			got := api.ParseDeadmanConfig(tc.matchers, tc.timeout, "critical")
+			if len(got.Matchers) != len(tc.want) {
+				t.Fatalf("got %d matchers %v, want %d", len(got.Matchers), got.Matchers, len(tc.want))
+			}
+			for i, w := range tc.want {
+				if got.Matchers[i].Name != w.Name {
+					t.Errorf("matcher %d: name %q, want %q", i, got.Matchers[i].Name, w.Name)
+				}
+				if len(got.Matchers[i].Labels) != len(w.Labels) {
+					t.Errorf("matcher %d: labels %v, want %v", i, got.Matchers[i].Labels, w.Labels)
+					continue
+				}
+				for k, v := range w.Labels {
+					if got.Matchers[i].Labels[k] != v {
+						t.Errorf("matcher %d: label %s=%q, want %q", i, k, got.Matchers[i].Labels[k], v)
+					}
+				}
+			}
+		})
+	}
+}
+
+// A zero config is off, which is what keeps the feature opt-in for anything
+// building a router without one.
+func TestDeadman_DisabledConfigIsInert(t *testing.T) {
+	s, _ := deadmanTS(t, api.DeadmanConfig{})
+
+	heartbeat(t, s, "fp-watchdog", nil)
+	silence(t, s, "fp-watchdog", 48*time.Hour)
+	sweep(t, s, time.Hour)
+
+	// Ordinary alert handling: an incident from the arrival, not the absence.
+	if got := s.countIncidents(t); got != 1 {
+		t.Fatalf("expected plain alert handling with deadman off, got %d incidents", got)
+	}
+	if _, source, _ := s.alertRow(t, "fp-watchdog"); source == nil || *source != "expiry" {
+		t.Errorf("expected the generic sweeper to own the alert, got %v", source)
+	}
+}

internal/api/incident_store.go

@@ -16,6 +16,11 @@ import (
 const (
 	incidentResolutionAlerts = "alerts"
 	incidentResolutionManual = "manual"
+
+	// incidentResolutionRecovered closes a dead man's switch incident whose
+	// heartbeat started arriving again. It cannot be "alerts": these incidents
+	// have no member alerts for the cascade to work from.
+	incidentResolutionRecovered = "recovered"
 )
 
 // Incident timeline event types. Stored as free text so adding one later is not
@@ -31,6 +36,7 @@ const (
 	evUnsnoozed      = "unsnoozed"
 	evResolved       = "resolved"
 	evNote           = "note"
+	evDeadmanSilent  = "deadman_silent"
 )
 
 // severityLabel is the Alertmanager label an incident's severity is derived from.

internal/api/incidents_test.go

@@ -613,7 +613,7 @@ func TestSweeper_ArchivesResolvedIncidents(t *testing.T) {
 
 	s.exec(t, "UPDATE incidents SET resolved_at = ? WHERE id = 1",
 		time.Now().Add(-30*24*time.Hour).Unix())
-	api.Sweep(context.Background(), s.db, 7*24*time.Hour, 6*time.Hour)
+	api.Sweep(context.Background(), s.db, 7*24*time.Hour, 6*time.Hour, s.deadman, s.notify)
 
 	if inc := getIncident(t, s, 1); inc["archived_at"] == nil {
 		t.Error("expected the sweeper to archive a long-resolved incident")

internal/api/notify_test.go

@@ -438,7 +438,7 @@ func TestNotify_SweepPurgesExpiredAckTokens(t *testing.T) {
 	s.sweepNotify(t)
 	s.exec(t, "UPDATE incident_ack_tokens SET expires_at = ?", time.Now().Add(-time.Minute).Unix())
 
-	api.Sweep(context.Background(), s.db, 168*time.Hour, 6*time.Hour)
+	api.Sweep(context.Background(), s.db, 168*time.Hour, 6*time.Hour, s.deadman, s.notify)
 
 	var n int
 	if err := s.db.QueryRow("SELECT COUNT(*) FROM incident_ack_tokens").Scan(&n); err != nil {

internal/api/router.go

@@ -8,10 +8,11 @@ import (
 	"github.com/go-chi/chi/v5/middleware"
 )
 
-// NewRouter builds the HTTP surface. notify is passed through to the webhook,
-// the only handler that has to decide where a new incident's page goes; a zero
-// value disables notifications.
-func NewRouter(db *sql.DB, notify NotifyConfig) http.Handler {
+// NewRouter builds the HTTP surface. notify and deadman are passed through to
+// the webhook, the only handler that has to decide where a new incident's page
+// goes and which arriving alerts are heartbeats rather than problems. A zero
+// notify disables notifications; a zero deadman disables dead man's switches.
+func NewRouter(db *sql.DB, notify NotifyConfig, deadman DeadmanConfig) http.Handler {
 	r := chi.NewRouter()
 	r.Use(middleware.Logger)
 	r.Use(middleware.Recoverer)
@@ -25,7 +26,7 @@ func NewRouter(db *sql.DB, notify NotifyConfig) http.Handler {
 	// the scoped token in its path rather than an API key, and has to stay
 	// reachable from outside the cluster for the button to work.
 	r.Post("/api/bootstrap", handleBootstrap(db))
-	r.Post("/api/alertmanager/webhook", handleAlertmanagerWebhook(db, notify))
+	r.Post("/api/alertmanager/webhook", handleAlertmanagerWebhook(db, notify, deadman))
 	r.Post("/api/notify/ack/{token}", handleNotifyAck(db))
 
 	// All other /api routes require a valid API key.

internal/config/config.go

@@ -15,6 +15,25 @@ type Config struct {
 	// repeat_interval (default 4h), which is what refreshes the alert.
 	StaleAfter time.Duration
 
+	// DeadmanMatchers selects the alerts that are heartbeats rather than
+	// problems: receiving one opens no incident, and the absence of one does.
+	//
+	// ";" separates matchers, "," the label conditions within one, "=" is exact
+	// equality — `alertname=Watchdog,cluster=prod; alertname=Heartbeat`. Every
+	// matcher must name an alertname. See api.ParseDeadmanConfig.
+	DeadmanMatchers string
+
+	// DeadmanTimeout is how long a heartbeat may go unheard before its switch is
+	// declared dead. It must be *shorter* than the Alertmanager repeat_interval
+	// of the route carrying the heartbeat — the opposite of StaleAfter, and the
+	// reason a dead man's switch usually wants a route of its own. Zero disables
+	// dead man's switch handling entirely.
+	DeadmanTimeout time.Duration
+
+	// DeadmanSeverity is the severity a dead man's switch incident opens at.
+	// These incidents have no member alerts to derive one from.
+	DeadmanSeverity string
+
 	// NtfyURL is the ntfy server push notifications are published to. Empty
 	// disables notifications entirely.
 	NtfyURL string
@@ -44,12 +63,24 @@ func Load() Config {
 	if dbPath == "" {
 		dbPath = "terdut.db"
 	}
+	deadmanMatchers := os.Getenv("TERDUT_DEADMAN_MATCHERS")
+	if deadmanMatchers == "" {
+		deadmanMatchers = "alertname=Watchdog"
+	}
+	deadmanSeverity := os.Getenv("TERDUT_DEADMAN_SEVERITY")
+	if deadmanSeverity == "" {
+		deadmanSeverity = "critical"
+	}
 	return Config{
 		Addr:         addr,
 		DBPath:       dbPath,
 		ArchiveAfter: duration("TERDUT_ARCHIVE_AFTER", 7*24*time.Hour),
 		StaleAfter:   duration("TERDUT_STALE_AFTER", 6*time.Hour),
 
+		DeadmanMatchers: deadmanMatchers,
+		DeadmanTimeout:  duration("TERDUT_DEADMAN_TIMEOUT", 15*time.Minute),
+		DeadmanSeverity: deadmanSeverity,
+
 		NtfyURL:           os.Getenv("TERDUT_NTFY_URL"),
 		NtfyToken:         os.Getenv("TERDUT_NTFY_TOKEN"),
 		NtfyFallbackTopic: os.Getenv("TERDUT_NTFY_FALLBACK_TOPIC"),