Notice when the Watchdog alert stops arriving

Everything this server does assumes alerts arrive. If Prometheus stops
evaluating, or Alertmanager cannot reach us, nothing arrives — and
silence is indistinguishable from everything being fine. The cluster has
shipped the alert for exactly this case all along: Watchdog is
expr: vector(1), so it fires permanently and is re-sent forever, and it
is worth nothing unless something downstream notices it stop. Nothing
did. It arrived, opened no incident because a repeat_interval re-send is
not a new occurrence, and when the monitoring stack died the sweeper
quietly expired it and paged nobody.

So the handling is inverted for a configurable set of alerts: receiving
one opens no incident, and the absence of one does. TERDUT_DEADMAN_MATCHERS
selects them as label matchers, defaulting to alertname=Watchdog.

The unit of monitoring is the fingerprint rather than the alert name. Two
clusters sending the same Watchdog are two independent switches, so a
healthy one can never mask a dead one. Every matcher must name an
alertname, which keeps the sweeper's candidate query on alerts_name_idx
instead of JSON-extracting labels from every row, and leaves matching
with a single implementation.

A switch is dormant until its first heartbeat: a matcher nothing has ever
sent opens nothing, so a fresh deploy or a restored database does not
page. Resolving the incident by hand sticks, exactly as it does for an
alert-backed one, so a decommissioned source is a one-time page rather
than a nag; the switch re-arms only when the heartbeat comes back, and
dying again is a new incident.

The incident has no member alerts on purpose. Linking the heartbeat would
have the settled-incident cascade close it on the very sweep that opened
it, and there is no alert describing the problem anyway — the problem is
that no alert arrived. What happened is on the timeline instead, and
recovery is the only automatic way out.

One narrow exemption in the ingest guard makes recovery possible at all.
A heartbeat we declared dead is marked resolved, and the one that proves
us wrong carries the unchanged startsAt of an alert that never stopped
firing — so "resolution is terminal within an instance" would discard it
forever and a switch could die exactly once. The exemption is scoped to
resolution_source = 'deadman', which is the only resolution this server
infers from silence on a timeout of its own, so nothing another writer
set can be undone by a stale retry. Matched alerts are also held back
from the generic staleness expiry, which would otherwise resolve a
heartbeat as 'expiry' long before its own tighter deadline.

The timeout points the opposite way to TERDUT_STALE_AFTER: staleness is a
generous grace period around a repeat_interval you do not control, while
this is a deadline you set deliberately and configure the heartbeat's
route to beat. Inheriting a 4h or 12h repeat_interval gives a dead man's
switch with a twelve hour fuse, so the README spells out the route the
heartbeat needs.

internal/api/alertmanager.go

@@ -14,6 +14,12 @@ import (
 const (
 	resolutionAlertmanager = "alertmanager"
 	resolutionExpiry       = "expiry"
+
+	// resolutionDeadman marks a heartbeat the dead man's switch sweeper declared
+	// dead. Distinct from expiry because it is load-bearing, not just
+	// descriptive: it is the one resolution the ingest upsert will let a
+	// same-instance re-fire undo, so a switch that comes back can be heard.
+	resolutionDeadman = "deadman"
 )
 
 // amPayload mirrors the Alertmanager webhook v4 payload.
@@ -56,9 +62,15 @@ type ingested struct {
 
 	// justResolved marks the firing → resolved edge, worth a timeline entry.
 	justResolved bool
+
+	// deadman marks a heartbeat: an alert whose arrival means everything is
+	// fine. It is stored like any other alert — received_at is the heartbeat —
+	// but it never reaches an incident. Its absence is what opens one, which
+	// sweepDeadman decides later and elsewhere.
+	deadman bool
 }
 
-func handleAlertmanagerWebhook(db *sql.DB, notify NotifyConfig) http.HandlerFunc {
+func handleAlertmanagerWebhook(db *sql.DB, notify NotifyConfig, deadman DeadmanConfig) http.HandlerFunc {
 	return func(w http.ResponseWriter, r *http.Request) {
 		var payload amPayload
 		if err := decodeJSON(r, &payload); err != nil {
@@ -69,7 +81,7 @@ func handleAlertmanagerWebhook(db *sql.DB, notify NotifyConfig) http.HandlerFunc
 		// Alertmanager retries anything that is not 2xx, and a retry of a payload
 		// we failed to store is more useful than an error it cannot act on — so
 		// failures are logged, not surfaced.
-		if err := ingest(r.Context(), db, notify, payload); err != nil {
+		if err := ingest(r.Context(), db, notify, deadman, payload); err != nil {
 			log.Printf("webhook ingest (group %q): %v", payload.GroupKey, err)
 		}
 
@@ -80,14 +92,14 @@ func handleAlertmanagerWebhook(db *sql.DB, notify NotifyConfig) http.HandlerFunc
 // ingest stores a payload's alerts and reconciles the incident for its group.
 // The whole payload is one transaction: an incident that opened but whose alerts
 // failed to link would be a work item nobody could act on.
-func ingest(ctx context.Context, db *sql.DB, notify NotifyConfig, payload amPayload) error {
+func ingest(ctx context.Context, db *sql.DB, notify NotifyConfig, deadman DeadmanConfig, payload amPayload) error {
 	tx, err := db.BeginTx(ctx, nil)
 	if err != nil {
 		return err
 	}
 	defer tx.Rollback() //nolint:errcheck
 
-	accepted, err := upsertAlerts(ctx, tx, payload.Alerts)
+	accepted, err := upsertAlerts(ctx, tx, deadman, payload.Alerts)
 	if err != nil {
 		return err
 	}
@@ -103,7 +115,7 @@ func ingest(ctx context.Context, db *sql.DB, notify NotifyConfig, payload amPayl
 	if incidentID != 0 {
 		touched[incidentID] = true
 		for _, a := range accepted {
-			if !a.firing {
+			if !a.firing || a.deadman {
 				continue
 			}
 			if err := linkAlert(ctx, tx, incidentID, a.id); err != nil {
@@ -113,7 +125,7 @@ func ingest(ctx context.Context, db *sql.DB, notify NotifyConfig, payload amPayl
 	}
 
 	for _, a := range accepted {
-		if !a.justResolved {
+		if !a.justResolved || a.deadman {
 			continue
 		}
 		id, err := openIncidentForAlert(ctx, tx, a.id)
@@ -144,7 +156,7 @@ func ingest(ctx context.Context, db *sql.DB, notify NotifyConfig, payload amPayl
 
 // upsertAlerts stores each alert of a payload and reports what changed. Payloads
 // the ordering guard rejected are left out entirely.
-func upsertAlerts(ctx context.Context, tx *sql.Tx, alerts []amAlert) ([]ingested, error) {
+func upsertAlerts(ctx context.Context, tx *sql.Tx, deadman DeadmanConfig, alerts []amAlert) ([]ingested, error) {
 	now := time.Now().Unix()
 	accepted := make([]ingested, 0, len(alerts))
 
@@ -187,7 +199,15 @@ func upsertAlerts(ctx context.Context, tx *sql.Tx, alerts []amAlert) ([]ingested
 		// older than the stored one. Alertmanager retries failed notifications,
 		// so a stale firing retry can arrive after the resolved one; it carries
 		// the same startsAt, whereas a genuine re-fire carries a newer one.
-		// Within a single instance, resolution is terminal.
+		// Within a single instance, resolution is terminal — with one exception.
+		//
+		// A resolution this server synthesised for a dead man's switch is not
+		// Alertmanager's word that the instance ended; it is our inference from
+		// silence. The heartbeat that proves us wrong carries the unchanged
+		// startsAt of an alert that never stopped firing, so without the
+		// exemption a switch could go dead exactly once and never be heard from
+		// again. Scoped to 'deadman' so no resolution anybody else wrote can be
+		// undone by a stale retry.
 		if _, err := tx.ExecContext(ctx, `
 			INSERT INTO alerts
 				(fingerprint, name, status, labels, annotations, starts_at, ends_at,
@@ -211,7 +231,8 @@ func upsertAlerts(ctx context.Context, tx *sql.Tx, alerts []amAlert) ([]ingested
 				                        THEN NULL ELSE alerts.archived_at END
 			WHERE excluded.starts_at > alerts.starts_at
 			   OR (excluded.starts_at = alerts.starts_at
-			       AND NOT (alerts.status = 'resolved' AND excluded.status = 'firing'))`,
+			       AND (alerts.resolution_source = '`+resolutionDeadman+`'
+			            OR NOT (alerts.status = 'resolved' AND excluded.status = 'firing')))`,
 			a.Fingerprint, name, a.Status,
 			string(labelsJSON), string(annotationsJSON),
 			a.StartsAt.Unix(), endsAtUnix,
@@ -244,6 +265,7 @@ func upsertAlerts(ctx context.Context, tx *sql.Tx, alerts []amAlert) ([]ingested
 			firing:        firing,
 			newOccurrence: firing && (!existed || a.StartsAt.Unix() > prevStartsAt || prevStatus == "resolved"),
 			justResolved:  !firing && existed && prevStatus == "firing",
+			deadman:       deadman.isDeadman(a.Labels),
 		})
 	}
 
@@ -258,10 +280,17 @@ func upsertAlerts(ctx context.Context, tx *sql.Tx, alerts []amAlert) ([]ingested
 // something actually started firing. Without that, a manually resolved incident
 // would reappear on the next repeat_interval re-send of an alert that never
 // stopped, and manual resolution would be meaningless.
+//
+// Heartbeats do not count as anything here. A group of nothing but dead man's
+// switch alerts opens no incident at all, and a mixed group gets an incident for
+// its real alerts only.
 func incidentForGroup(ctx context.Context, tx *sql.Tx, notify NotifyConfig, payload amPayload, accepted []ingested) (int64, error) {
 	var firstName string
 	anyFiring, anyNew := false, false
 	for _, a := range accepted {
+		if a.deadman {
+			continue
+		}
 		if a.firing {
 			if !anyFiring {
 				firstName = a.name
@@ -297,13 +326,20 @@ func incidentForGroup(ctx context.Context, tx *sql.Tx, notify NotifyConfig, payl
 	if !anyNew {
 		return 0, nil
 	}
-	return openIncident(ctx, tx, notify, groupKey, payload.GroupLabels, firstName)
+	return openIncident(ctx, tx, notify, groupKey,
+		incidentTitle(payload.GroupLabels, firstName), payload.GroupLabels, nil)
 }
 
-// openIncident creates an incident for a group and assigns it to whoever is on
-// call today, which is the point at which the schedule stops being decorative.
-func openIncident(ctx context.Context, tx *sql.Tx, notify NotifyConfig, groupKey string, groupLabels map[string]string, fallbackName string) (int64, error) {
-	onCall, err := currentOnCall(ctx, tx)
+// openIncident creates an incident and assigns it to whoever is on call today,
+// which is the point at which the schedule stops being decorative.
+//
+// The one place an incident is born, for both of the things that can raise one:
+// the webhook, inside its transaction, and the dead man's switch sweeper, inside
+// its own. Hence the querier rather than a *sql.Tx. A nil severity leaves the
+// column for refreshSeverity to fill from the member alerts; the sweeper passes
+// one because its incidents have no members to derive it from.
+func openIncident(ctx context.Context, q querier, notify NotifyConfig, groupKey, title string, groupLabels map[string]string, severity *string) (int64, error) {
+	onCall, err := currentOnCall(ctx, q)
 	if err != nil {
 		return 0, err
 	}
@@ -313,10 +349,10 @@ func openIncident(ctx context.Context, tx *sql.Tx, notify NotifyConfig, groupKey
 		labelsJSON = []byte("{}")
 	}
 
-	res, err := tx.ExecContext(ctx, `
-		INSERT INTO incidents (group_key, title, group_labels, status, triggered_at, assigned_to)
-		VALUES (?, ?, ?, 'triggered', ?, ?)`,
-		groupKey, incidentTitle(groupLabels, fallbackName), string(labelsJSON),
+	res, err := q.ExecContext(ctx, `
+		INSERT INTO incidents (group_key, title, group_labels, status, severity, triggered_at, assigned_to)
+		VALUES (?, ?, ?, 'triggered', ?, ?, ?)`,
+		groupKey, title, string(labelsJSON), severity,
 		time.Now().Unix(), onCall)
 	if err != nil {
 		return 0, err
@@ -326,20 +362,20 @@ func openIncident(ctx context.Context, tx *sql.Tx, notify NotifyConfig, groupKey
 		return 0, err
 	}
 
-	if err := logEvent(ctx, tx, id, evTriggered, nil, nil, nil); err != nil {
+	if err := logEvent(ctx, q, id, evTriggered, nil, nil, nil); err != nil {
 		return 0, err
 	}
 	if onCall != nil {
 		// On an "assigned" event user_id is the assignee, not the actor.
-		if err := logEvent(ctx, tx, id, evAssigned, onCall, nil, nil); err != nil {
+		if err := logEvent(ctx, q, id, evAssigned, onCall, nil, nil); err != nil {
 			return 0, err
 		}
 	}
 
-	// Queue the page, but do not send it here: this runs inside the webhook's
-	// transaction on a single-connection pool, so an HTTP call would hold up
-	// every other request. The notifier picks the row up within a tick.
-	if err := enqueueOpened(ctx, tx, notify, id, onCall); err != nil {
+	// Queue the page, but do not send it here: this runs inside a transaction on
+	// a single-connection pool, so an HTTP call would hold up every other
+	// request. The notifier picks the row up within a tick.
+	if err := enqueueOpened(ctx, q, notify, id, onCall); err != nil {
 		return 0, err
 	}
 	return id, nil