Turn incoming alerts into incidents

The alerts row was both Alertmanager's record and the human work queue, and
the two have different owners. The webhook upsert rewrites that row on every
notification; acknowledgement, comments and archiving were columns on it that
the upsert happened not to touch. So an alert that resolved and re-fired days
later still read as acknowledged by whoever acked the first occurrence — the
ack outlived the thing it referred to. Nothing recorded transitions either:
rows are mutated in place, so there was no timeline and no way to compute how
long anything took.

Alerts are now read-only signal records with two states, and incidents are
the work item: triggered, acknowledged or resolved, with an assignee, a
snooze, notes and an append-only timeline. Many alerts map to one incident,
and a new occurrence opens a new incident, which is what makes a stale ack
impossible rather than merely unlikely.

Correlation uses Alertmanager's own groupKey. It already grouped the alerts
according to the group_by routing tree the operator configured and sends the
result on every webhook, where it was being discarded; adopting it means
changing group_by in alertmanager.yml changes correlation here, with no
second grouping scheme to configure and keep in sync.

An incident opens only when an alert transitions into firing — an unseen
fingerprint, a newer startsAt, or a resolved alert starting again. The
unchanged notifications Alertmanager re-sends every repeat_interval are none
of those. That rule is what lets manual resolution be terminal: without it,
closing an incident by hand would be undone by the next re-send of an alert
that never stopped firing, and the button would be a lie. Snooze covers the
"not now" case instead. Incidents otherwise resolve by cascade, once every
alert under them has stopped firing, whether by webhook or by expiry.

New incidents are assigned to whoever holds today's schedule entry. The
schedule table has existed since the first release with nothing reading it.

Also here, following from the split:

  - Incident severity is a high-water mark over its alerts, never lowered.
    An incident that hit critical was a critical incident, and downgrading a
    live one would demote it in the queue while the work is still open.
  - /api/stats/incidents reports MTTA and MTTR, null rather than zero until
    there is something to average. Neither was computable before.
  - Alert archiving becomes sweeper-only housekeeping; the archive people
    interact with is the incident's.

Breaking: the alert acknowledge, archive and comment endpoints are gone, and
the alert object drops the acknowledgement fields and gains incident_id. The
README maps each removed endpoint to its replacement. Migration 008 backfills
an incident per existing alert, archived ones included so no comment is
orphaned, carrying acknowledgements across and turning comments into timeline
notes.

Both documented alert contracts are untouched: received_at still advances on
every accepted payload, re-sends included, and resolution_source still says
how much to trust ends_at. The upsert is byte-for-byte what it was, now
running inside the ingest transaction.

internal/api/incident_store.go

@@ -0,0 +1,269 @@
+package api
+
+import (
+	"context"
+	"database/sql"
+	"encoding/json"
+	"sort"
+	"strings"
+	"time"
+
+	"github.com/yeniklas/terdut-server/internal/models"
+)
+
+// Values for incidents.resolution_source, recording who closed the incident:
+// every member alert stopped firing, or a person decided it was done.
+const (
+	incidentResolutionAlerts = "alerts"
+	incidentResolutionManual = "manual"
+)
+
+// Incident timeline event types. Stored as free text so adding one later is not
+// a migration, but these are the ones the server writes.
+const (
+	evTriggered      = "triggered"
+	evAlertAdded     = "alert_added"
+	evAlertResolved  = "alert_resolved"
+	evAcknowledged   = "acknowledged"
+	evUnacknowledged = "unacknowledged"
+	evAssigned       = "assigned"
+	evSnoozed        = "snoozed"
+	evUnsnoozed      = "unsnoozed"
+	evResolved       = "resolved"
+	evNote           = "note"
+)
+
+// severityLabel is the Alertmanager label an incident's severity is derived from.
+const severityLabel = "severity"
+
+// querier is satisfied by both *sql.DB and *sql.Tx, so the helpers below work
+// inside the webhook's transaction and standalone from handlers and the sweeper.
+type querier interface {
+	ExecContext(ctx context.Context, query string, args ...any) (sql.Result, error)
+	QueryContext(ctx context.Context, query string, args ...any) (*sql.Rows, error)
+	QueryRowContext(ctx context.Context, query string, args ...any) *sql.Row
+}
+
+const incidentSelectFrom = `
+	SELECT i.id, i.group_key, i.title, i.group_labels, i.status, i.severity,
+	       i.triggered_at,
+	       i.acknowledged_by, i.acknowledged_at, ack.username,
+	       i.assigned_to, asg.username, i.snoozed_until,
+	       i.resolved_at, i.resolution_source, i.archived_at
+	FROM   incidents i
+	LEFT   JOIN users ack ON ack.id = i.acknowledged_by
+	LEFT   JOIN users asg ON asg.id = i.assigned_to`
+
+func scanIncident(s scanner) (models.Incident, error) {
+	var i models.Incident
+	var groupLabelsJSON string
+	var triggeredAt int64
+	var ackAt, snoozedUntil, resolvedAt, archivedAt *int64
+
+	if err := s.Scan(
+		&i.ID, &i.GroupKey, &i.Title, &groupLabelsJSON, &i.Status, &i.Severity,
+		&triggeredAt,
+		&i.AcknowledgedByID, &ackAt, &i.AcknowledgedByUser,
+		&i.AssignedToID, &i.AssignedToUser, &snoozedUntil,
+		&resolvedAt, &i.ResolutionSource, &archivedAt,
+	); err != nil {
+		return i, err
+	}
+
+	json.Unmarshal([]byte(groupLabelsJSON), &i.GroupLabels) //nolint:errcheck
+	i.TriggeredAt = time.Unix(triggeredAt, 0).UTC()
+	i.AcknowledgedAt = unixPtr(ackAt)
+	i.SnoozedUntil = unixPtr(snoozedUntil)
+	i.ResolvedAt = unixPtr(resolvedAt)
+	i.ArchivedAt = unixPtr(archivedAt)
+	return i, nil
+}
+
+// unixPtr converts a nullable Unix-second column to a nullable UTC time.
+func unixPtr(sec *int64) *time.Time {
+	if sec == nil {
+		return nil
+	}
+	t := time.Unix(*sec, 0).UTC()
+	return &t
+}
+
+func fetchIncident(ctx context.Context, q querier, id int64) (models.Incident, error) {
+	return scanIncident(q.QueryRowContext(ctx, incidentSelectFrom+" WHERE i.id = ?", id))
+}
+
+// logEvent appends one entry to an incident's timeline. A nil userID means the
+// server acted rather than a person.
+func logEvent(ctx context.Context, q querier, incidentID int64, evType string, userID, alertID *int64, detail *string) error {
+	_, err := q.ExecContext(ctx, `
+		INSERT INTO incident_events (incident_id, type, user_id, alert_id, detail, created_at)
+		VALUES (?, ?, ?, ?, ?, ?)`,
+		incidentID, evType, userID, alertID, detail, time.Now().Unix())
+	return err
+}
+
+// todayUTC is the schedule's day key. The schedule's smallest unit is one UTC day.
+func todayUTC() string {
+	return time.Now().UTC().Format("2006-01-02")
+}
+
+// currentOnCall returns today's on-call user, or nil when nobody is scheduled.
+// A missing schedule entry is not an error — incidents just open unassigned.
+func currentOnCall(ctx context.Context, q querier) (*int64, error) {
+	var userID int64
+	err := q.QueryRowContext(ctx,
+		"SELECT user_id FROM schedule_entries WHERE date = ?", todayUTC()).Scan(&userID)
+	if err == sql.ErrNoRows {
+		return nil, nil
+	}
+	if err != nil {
+		return nil, err
+	}
+	return &userID, nil
+}
+
+// severityRank orders the conventional Alertmanager severity label values.
+// Anything unrecognised sorts below all of them rather than being dropped.
+func severityRank(s string) int {
+	switch strings.ToLower(s) {
+	case "critical":
+		return 4
+	case "error":
+		return 3
+	case "warning":
+		return 2
+	case "info":
+		return 1
+	default:
+		return 0
+	}
+}
+
+// refreshSeverity raises an incident's severity to the highest `severity` label
+// seen across its alerts.
+//
+// It is a high-water mark, never lowered: an incident that hit critical was a
+// critical incident, even after the critical alert clears and a warning is all
+// that is left firing. Downgrading a live incident would also quietly demote it
+// in the queue while the work is still open.
+func refreshSeverity(ctx context.Context, q querier, incidentID int64) error {
+	rows, err := q.QueryContext(ctx, `
+		SELECT json_extract(a.labels, '$.'||?)
+		FROM   incident_alerts ia
+		JOIN   alerts a ON a.id = ia.alert_id
+		WHERE  ia.incident_id = ?`, severityLabel, incidentID)
+	if err != nil {
+		return err
+	}
+
+	best := ""
+	for rows.Next() {
+		var sev *string
+		if err := rows.Scan(&sev); err != nil {
+			rows.Close()
+			return err
+		}
+		if sev != nil && severityRank(*sev) > severityRank(best) {
+			best = *sev
+		}
+	}
+	if err := rows.Err(); err != nil {
+		rows.Close()
+		return err
+	}
+	rows.Close()
+
+	if best == "" {
+		return nil
+	}
+	// The comparison lives in SQL so an unrelated concurrent update cannot be
+	// clobbered by a stale read.
+	_, err = q.ExecContext(ctx, `
+		UPDATE incidents SET severity = ?
+		 WHERE id = ?
+		   AND (severity IS NULL OR `+severityRankSQL("severity")+` < ?)`,
+		best, incidentID, severityRank(best))
+	return err
+}
+
+// severityRankSQL mirrors severityRank for use inside a statement. SQL cannot
+// order these strings meaningfully on its own.
+func severityRankSQL(col string) string {
+	return `CASE lower(COALESCE(` + col + `, ''))
+		WHEN 'critical' THEN 4
+		WHEN 'error'    THEN 3
+		WHEN 'warning'  THEN 2
+		WHEN 'info'     THEN 1
+		ELSE 0 END`
+}
+
+// resolveIfSettled closes an incident once every alert under it has stopped
+// firing — PagerDuty's cascade, and the only automatic route out of the open
+// state. Reports whether it actually resolved anything.
+func resolveIfSettled(ctx context.Context, q querier, incidentID int64) (bool, error) {
+	res, err := q.ExecContext(ctx, `
+		UPDATE incidents
+		   SET status            = 'resolved',
+		       resolved_at       = ?,
+		       resolution_source = ?
+		 WHERE id = ?
+		   AND resolved_at IS NULL
+		   -- An incident with no members yet is mid-creation, not settled.
+		   AND EXISTS (SELECT 1 FROM incident_alerts ia WHERE ia.incident_id = incidents.id)
+		   AND NOT EXISTS (SELECT 1
+		                     FROM incident_alerts ia
+		                     JOIN alerts a ON a.id = ia.alert_id
+		                    WHERE ia.incident_id = incidents.id
+		                      AND a.status = 'firing')`,
+		time.Now().Unix(), incidentResolutionAlerts, incidentID)
+	if err != nil {
+		return false, err
+	}
+	n, _ := res.RowsAffected()
+	if n == 0 {
+		return false, nil
+	}
+	return true, logEvent(ctx, q, incidentID, evResolved, nil, nil, nil)
+}
+
+// openIncidentForAlert returns the open incident an alert currently belongs to,
+// or 0 when it has none. Used when an alert resolves or expires so the event
+// lands on the right timeline.
+func openIncidentForAlert(ctx context.Context, q querier, alertID int64) (int64, error) {
+	var id int64
+	err := q.QueryRowContext(ctx, `
+		SELECT i.id
+		FROM   incident_alerts ia
+		JOIN   incidents i ON i.id = ia.incident_id
+		WHERE  ia.alert_id = ? AND i.resolved_at IS NULL`, alertID).Scan(&id)
+	if err == sql.ErrNoRows {
+		return 0, nil
+	}
+	return id, err
+}
+
+// incidentTitle renders a human-readable title from Alertmanager's groupLabels,
+// leading with the alert name and appending whatever else the operator grouped
+// by. Falls back to the alert's own name when the payload carried no groupLabels.
+func incidentTitle(groupLabels map[string]string, fallback string) string {
+	name := groupLabels["alertname"]
+	if name == "" {
+		name = fallback
+	}
+	if name == "" {
+		name = "Incident"
+	}
+
+	rest := make([]string, 0, len(groupLabels))
+	for k, v := range groupLabels {
+		if k == "alertname" {
+			continue
+		}
+		rest = append(rest, k+"="+v)
+	}
+	if len(rest) == 0 {
+		return name
+	}
+	sort.Strings(rest)
+	return name + " (" + strings.Join(rest, ", ") + ")"
+}