Turn incoming alerts into incidents

The alerts row was both Alertmanager's record and the human work queue, and
the two have different owners. The webhook upsert rewrites that row on every
notification; acknowledgement, comments and archiving were columns on it that
the upsert happened not to touch. So an alert that resolved and re-fired days
later still read as acknowledged by whoever acked the first occurrence — the
ack outlived the thing it referred to. Nothing recorded transitions either:
rows are mutated in place, so there was no timeline and no way to compute how
long anything took.

Alerts are now read-only signal records with two states, and incidents are
the work item: triggered, acknowledged or resolved, with an assignee, a
snooze, notes and an append-only timeline. Many alerts map to one incident,
and a new occurrence opens a new incident, which is what makes a stale ack
impossible rather than merely unlikely.

Correlation uses Alertmanager's own groupKey. It already grouped the alerts
according to the group_by routing tree the operator configured and sends the
result on every webhook, where it was being discarded; adopting it means
changing group_by in alertmanager.yml changes correlation here, with no
second grouping scheme to configure and keep in sync.

An incident opens only when an alert transitions into firing — an unseen
fingerprint, a newer startsAt, or a resolved alert starting again. The
unchanged notifications Alertmanager re-sends every repeat_interval are none
of those. That rule is what lets manual resolution be terminal: without it,
closing an incident by hand would be undone by the next re-send of an alert
that never stopped firing, and the button would be a lie. Snooze covers the
"not now" case instead. Incidents otherwise resolve by cascade, once every
alert under them has stopped firing, whether by webhook or by expiry.

New incidents are assigned to whoever holds today's schedule entry. The
schedule table has existed since the first release with nothing reading it.

Also here, following from the split:

  - Incident severity is a high-water mark over its alerts, never lowered.
    An incident that hit critical was a critical incident, and downgrading a
    live one would demote it in the queue while the work is still open.
  - /api/stats/incidents reports MTTA and MTTR, null rather than zero until
    there is something to average. Neither was computable before.
  - Alert archiving becomes sweeper-only housekeeping; the archive people
    interact with is the incident's.

Breaking: the alert acknowledge, archive and comment endpoints are gone, and
the alert object drops the acknowledgement fields and gains incident_id. The
README maps each removed endpoint to its replacement. Migration 008 backfills
an incident per existing alert, archived ones included so no comment is
orphaned, carrying acknowledgements across and turning comments into timeline
notes.

Both documented alert contracts are untouched: received_at still advances on
every accepted payload, re-sends included, and resolution_source still says
how much to trust ends_at. The upsert is byte-for-byte what it was, now
running inside the ingest transaction.

README.md

@@ -1,11 +1,12 @@
 # Terminal Duty (terdut-server)
 
-On-call alert management server for teams using Prometheus Alertmanager.
+Incident management server for teams using Prometheus Alertmanager.
 
 - Receives Alertmanager webhooks directly — no adapter needed
-- Stores and queries alerts (acknowledge, comment)
-- On-call schedule management (user-to-day assignments)
-- Alert statistics (by status, by hour, by day)
+- Turns alerts into **incidents**, correlated by Alertmanager's own `groupKey`
+- Incident workflow: acknowledge, assign, snooze, note, resolve, with a full timeline
+- On-call schedule management, with new incidents auto-assigned to whoever is on call
+- Alert and incident statistics, including MTTA and MTTR
 - REST API with per-user API key authentication
 - Single binary, SQLite storage — trivial to self-host
 
@@ -57,7 +58,7 @@ docker run -p 8080:8080 -v $(pwd)/data:/data \
 |---|---|---|
 | `TERDUT_ADDR` | `:8080` | TCP address to listen on |
 | `TERDUT_DB_PATH` | `terdut.db` | Path to the SQLite database file |
-| `TERDUT_ARCHIVE_AFTER` | `168h` (7d) | How long a resolved alert stays in the default list before being auto-archived |
+| `TERDUT_ARCHIVE_AFTER` | `168h` (7d) | How long a resolved alert or incident stays in the default list before being auto-archived |
 | `TERDUT_STALE_AFTER` | `6h` | How long a firing alert may go without a refreshing webhook before it is treated as resolved — **must exceed your Alertmanager `repeat_interval`** |
 
 Durations use Go syntax (`30m`, `12h`, `168h`). An unparseable value falls back to the default.
@@ -83,6 +84,67 @@ route:
 
 The webhook endpoint requires no authentication.
 
+---
+
+## Alerts and incidents
+
+There are two objects, and the difference between them is the whole design.
+
+**An alert is Alertmanager's record.** It has two states, `firing` and
+`resolved`, one row per fingerprint, and no human ever writes to it. The API
+exposes alerts read-only.
+
+**An incident is the work item.** It goes `triggered → acknowledged → resolved`,
+carries an assignee, a snooze, notes and a timeline, and is the only thing people
+act on. Many alerts belong to one incident.
+
+### Correlation uses Alertmanager's `groupKey`
+
+Alertmanager has already grouped alerts according to the `group_by` routing tree
+you configured, and it sends the resulting `groupKey` and `groupLabels` on every
+webhook. Incidents adopt that answer rather than re-grouping alerts a second
+time — if you want different correlation, change `group_by` in
+`alertmanager.yml` and terdut follows.
+
+At most one incident is open per `groupKey` at a time. Alerts firing in a group
+that already has an open incident join it. The incident's `severity` is a
+high-water mark — the highest `severity` label any of its alerts has carried — so
+an incident that hit `critical` still reads as critical after the critical alert
+clears.
+
+### An incident opens only on a new occurrence
+
+An incident opens when an alert **transitions into firing**: a fingerprint that
+was never seen, an alert with a newer `startsAt`, or a resolved alert that
+started again. The unchanged firing notifications Alertmanager re-sends every
+`repeat_interval` are none of those, and open nothing.
+
+This is what makes closing an incident by hand mean something. Without the rule,
+`POST /api/incidents/{id}/resolve` would be undone by the next re-send of an
+alert that never stopped firing.
+
+### Leaving the open state
+
+- **Automatically**, once every alert under the incident has stopped firing —
+  whether by a resolved webhook or by the sweeper's
+  [stale-alert expiry](#stale-alert-expiry). The incident gets
+  `"resolution_source": "alerts"`.
+- **By hand**, via `POST /api/incidents/{id}/resolve`
+  (`"resolution_source": "manual"`). This is **terminal**: a later occurrence in
+  that group opens a *new* incident rather than reopening this one. If the alert
+  underneath never stops firing, the incident stays closed — that is what
+  resolving by hand asserts.
+
+To quieten an incident you expect to come back, snooze it instead
+(`POST /api/incidents/{id}/snooze`). A snooze hides the incident from the default
+list without closing it, and expires by simply falling into the past.
+
+### On-call assignment
+
+A new incident is assigned to whoever holds today's schedule entry at the moment
+it opens (`GET /api/schedule/current`). If nobody is scheduled it opens
+unassigned. Reassign with `POST /api/incidents/{id}/assign`.
+
 ### Stale alert expiry
 
 A resolved webhook is the only signal that an alert has stopped firing, so a
@@ -99,6 +161,9 @@ than your `repeat_interval` (default 4h), or live alerts will be resolved
 prematurely. Alerts resolved this way are marked `"resolution_source": "expiry"`
 to distinguish them from a real Alertmanager resolve (`"alertmanager"`).
 
+An expiry cascades: once it leaves an incident with nothing firing under it, the
+incident resolves too, in the same sweep.
+
 ---
 
 ## API reference
@@ -128,22 +193,84 @@ Authorization: Bearer <api-key>
 |---|---|---|
 | `POST` | `/api/alertmanager/webhook` | Alertmanager v4 webhook receiver (no auth) |
 
-### Alerts
+### Incidents
 
 | Method | Path | Description |
 |---|---|---|
-| `GET` | `/api/alerts` | List alerts. Filters: `?status=firing\|resolved`, `?name=`, `?archived=true`, `?from=YYYY-MM-DD`, `?to=YYYY-MM-DD`, `?limit=` (default 50, max 500) |
+| `GET` | `/api/incidents` | List incidents. Filters: `?status=triggered\|acknowledged\|resolved`, `?severity=`, `?assigned_to=<user id>`, `?archived=true`, `?snoozed=true`, `?from=YYYY-MM-DD`, `?to=YYYY-MM-DD`, `?sort=severity`, `?limit=` (default 50, max 500) |
+| `GET` | `/api/incidents/{id}` | Get single incident, with its alerts inline |
+| `GET` | `/api/incidents/{id}/alerts` | Alerts under this incident |
+| `GET` | `/api/incidents/{id}/timeline` | Full event history, chronological |
+| `POST` | `/api/incidents/{id}/acknowledge` | Acknowledge (stamps authed user + time) |
+| `DELETE` | `/api/incidents/{id}/acknowledge` | Clear acknowledgement, back to `triggered` |
+| `POST` | `/api/incidents/{id}/resolve` | Close by hand — **terminal**, see above |
+| `POST` | `/api/incidents/{id}/assign` | Reassign `{"user_id"}` |
+| `POST` | `/api/incidents/{id}/snooze` | Hide until `{"until": RFC3339}` or `{"duration": "2h"}` |
+| `DELETE` | `/api/incidents/{id}/snooze` | Un-snooze |
+| `POST` | `/api/incidents/{id}/archive` | Archive (hides from the default list) |
+| `DELETE` | `/api/incidents/{id}/archive` | Un-archive |
+| `POST` | `/api/incidents/{id}/notes` | Add a note `{"content"}` |
+| `DELETE` | `/api/incidents/{id}/notes/{eventID}` | Delete own note |
+
+With no `?status=` filter, `GET /api/incidents` returns **open** incidents only —
+the queue an on-call person wants. Currently snoozed and archived incidents are
+excluded unless asked for. Actions that only make sense on an open incident
+return `409` once it is resolved.
+
+Notes are ordinary timeline events of type `note`; only they are deletable, and
+only by their author. The rest of the timeline is a record of what happened.
+
+#### The incident object
+
+| Field | Type | Notes |
+|---|---|---|
+| `id` | integer | Server-assigned |
+| `group_key` | string | Alertmanager's `groupKey` — opaque, treat as an identifier |
+| `title` | string | Rendered from `groupLabels` |
+| `group_labels` | object | String→string, as sent by Alertmanager |
+| `status` | string | `"triggered"`, `"acknowledged"` or `"resolved"` |
+| `severity` | string | *optional* — high-water mark across the incident's alerts; never lowered |
+| `triggered_at` | timestamp | When the incident opened |
+| `acknowledged_by_id` / `acknowledged_by` / `acknowledged_at` | | *optional* — user id, username, time |
+| `assigned_to_id` / `assigned_to` | | *optional* — user id, username |
+| `snoozed_until` | timestamp | *optional* — a value in the past reads as not snoozed |
+| `resolved_at` | timestamp | *optional* |
+| `resolution_source` | string | *optional* — `"alerts"` or `"manual"` |
+| `archived_at` | timestamp | *optional* |
+| `alerts` | array | Only on `GET /api/incidents/{id}` |
+
+Treat `resolution_source` as an open set, as with the alert field of the same
+name: degrade unknown values to "resolved, reason unknown".
+
+#### The timeline event object
+
+| Field | Type | Notes |
+|---|---|---|
+| `id` | integer | |
+| `incident_id` | integer | |
+| `type` | string | See below — treat as an open set |
+| `user_id` / `username` | | *optional* — absent when the server acted rather than a person |
+| `alert_id` | integer | *optional* — the alert an `alert_added` / `alert_resolved` event refers to |
+| `detail` | string | *optional* — the note body, the snooze deadline, etc. |
+| `created_at` | timestamp | |
+
+Types written today: `triggered`, `alert_added`, `alert_resolved`,
+`acknowledged`, `unacknowledged`, `assigned`, `snoozed`, `unsnoozed`, `resolved`,
+`note`. On an `assigned` event `user_id` is the **assignee**, not the actor. New
+types may be added; render unknown ones generically rather than dropping them.
+
+### Alerts
+
+Alerts are read-only. Everything a person does happens on the incident.
+
+| Method | Path | Description |
+|---|---|---|
+| `GET` | `/api/alerts` | List alerts. Filters: `?status=firing\|resolved`, `?name=`, `?incident_id=`, `?archived=true`, `?from=YYYY-MM-DD`, `?to=YYYY-MM-DD`, `?limit=` (default 50, max 500) |
 | `GET` | `/api/alerts/{id}` | Get single alert |
-| `POST` | `/api/alerts/{id}/acknowledge` | Acknowledge alert (stamps authed user + time) |
-| `DELETE` | `/api/alerts/{id}/acknowledge` | Clear acknowledgement |
-| `POST` | `/api/alerts/{id}/archive` | Archive alert (hides it from the default list) |
-| `DELETE` | `/api/alerts/{id}/archive` | Un-archive alert |
-| `GET` | `/api/alerts/{id}/comments` | List comments (chronological) |
-| `POST` | `/api/alerts/{id}/comments` | Add comment `{"content"}` |
-| `DELETE` | `/api/alerts/{id}/comments/{commentID}` | Delete own comment |
 
 Archived alerts are hidden from `GET /api/alerts` unless `?archived=true` is
-passed. Resolved alerts carry `resolution_source`: `"alertmanager"` for a real
+passed; alert archiving is automatic housekeeping by the sweeper, not a user
+action. Resolved alerts carry `resolution_source`: `"alertmanager"` for a real
 resolved webhook, `"expiry"` when the sweeper inferred it (see
 [Stale alert expiry](#stale-alert-expiry)).
 
@@ -165,9 +292,7 @@ when unset, so clients must treat them as nullable.
 | `ends_at` | timestamp | *optional* — absent while no end is known |
 | `generator_url` | string | Link back to the originating Prometheus |
 | `received_at` | timestamp | When the server last accepted a webhook for this alert — see below |
-| `acknowledged_by_id` | integer | *optional* — user id |
-| `acknowledged_by` | string | *optional* — username |
-| `acknowledged_at` | timestamp | *optional* |
+| `incident_id` | integer | *optional* — the most recent incident this alert belongs to |
 | `resolution_source` | string | *optional* — `"alertmanager"` or `"expiry"` |
 | `archived_at` | timestamp | *optional* — set while archived |
 
@@ -238,15 +363,47 @@ unknown" rather than being rejected.
 
 ### Statistics
 
-All stat endpoints accept optional `?from=YYYY-MM-DD` and `?to=YYYY-MM-DD` to filter by `received_at`. Archived alerts are excluded, matching the default alert list.
+All stat endpoints accept optional `?from=YYYY-MM-DD` and `?to=YYYY-MM-DD`, and exclude archived rows to match the default list views. Alert stats filter on `received_at`; incident stats filter on `triggered_at`.
 
 | Method | Path | Description |
 |---|---|---|
+| `GET` | `/api/stats/incidents` | `{total, triggered, acknowledged, resolved, mtta_seconds, mttr_seconds}` |
 | `GET` | `/api/stats/alerts` | `{total, firing, resolved}` counts |
 | `GET` | `/api/stats/alerts/top` | Most frequent alert names. `?limit=` (default 10, max 100) |
 | `GET` | `/api/stats/alerts/by-hour` | Count per hour-of-day (UTC), all 24 slots returned |
 | `GET` | `/api/stats/alerts/by-day` | Count per day-of-week, all 7 slots with names returned |
 
+`mtta_seconds` (time to acknowledge) and `mttr_seconds` (time to resolve) are
+averages over incidents that have actually been acknowledged or resolved, and are
+**null** until there are any — null means "no data", not zero.
+
+---
+
+## Upgrading to incidents
+
+The incidents release moves the workflow off alerts, which is a **breaking API
+change**. These endpoints are gone:
+
+| Removed | Replacement |
+|---|---|
+| `POST`/`DELETE` `/api/alerts/{id}/acknowledge` | `POST`/`DELETE` `/api/incidents/{id}/acknowledge` |
+| `POST`/`DELETE` `/api/alerts/{id}/archive` | `POST`/`DELETE` `/api/incidents/{id}/archive` (alert archiving is now sweeper-only) |
+| `GET`/`POST` `/api/alerts/{id}/comments` | `GET /api/incidents/{id}/timeline`, `POST /api/incidents/{id}/notes` |
+| `DELETE /api/alerts/{id}/comments/{commentID}` | `DELETE /api/incidents/{id}/notes/{eventID}` |
+
+The alert object also drops `acknowledged_by_id`, `acknowledged_by` and
+`acknowledged_at`, and gains `incident_id`.
+
+Migration `008_incidents.sql` runs automatically on start and preserves existing
+data: every alert gets a backfilled incident carrying its acknowledgement, and
+comments become timeline notes. Backfilled incidents have a `group_key` of
+`backfill:<fingerprint>` — there is no historical `groupKey` to correlate on, so
+they are one-per-alert rather than grouped.
+
+Nothing about the two documented alert contracts changes: `received_at` is still
+advanced on every accepted webhook, and `resolution_source` still means what it
+did.
+
 ---
 
 ## Development