Expire stale firing alerts

A resolved webhook was the only path out of the firing state, so a
notification that was dropped, silenced, or lost to a restart pinned an
alert as firing forever — Prometheus showed it resolved while
terdut-server kept listing it. The archiver only ever touched resolved
alerts, and both the list and stats queries compared status with plain
equality, so a stale row was indistinguishable from a live one.

A sweeper pass now resolves firing alerts on either of two signals: the
ends_at watermark Alertmanager sets on outgoing firing notifications has
passed (plus a grace period for clock skew), or no webhook has refreshed
the alert within TERDUT_STALE_AFTER (default 6h, above Alertmanager's 4h
repeat_interval). Such alerts get resolution_source = 'expiry',
distinguishing them from a real 'alertmanager' resolve.

Two related webhook bugs fixed alongside:

  - The upsert had no ordering guard, so a retried firing notification
    arriving after the resolved one resurrected the alert. Payloads for
    an older alert instance are now discarded: a stale retry carries the
    same startsAt, a genuine re-fire a newer one.
  - archived_at was never cleared on re-fire, leaving a re-fired alert
    archived and invisible in the default list.

Stats now exclude archived alerts to match the default list view; this
lowers historical firing/resolved totals.

The chart exposes both sweeper durations via sweeper.staleAfter and
sweeper.archiveAfter.

README.md

@@ -57,6 +57,12 @@ docker run -p 8080:8080 -v $(pwd)/data:/data \
 |---|---|---|
 | `TERDUT_ADDR` | `:8080` | TCP address to listen on |
 | `TERDUT_DB_PATH` | `terdut.db` | Path to the SQLite database file |
+| `TERDUT_ARCHIVE_AFTER` | `168h` (7d) | How long a resolved alert stays in the default list before being auto-archived |
+| `TERDUT_STALE_AFTER` | `6h` | How long a firing alert may go without a refreshing webhook before it is treated as resolved — **must exceed your Alertmanager `repeat_interval`** |
+
+Durations use Go syntax (`30m`, `12h`, `168h`). An unparseable value falls back to the default.
+
+In the Helm chart the two sweeper durations are set via `sweeper.staleAfter` and `sweeper.archiveAfter`.
 
 ---
 
@@ -77,6 +83,22 @@ route:
 
 The webhook endpoint requires no authentication.
 
+### Stale alert expiry
+
+A resolved webhook is the only signal that an alert has stopped firing, so a
+notification that is dropped, silenced, or lost to a restart would otherwise pin
+that alert as firing forever. A background sweeper resolves firing alerts that
+Alertmanager has stopped refreshing, using either signal:
+
+- the `endsAt` watermark on the last notification has passed, or
+- no webhook has refreshed the alert within `TERDUT_STALE_AFTER`.
+
+Alertmanager re-sends firing notifications every `repeat_interval`, which is what
+keeps a live alert fresh — so `TERDUT_STALE_AFTER` must be comfortably larger
+than your `repeat_interval` (default 4h), or live alerts will be resolved
+prematurely. Alerts resolved this way are marked `"resolution_source": "expiry"`
+to distinguish them from a real Alertmanager resolve (`"alertmanager"`).
+
 ---
 
 ## API reference
@@ -110,14 +132,21 @@ Authorization: Bearer <api-key>
 
 | Method | Path | Description |
 |---|---|---|
-| `GET` | `/api/alerts` | List alerts. Filters: `?status=firing\|resolved`, `?name=`, `?from=YYYY-MM-DD`, `?to=YYYY-MM-DD`, `?limit=` (default 50, max 500) |
+| `GET` | `/api/alerts` | List alerts. Filters: `?status=firing\|resolved`, `?name=`, `?archived=true`, `?from=YYYY-MM-DD`, `?to=YYYY-MM-DD`, `?limit=` (default 50, max 500) |
 | `GET` | `/api/alerts/{id}` | Get single alert |
 | `POST` | `/api/alerts/{id}/acknowledge` | Acknowledge alert (stamps authed user + time) |
 | `DELETE` | `/api/alerts/{id}/acknowledge` | Clear acknowledgement |
+| `POST` | `/api/alerts/{id}/archive` | Archive alert (hides it from the default list) |
+| `DELETE` | `/api/alerts/{id}/archive` | Un-archive alert |
 | `GET` | `/api/alerts/{id}/comments` | List comments (chronological) |
 | `POST` | `/api/alerts/{id}/comments` | Add comment `{"content"}` |
 | `DELETE` | `/api/alerts/{id}/comments/{commentID}` | Delete own comment |
 
+Archived alerts are hidden from `GET /api/alerts` unless `?archived=true` is
+passed. Resolved alerts carry `resolution_source`: `"alertmanager"` for a real
+resolved webhook, `"expiry"` when the sweeper inferred it (see
+[Stale alert expiry](#stale-alert-expiry)).
+
 ### On-call schedule
 
 | Method | Path | Description |
@@ -129,7 +158,7 @@ Authorization: Bearer <api-key>
 
 ### Statistics
 
-All stat endpoints accept optional `?from=YYYY-MM-DD` and `?to=YYYY-MM-DD` to filter by `received_at`.
+All stat endpoints accept optional `?from=YYYY-MM-DD` and `?to=YYYY-MM-DD` to filter by `received_at`. Archived alerts are excluded, matching the default alert list.
 
 | Method | Path | Description |
 |---|---|---|