Expire stale firing alerts

A resolved webhook was the only path out of the firing state, so a
notification that was dropped, silenced, or lost to a restart pinned an
alert as firing forever — Prometheus showed it resolved while
terdut-server kept listing it. The archiver only ever touched resolved
alerts, and both the list and stats queries compared status with plain
equality, so a stale row was indistinguishable from a live one.

A sweeper pass now resolves firing alerts on either of two signals: the
ends_at watermark Alertmanager sets on outgoing firing notifications has
passed (plus a grace period for clock skew), or no webhook has refreshed
the alert within TERDUT_STALE_AFTER (default 6h, above Alertmanager's 4h
repeat_interval). Such alerts get resolution_source = 'expiry',
distinguishing them from a real 'alertmanager' resolve.

Two related webhook bugs fixed alongside:

  - The upsert had no ordering guard, so a retried firing notification
    arriving after the resolved one resurrected the alert. Payloads for
    an older alert instance are now discarded: a stale retry carries the
    same startsAt, a genuine re-fire a newer one.
  - archived_at was never cleared on re-fire, leaving a re-fired alert
    archived and invisible in the default list.

Stats now exclude archived alerts to match the default list view; this
lowers historical firing/resolved totals.

The chart exposes both sweeper durations via sweeper.staleAfter and
sweeper.archiveAfter.

.github/workflows/release.yml

@@ -4,6 +4,7 @@ on:
   push:
     tags:
       - 'v*'
+  workflow_dispatch:
 
 jobs:
   build:
@@ -94,12 +95,16 @@ jobs:
       - name: Update chart versions
         run: |
           VERSION="${{ github.ref_name }}"
-          CHART_VERSION="${VERSION#v}"
-          sed -i "s/^version:.*/version: ${CHART_VERSION}/" charts/terdut-server/Chart.yaml
-          sed -i "s/^appVersion:.*/appVersion: \"${VERSION}\"/" charts/terdut-server/Chart.yaml
+          if [[ "$VERSION" =~ ^v[0-9] ]]; then
+            CHART_VERSION="${VERSION#v}"
+            sed -i "s/^version:.*/version: ${CHART_VERSION}/" charts/terdut-server/Chart.yaml
+            sed -i "s/^appVersion:.*/appVersion: \"${VERSION}\"/" charts/terdut-server/Chart.yaml
+          fi
 
       - name: Run chart-releaser
         uses: helm/chart-releaser-action@v1.6.0
+        with:
+          skip_existing: true
         env:
           CR_TOKEN: "${{ secrets.GITHUB_TOKEN }}"
 

README.md

@@ -57,6 +57,12 @@ docker run -p 8080:8080 -v $(pwd)/data:/data \
 |---|---|---|
 | `TERDUT_ADDR` | `:8080` | TCP address to listen on |
 | `TERDUT_DB_PATH` | `terdut.db` | Path to the SQLite database file |
+| `TERDUT_ARCHIVE_AFTER` | `168h` (7d) | How long a resolved alert stays in the default list before being auto-archived |
+| `TERDUT_STALE_AFTER` | `6h` | How long a firing alert may go without a refreshing webhook before it is treated as resolved — **must exceed your Alertmanager `repeat_interval`** |
+
+Durations use Go syntax (`30m`, `12h`, `168h`). An unparseable value falls back to the default.
+
+In the Helm chart the two sweeper durations are set via `sweeper.staleAfter` and `sweeper.archiveAfter`.
 
 ---
 
@@ -77,6 +83,22 @@ route:
 
 The webhook endpoint requires no authentication.
 
+### Stale alert expiry
+
+A resolved webhook is the only signal that an alert has stopped firing, so a
+notification that is dropped, silenced, or lost to a restart would otherwise pin
+that alert as firing forever. A background sweeper resolves firing alerts that
+Alertmanager has stopped refreshing, using either signal:
+
+- the `endsAt` watermark on the last notification has passed, or
+- no webhook has refreshed the alert within `TERDUT_STALE_AFTER`.
+
+Alertmanager re-sends firing notifications every `repeat_interval`, which is what
+keeps a live alert fresh — so `TERDUT_STALE_AFTER` must be comfortably larger
+than your `repeat_interval` (default 4h), or live alerts will be resolved
+prematurely. Alerts resolved this way are marked `"resolution_source": "expiry"`
+to distinguish them from a real Alertmanager resolve (`"alertmanager"`).
+
 ---
 
 ## API reference
@@ -110,14 +132,21 @@ Authorization: Bearer <api-key>
 
 | Method | Path | Description |
 |---|---|---|
-| `GET` | `/api/alerts` | List alerts. Filters: `?status=firing\|resolved`, `?name=`, `?from=YYYY-MM-DD`, `?to=YYYY-MM-DD`, `?limit=` (default 50, max 500) |
+| `GET` | `/api/alerts` | List alerts. Filters: `?status=firing\|resolved`, `?name=`, `?archived=true`, `?from=YYYY-MM-DD`, `?to=YYYY-MM-DD`, `?limit=` (default 50, max 500) |
 | `GET` | `/api/alerts/{id}` | Get single alert |
 | `POST` | `/api/alerts/{id}/acknowledge` | Acknowledge alert (stamps authed user + time) |
 | `DELETE` | `/api/alerts/{id}/acknowledge` | Clear acknowledgement |
+| `POST` | `/api/alerts/{id}/archive` | Archive alert (hides it from the default list) |
+| `DELETE` | `/api/alerts/{id}/archive` | Un-archive alert |
 | `GET` | `/api/alerts/{id}/comments` | List comments (chronological) |
 | `POST` | `/api/alerts/{id}/comments` | Add comment `{"content"}` |
 | `DELETE` | `/api/alerts/{id}/comments/{commentID}` | Delete own comment |
 
+Archived alerts are hidden from `GET /api/alerts` unless `?archived=true` is
+passed. Resolved alerts carry `resolution_source`: `"alertmanager"` for a real
+resolved webhook, `"expiry"` when the sweeper inferred it (see
+[Stale alert expiry](#stale-alert-expiry)).
+
 ### On-call schedule
 
 | Method | Path | Description |
@@ -129,7 +158,7 @@ Authorization: Bearer <api-key>
 
 ### Statistics
 
-All stat endpoints accept optional `?from=YYYY-MM-DD` and `?to=YYYY-MM-DD` to filter by `received_at`.
+All stat endpoints accept optional `?from=YYYY-MM-DD` and `?to=YYYY-MM-DD` to filter by `received_at`. Archived alerts are excluded, matching the default alert list.
 
 | Method | Path | Description |
 |---|---|---|

charts/terdut-server/Chart.yaml

@@ -2,5 +2,5 @@ apiVersion: v2
 name: terdut-server
 description: A Helm chart for Terminal Duty — on-call alert management server
 type: application
-version: 0.1.0
+version: 0.2.0
 appVersion: "latest"

charts/terdut-server/templates/_helpers.tpl

@@ -32,3 +32,7 @@ app.kubernetes.io/managed-by: {{ .Release.Service }}
 app.kubernetes.io/name: {{ include "terdut-server.name" . }}
 app.kubernetes.io/instance: {{ .Release.Name }}
 {{- end }}
+
+{{- define "terdut-server.bootstrapSecretName" -}}
+{{- .Values.bootstrap.secretName | default (printf "%s-admin-key" (include "terdut-server.fullname" .)) }}
+{{- end }}

charts/terdut-server/templates/bootstrap-job.yaml

@@ -0,0 +1,91 @@
+{{- if .Values.bootstrap.enabled }}
+---
+apiVersion: batch/v1
+kind: Job
+metadata:
+  name: {{ include "terdut-server.fullname" . }}-bootstrap
+  namespace: {{ .Release.Namespace }}
+  labels:
+    {{- include "terdut-server.labels" . | nindent 4 }}
+  annotations:
+    helm.sh/hook: post-install,post-upgrade
+    helm.sh/hook-weight: "0"
+    helm.sh/hook-delete-policy: before-hook-creation,hook-succeeded
+spec:
+  backoffLimit: 3
+  template:
+    metadata:
+      labels:
+        {{- include "terdut-server.selectorLabels" . | nindent 8 }}
+        app.kubernetes.io/component: bootstrap
+    spec:
+      restartPolicy: OnFailure
+      serviceAccountName: {{ include "terdut-server.fullname" . }}-bootstrap
+      containers:
+        - name: bootstrap
+          image: alpine:3
+          command:
+            - /bin/sh
+            - -c
+            - |
+              apk add --no-cache curl > /dev/null 2>&1
+
+              SERVICE_URL="http://{{ include "terdut-server.fullname" . }}:{{ .Values.service.port }}"
+              SECRET_NAME="{{ include "terdut-server.bootstrapSecretName" . }}"
+              K8S_API="https://kubernetes.default.svc"
+              SA_TOKEN="$(cat /var/run/secrets/kubernetes.io/serviceaccount/token)"
+              CA_CERT="/var/run/secrets/kubernetes.io/serviceaccount/ca.crt"
+              NAMESPACE="$(cat /var/run/secrets/kubernetes.io/serviceaccount/namespace)"
+
+              echo "Waiting for terdut-server to be ready..."
+              RETRIES=60
+              while [ "$RETRIES" -gt 0 ]; do
+                curl -sf "$SERVICE_URL/healthz" > /dev/null 2>&1 && break
+                RETRIES=$((RETRIES - 1))
+                sleep 2
+              done
+              if [ "$RETRIES" -eq 0 ]; then
+                echo "Timed out waiting for server to be ready."
+                exit 1
+              fi
+              echo "Server is ready."
+
+              RESPONSE=$(curl -s -w "\n%{http_code}" -X POST "$SERVICE_URL/api/bootstrap" \
+                -H "Content-Type: application/json" \
+                -d '{"username":"{{ .Values.bootstrap.username }}","email":"{{ .Values.bootstrap.email }}"}')
+
+              HTTP_CODE=$(echo "$RESPONSE" | tail -1)
+              BODY=$(echo "$RESPONSE" | head -1)
+
+              if [ "$HTTP_CODE" = "403" ]; then
+                echo "Server already bootstrapped, nothing to do."
+                exit 0
+              fi
+
+              if [ "$HTTP_CODE" != "201" ]; then
+                echo "Bootstrap failed (HTTP $HTTP_CODE): $BODY"
+                exit 1
+              fi
+
+              API_KEY=$(echo "$BODY" | grep -o '"key":"[^"]*"' | cut -d'"' -f4)
+              if [ -z "$API_KEY" ]; then
+                echo "Failed to extract API key from response."
+                exit 1
+              fi
+
+              echo "Bootstrap succeeded. Storing API key in secret '$SECRET_NAME'."
+
+              HTTP_CODE=$(curl -s -o /dev/null -w "%{http_code}" \
+                -X POST "$K8S_API/api/v1/namespaces/$NAMESPACE/secrets" \
+                --cacert "$CA_CERT" \
+                -H "Authorization: Bearer $SA_TOKEN" \
+                -H "Content-Type: application/json" \
+                -d "$(printf '{"apiVersion":"v1","kind":"Secret","metadata":{"name":"%s"},"stringData":{"api-key":"%s"}}' "$SECRET_NAME" "$API_KEY")")
+
+              if [ "$HTTP_CODE" != "201" ]; then
+                echo "Failed to create secret (HTTP $HTTP_CODE)."
+                exit 1
+              fi
+
+              echo "Secret '$SECRET_NAME' created successfully."
+{{- end }}

charts/terdut-server/templates/bootstrap-rbac.yaml

@@ -0,0 +1,50 @@
+{{- if .Values.bootstrap.enabled }}
+---
+apiVersion: v1
+kind: ServiceAccount
+metadata:
+  name: {{ include "terdut-server.fullname" . }}-bootstrap
+  namespace: {{ .Release.Namespace }}
+  labels:
+    {{- include "terdut-server.labels" . | nindent 4 }}
+  annotations:
+    helm.sh/hook: post-install,post-upgrade
+    helm.sh/hook-weight: "-1"
+    helm.sh/hook-delete-policy: before-hook-creation
+---
+apiVersion: rbac.authorization.k8s.io/v1
+kind: Role
+metadata:
+  name: {{ include "terdut-server.fullname" . }}-bootstrap
+  namespace: {{ .Release.Namespace }}
+  labels:
+    {{- include "terdut-server.labels" . | nindent 4 }}
+  annotations:
+    helm.sh/hook: post-install,post-upgrade
+    helm.sh/hook-weight: "-1"
+    helm.sh/hook-delete-policy: before-hook-creation
+rules:
+  - apiGroups: [""]
+    resources: ["secrets"]
+    verbs: ["create"]
+---
+apiVersion: rbac.authorization.k8s.io/v1
+kind: RoleBinding
+metadata:
+  name: {{ include "terdut-server.fullname" . }}-bootstrap
+  namespace: {{ .Release.Namespace }}
+  labels:
+    {{- include "terdut-server.labels" . | nindent 4 }}
+  annotations:
+    helm.sh/hook: post-install,post-upgrade
+    helm.sh/hook-weight: "-1"
+    helm.sh/hook-delete-policy: before-hook-creation
+roleRef:
+  apiGroup: rbac.authorization.k8s.io
+  kind: Role
+  name: {{ include "terdut-server.fullname" . }}-bootstrap
+subjects:
+  - kind: ServiceAccount
+    name: {{ include "terdut-server.fullname" . }}-bootstrap
+    namespace: {{ .Release.Namespace }}
+{{- end }}

charts/terdut-server/templates/deployment.yaml

@@ -29,6 +29,10 @@ spec:
               value: ":{{ .Values.service.port }}"
             - name: TERDUT_DB_PATH
               value: "/data/terdut.db"
+            - name: TERDUT_STALE_AFTER
+              value: "{{ .Values.sweeper.staleAfter }}"
+            - name: TERDUT_ARCHIVE_AFTER
+              value: "{{ .Values.sweeper.archiveAfter }}"
           volumeMounts:
             - name: data
               mountPath: /data

charts/terdut-server/values.yaml

@@ -14,3 +14,17 @@ storage:
 service:
   type: ClusterIP
   port: 8080
+
+sweeper:
+  # How long a firing alert may go without a refreshing webhook before it is
+  # treated as resolved. Must exceed your Alertmanager repeat_interval.
+  staleAfter: 6h
+  # How long a resolved alert stays in the default list before auto-archiving.
+  archiveAfter: 168h
+
+bootstrap:
+  enabled: true
+  username: admin
+  email: admin@example.com
+  # secretName overrides the default of <fullname>-admin-key
+  secretName: ""

cmd/terdut/main.go

@@ -41,6 +41,8 @@ func main() {
 	ctx, stop := signal.NotifyContext(context.Background(), syscall.SIGINT, syscall.SIGTERM)
 	defer stop()
 
+	go api.StartArchiver(ctx, database, cfg.ArchiveAfter, cfg.StaleAfter)
+
 	go func() {
 		log.Printf("terdut-server %s listening on %s", version, cfg.Addr)
 		if err := srv.ListenAndServe(); err != nil && err != http.ErrServerClosed {

internal/api/alertmanager.go

@@ -8,6 +8,13 @@ import (
 	"time"
 )
 
+// Values for alerts.resolution_source, recording why an alert left the firing
+// state: a real Alertmanager notification, or inference by the sweeper.
+const (
+	resolutionAlertmanager = "alertmanager"
+	resolutionExpiry       = "expiry"
+)
+
 // amPayload mirrors the Alertmanager webhook v4 payload.
 type amPayload struct {
 	Version string    `json:"version"`
@@ -39,28 +46,51 @@ func handleAlertmanagerWebhook(db *sql.DB) http.HandlerFunc {
 			labelsJSON, _ := json.Marshal(a.Labels)
 			annotationsJSON, _ := json.Marshal(a.Annotations)
 
-			// Alertmanager uses zero time ("0001-01-01T00:00:00Z") to mean "still firing".
+			// Zero time ("0001-01-01T00:00:00Z") means "no end known" — that is the
+			// convention of Alertmanager's ingest API. Outgoing notifications
+			// normally carry a real future endsAt instead, which is the watermark
+			// the sweeper uses to expire alerts that stop being refreshed.
 			var endsAtUnix *int64
 			if a.EndsAt.Year() > 1 {
 				t := a.EndsAt.Unix()
 				endsAtUnix = &t
 			}
 
+			var resolutionSource *string
+			if a.Status == "resolved" {
+				s := resolutionAlertmanager
+				resolutionSource = &s
+			}
+
+			// The WHERE clause discards payloads that describe an alert instance
+			// older than the stored one. Alertmanager retries failed notifications,
+			// so a stale firing retry can arrive after the resolved one; it carries
+			// the same startsAt, whereas a genuine re-fire carries a newer one.
+			// Within a single instance, resolution is terminal.
 			_, err := db.ExecContext(r.Context(), `
 				INSERT INTO alerts
-					(fingerprint, name, status, labels, annotations, starts_at, ends_at, generator_url, received_at)
-				VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?)
+					(fingerprint, name, status, labels, annotations, starts_at, ends_at,
+					 generator_url, received_at, resolution_source)
+				VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
 				ON CONFLICT(fingerprint) DO UPDATE SET
-					status        = excluded.status,
-					labels        = excluded.labels,
-					annotations   = excluded.annotations,
-					ends_at       = excluded.ends_at,
-					generator_url = excluded.generator_url,
-					received_at   = excluded.received_at`,
+					status            = excluded.status,
+					labels            = excluded.labels,
+					annotations       = excluded.annotations,
+					starts_at         = excluded.starts_at,
+					ends_at           = excluded.ends_at,
+					generator_url     = excluded.generator_url,
+					received_at       = excluded.received_at,
+					resolution_source = excluded.resolution_source,
+					-- A re-fire makes the alert current again, so it leaves the archive.
+					archived_at       = CASE WHEN excluded.status = 'firing'
+					                        THEN NULL ELSE alerts.archived_at END
+				WHERE excluded.starts_at > alerts.starts_at
+				   OR (excluded.starts_at = alerts.starts_at
+				       AND NOT (alerts.status = 'resolved' AND excluded.status = 'firing'))`,
 				a.Fingerprint, name, a.Status,
 				string(labelsJSON), string(annotationsJSON),
 				a.StartsAt.Unix(), endsAtUnix,
-				a.GeneratorURL, now,
+				a.GeneratorURL, now, resolutionSource,
 			)
 			if err != nil {
 				log.Printf("upsert alert %s: %v", a.Fingerprint, err)

internal/api/alerts.go

@@ -20,7 +20,8 @@ const alertSelectFrom = `
 	SELECT a.id, a.fingerprint, a.name, a.status,
 	       a.labels, a.annotations,
 	       a.starts_at, a.ends_at, a.generator_url, a.received_at,
-	       a.acknowledged_by, a.acknowledged_at, u.username
+	       a.acknowledged_by, a.acknowledged_at, u.username,
+	       a.resolution_source, a.archived_at
 	FROM   alerts a
 	LEFT   JOIN users u ON u.id = a.acknowledged_by`
 
@@ -39,6 +40,12 @@ func handleListAlerts(db *sql.DB) http.HandlerFunc {
 			where = append(where, "a.name = ?")
 			args = append(args, name)
 		}
+		if archived := q.Get("archived"); archived == "true" {
+			where = append(where, "a.archived_at IS NOT NULL")
+		} else {
+			where = append(where, "a.archived_at IS NULL")
+		}
+
 		if from := q.Get("from"); from != "" {
 			if t, err := time.Parse("2006-01-02", from); err == nil {
 				where = append(where, "a.received_at >= ?")
@@ -169,7 +176,7 @@ func scanAlert(s scanner) (models.Alert, error) {
 	var a models.Alert
 	var labelsJSON, annotationsJSON string
 	var startsAtUnix, receivedAtUnix int64
-	var endsAtUnix, ackAtUnix *int64
+	var endsAtUnix, ackAtUnix, archivedAtUnix *int64
 	var ackByID *int64
 	var ackByUser *string
 
@@ -179,6 +186,7 @@ func scanAlert(s scanner) (models.Alert, error) {
 		&startsAtUnix, &endsAtUnix,
 		&a.GeneratorURL, &receivedAtUnix,
 		&ackByID, &ackAtUnix, &ackByUser,
+		&a.ResolutionSource, &archivedAtUnix,
 	); err != nil {
 		return a, err
 	}
@@ -197,5 +205,52 @@ func scanAlert(s scanner) (models.Alert, error) {
 		a.AcknowledgedByUser = ackByUser
 		a.AcknowledgedAt = &t
 	}
+	if archivedAtUnix != nil {
+		t := time.Unix(*archivedAtUnix, 0).UTC()
+		a.ArchivedAt = &t
+	}
 	return a, nil
 }
+
+func handleArchive(db *sql.DB) http.HandlerFunc {
+	return func(w http.ResponseWriter, r *http.Request) {
+		id, err := strconv.ParseInt(chi.URLParam(r, "id"), 10, 64)
+		if err != nil {
+			respond(w, http.StatusBadRequest, errResp("invalid alert id"))
+			return
+		}
+		res, err := db.ExecContext(r.Context(),
+			"UPDATE alerts SET archived_at = unixepoch() WHERE id = ?", id)
+		if err != nil {
+			respond(w, http.StatusInternalServerError, errResp("internal error"))
+			return
+		}
+		if n, _ := res.RowsAffected(); n == 0 {
+			respond(w, http.StatusNotFound, errResp("alert not found"))
+			return
+		}
+		a, _ := fetchAlert(r.Context(), db, id)
+		respond(w, http.StatusOK, a)
+	}
+}
+
+func handleUnarchive(db *sql.DB) http.HandlerFunc {
+	return func(w http.ResponseWriter, r *http.Request) {
+		id, err := strconv.ParseInt(chi.URLParam(r, "id"), 10, 64)
+		if err != nil {
+			respond(w, http.StatusBadRequest, errResp("invalid alert id"))
+			return
+		}
+		res, err := db.ExecContext(r.Context(),
+			"UPDATE alerts SET archived_at = NULL WHERE id = ?", id)
+		if err != nil {
+			respond(w, http.StatusInternalServerError, errResp("internal error"))
+			return
+		}
+		if n, _ := res.RowsAffected(); n == 0 {
+			respond(w, http.StatusNotFound, errResp("alert not found"))
+			return
+		}
+		w.WriteHeader(http.StatusNoContent)
+	}
+}

internal/api/api_test.go

@@ -2,21 +2,26 @@ package api_test
 
 import (
 	"bytes"
+	"context"
+	"database/sql"
 	"encoding/json"
 	"fmt"
 	"io"
 	"net/http"
 	"net/http/httptest"
 	"testing"
+	"time"
 
 	"github.com/yeniklas/terdut-server/internal/api"
 	"github.com/yeniklas/terdut-server/internal/db"
 )
 
-// ts wraps httptest.Server with a pre-bootstrapped API key.
+// ts wraps httptest.Server with a pre-bootstrapped API key. db is exposed so
+// tests can age rows directly — the sweeper's inputs are wall-clock timestamps.
 type ts struct {
 	*httptest.Server
 	key string
+	db  *sql.DB
 }
 
 func newTS(t *testing.T) *ts {
@@ -44,7 +49,27 @@ func newTS(t *testing.T) *ts {
 	json.NewDecoder(resp.Body).Decode(&result)
 	key := result["api_key"].(map[string]any)["key"].(string)
 
-	return &ts{Server: srv, key: key}
+	return &ts{Server: srv, key: key, db: database}
+}
+
+// exec runs a statement against the test database.
+func (s *ts) exec(t *testing.T, query string, args ...any) {
+	t.Helper()
+	if _, err := s.db.Exec(query, args...); err != nil {
+		t.Fatalf("exec %q: %v", query, err)
+	}
+}
+
+// alertRow reads the sweeper-relevant columns of one alert straight from the DB.
+func (s *ts) alertRow(t *testing.T, fingerprint string) (status string, source *string, archivedAt *int64) {
+	t.Helper()
+	err := s.db.QueryRow(
+		"SELECT status, resolution_source, archived_at FROM alerts WHERE fingerprint = ?",
+		fingerprint).Scan(&status, &source, &archivedAt)
+	if err != nil {
+		t.Fatalf("read alert %s: %v", fingerprint, err)
+	}
+	return status, source, archivedAt
 }
 
 // req sends an authenticated request, optionally with a JSON body.
@@ -357,6 +382,250 @@ func TestStats_ByHourReturnsTwentyFourSlots(t *testing.T) {
 	}
 }
 
+// ---------------------------------------------------------------------------
+// Archive
+// ---------------------------------------------------------------------------
+
+func TestArchive_RoundTrip(t *testing.T) {
+	s := newTS(t)
+
+	postWebhook(t, s, []map[string]any{{
+		"status": "resolved", "fingerprint": "arch1",
+		"labels": map[string]string{"alertname": "Archivable"},
+		"annotations": map[string]string{},
+		"startsAt": "2026-05-20T10:00:00Z", "endsAt": "2026-05-20T11:00:00Z",
+		"generatorURL": "",
+	}})
+
+	// 1. Alert appears in default list (not archived).
+	var alerts []map[string]any
+	decode(t, s.req(t, http.MethodGet, "/api/alerts", nil), &alerts)
+	if len(alerts) != 1 {
+		t.Fatalf("expected 1 alert in default list, got %d", len(alerts))
+	}
+	id := int(alerts[0]["id"].(float64))
+
+	// 2. Archive it.
+	resp := s.req(t, http.MethodPost, fmt.Sprintf("/api/alerts/%d/archive", id), nil)
+	if resp.StatusCode != http.StatusOK {
+		t.Fatalf("archive: expected 200, got %d", resp.StatusCode)
+	}
+	var archived map[string]any
+	decode(t, resp, &archived)
+	if archived["archived_at"] == nil {
+		t.Error("expected archived_at to be set in response")
+	}
+
+	// 3. Default list excludes it.
+	decode(t, s.req(t, http.MethodGet, "/api/alerts", nil), &alerts)
+	if len(alerts) != 0 {
+		t.Errorf("expected archived alert to be hidden, got %d results", len(alerts))
+	}
+
+	// 4. archived=true shows it.
+	decode(t, s.req(t, http.MethodGet, "/api/alerts?archived=true", nil), &alerts)
+	if len(alerts) != 1 {
+		t.Fatalf("expected 1 archived alert, got %d", len(alerts))
+	}
+
+	// 5. Un-archive.
+	resp = s.req(t, http.MethodDelete, fmt.Sprintf("/api/alerts/%d/archive", id), nil)
+	if resp.StatusCode != http.StatusNoContent {
+		t.Fatalf("unarchive: expected 204, got %d", resp.StatusCode)
+	}
+	resp.Body.Close()
+
+	// 6. Back in default list.
+	decode(t, s.req(t, http.MethodGet, "/api/alerts", nil), &alerts)
+	if len(alerts) != 1 {
+		t.Errorf("expected unarchived alert to reappear, got %d results", len(alerts))
+	}
+}
+
+// ---------------------------------------------------------------------------
+// Stale-alert expiry
+// ---------------------------------------------------------------------------
+
+// noArchive is long enough that archiving never interferes with expiry tests.
+const noArchive = 365 * 24 * time.Hour
+
+// zeroTime is Alertmanager's "no end known" sentinel, which stores ends_at NULL.
+const zeroTime = "0001-01-01T00:00:00Z"
+
+// postAlert sends a single-alert webhook.
+func postAlert(t *testing.T, s *ts, fingerprint, status, startsAt, endsAt string) {
+	t.Helper()
+	postWebhook(t, s, []map[string]any{{
+		"status":       status,
+		"labels":       map[string]string{"alertname": "Stale"},
+		"annotations":  map[string]string{},
+		"startsAt":     startsAt,
+		"endsAt":       endsAt,
+		"generatorURL": "",
+		"fingerprint":  fingerprint,
+	}})
+}
+
+func sweep(t *testing.T, s *ts, staleAfter time.Duration) {
+	t.Helper()
+	api.Sweep(context.Background(), s.db, noArchive, staleAfter)
+}
+
+// A firing alert Alertmanager stopped refreshing is resolved via the
+// received_at heartbeat, even with no ends_at watermark to go on.
+func TestExpiry_StaleFiringAlert(t *testing.T) {
+	s := newTS(t)
+	postAlert(t, s, "stale1", "firing", time.Now().Add(-24*time.Hour).Format(time.RFC3339), zeroTime)
+
+	// Age the last-seen timestamp past the staleness window.
+	s.exec(t, "UPDATE alerts SET received_at = ? WHERE fingerprint = 'stale1'",
+		time.Now().Add(-10*time.Hour).Unix())
+
+	sweep(t, s, 6*time.Hour)
+
+	status, source, _ := s.alertRow(t, "stale1")
+	if status != "resolved" {
+		t.Errorf("expected status resolved, got %q", status)
+	}
+	if source == nil || *source != "expiry" {
+		t.Errorf("expected resolution_source=expiry, got %v", source)
+	}
+}
+
+// A fresh webhook whose ends_at watermark has already passed is expired without
+// waiting out the full staleness window.
+func TestExpiry_PastEndsAt(t *testing.T) {
+	s := newTS(t)
+	postAlert(t, s, "stale2", "firing",
+		time.Now().Add(-2*time.Hour).Format(time.RFC3339),
+		time.Now().Add(-30*time.Minute).Format(time.RFC3339))
+
+	sweep(t, s, 6*time.Hour) // received_at is fresh; only ends_at can trigger
+
+	status, source, _ := s.alertRow(t, "stale2")
+	if status != "resolved" {
+		t.Errorf("expected status resolved, got %q", status)
+	}
+	if source == nil || *source != "expiry" {
+		t.Errorf("expected resolution_source=expiry, got %v", source)
+	}
+}
+
+// The regression that matters most: a genuinely firing alert must survive a
+// sweep untouched.
+func TestExpiry_LeavesFreshAlertsAlone(t *testing.T) {
+	s := newTS(t)
+	postAlert(t, s, "fresh1", "firing",
+		time.Now().Add(-10*time.Minute).Format(time.RFC3339),
+		time.Now().Add(1*time.Hour).Format(time.RFC3339))
+
+	sweep(t, s, 6*time.Hour)
+
+	status, source, _ := s.alertRow(t, "fresh1")
+	if status != "firing" {
+		t.Errorf("expected fresh alert to stay firing, got %q", status)
+	}
+	if source != nil {
+		t.Errorf("expected no resolution_source, got %q", *source)
+	}
+}
+
+// An ends_at only just past must not trip expiry — that grace absorbs clock skew.
+func TestExpiry_RespectsGraceOnEndsAt(t *testing.T) {
+	s := newTS(t)
+	postAlert(t, s, "grace1", "firing",
+		time.Now().Add(-time.Hour).Format(time.RFC3339),
+		time.Now().Add(-1*time.Minute).Format(time.RFC3339))
+
+	sweep(t, s, 6*time.Hour)
+
+	if status, _, _ := s.alertRow(t, "grace1"); status != "firing" {
+		t.Errorf("expected alert within grace period to stay firing, got %q", status)
+	}
+}
+
+// ---------------------------------------------------------------------------
+// Webhook resolution bookkeeping
+// ---------------------------------------------------------------------------
+
+func TestWebhook_ResolvedSetsSource(t *testing.T) {
+	s := newTS(t)
+	start := time.Now().Add(-time.Hour).Format(time.RFC3339)
+	postAlert(t, s, "src1", "firing", start, zeroTime)
+
+	if _, source, _ := s.alertRow(t, "src1"); source != nil {
+		t.Errorf("expected firing alert to have no resolution_source, got %q", *source)
+	}
+
+	postAlert(t, s, "src1", "resolved", start, time.Now().Format(time.RFC3339))
+
+	status, source, _ := s.alertRow(t, "src1")
+	if status != "resolved" {
+		t.Errorf("expected status resolved, got %q", status)
+	}
+	if source == nil || *source != "alertmanager" {
+		t.Errorf("expected resolution_source=alertmanager, got %v", source)
+	}
+}
+
+// A re-fire under the same fingerprint must leave the archive and clear the
+// stale expiry marker, otherwise the alert stays invisible in the default list.
+func TestWebhook_RefireUnarchivesAndClearsSource(t *testing.T) {
+	s := newTS(t)
+	postAlert(t, s, "refire1", "firing", time.Now().Add(-24*time.Hour).Format(time.RFC3339), zeroTime)
+
+	// Expire it, then archive it.
+	s.exec(t, "UPDATE alerts SET received_at = ? WHERE fingerprint = 'refire1'",
+		time.Now().Add(-10*time.Hour).Unix())
+	sweep(t, s, 6*time.Hour)
+	s.exec(t, "UPDATE alerts SET archived_at = unixepoch() WHERE fingerprint = 'refire1'")
+
+	var alerts []map[string]any
+	decode(t, s.req(t, http.MethodGet, "/api/alerts", nil), &alerts)
+	if len(alerts) != 0 {
+		t.Fatalf("expected archived alert to be hidden, got %d", len(alerts))
+	}
+
+	// Fires again: a new alert instance, so a newer startsAt.
+	postAlert(t, s, "refire1", "firing", time.Now().Format(time.RFC3339), zeroTime)
+
+	status, source, archivedAt := s.alertRow(t, "refire1")
+	if status != "firing" {
+		t.Errorf("expected status firing after re-fire, got %q", status)
+	}
+	if source != nil {
+		t.Errorf("expected resolution_source cleared on re-fire, got %q", *source)
+	}
+	if archivedAt != nil {
+		t.Errorf("expected archived_at cleared on re-fire, got %d", *archivedAt)
+	}
+
+	decode(t, s.req(t, http.MethodGet, "/api/alerts", nil), &alerts)
+	if len(alerts) != 1 {
+		t.Errorf("expected re-fired alert back in default list, got %d", len(alerts))
+	}
+}
+
+// Alertmanager retries failed notifications, so a firing payload for an
+// already-resolved instance can arrive late. It must not resurrect the alert.
+func TestWebhook_IgnoresOutOfOrderRetry(t *testing.T) {
+	s := newTS(t)
+	start := time.Now().Add(-time.Hour).Format(time.RFC3339)
+	end := time.Now().Format(time.RFC3339)
+
+	postAlert(t, s, "ooo1", "firing", start, zeroTime)
+	postAlert(t, s, "ooo1", "resolved", start, end)
+	postAlert(t, s, "ooo1", "firing", start, zeroTime) // stale retry, same instance
+
+	status, source, _ := s.alertRow(t, "ooo1")
+	if status != "resolved" {
+		t.Errorf("expected alert to stay resolved after stale retry, got %q", status)
+	}
+	if source == nil || *source != "alertmanager" {
+		t.Errorf("expected resolution_source=alertmanager, got %v", source)
+	}
+}
+
 func TestStats_ByDayReturnsSevenSlots(t *testing.T) {
 	s := newTS(t)
 	resp := s.req(t, http.MethodGet, "/api/stats/alerts/by-day", nil)

internal/api/archiver.go

@@ -0,0 +1,91 @@
+package api
+
+import (
+	"context"
+	"database/sql"
+	"log"
+	"time"
+)
+
+const (
+	// sweepInterval is how often the background sweeper runs.
+	sweepInterval = 15 * time.Minute
+
+	// expiryGrace absorbs clock skew and notification latency before an alert
+	// whose ends_at watermark has passed is treated as stale.
+	expiryGrace = 5 * time.Minute
+)
+
+// StartArchiver runs the alert sweeper until ctx is cancelled, starting with an
+// immediate pass so a restart reconciles state right away.
+func StartArchiver(ctx context.Context, db *sql.DB, archiveAfter, staleAfter time.Duration) {
+	ticker := time.NewTicker(sweepInterval)
+	defer ticker.Stop()
+
+	Sweep(ctx, db, archiveAfter, staleAfter)
+	for {
+		select {
+		case <-ticker.C:
+			Sweep(ctx, db, archiveAfter, staleAfter)
+		case <-ctx.Done():
+			return
+		}
+	}
+}
+
+// Sweep runs a single pass: expire stale firing alerts, then archive resolved
+// ones. Expiry runs first so an alert can expire and be archived in one pass.
+// Exported so tests can drive a pass without waiting on the ticker.
+func Sweep(ctx context.Context, db *sql.DB, archiveAfter, staleAfter time.Duration) {
+	expireStale(ctx, db, staleAfter)
+	archiveResolved(ctx, db, archiveAfter)
+}
+
+// expireStale resolves firing alerts that Alertmanager has stopped refreshing.
+//
+// A resolved webhook is otherwise the only way out of the firing state, so a
+// notification that is dropped, silenced, or lost to a restart would pin the
+// alert as firing forever. Two independent signals mark an alert stale:
+//
+//   - ends_at, the "valid until" watermark Alertmanager sets on outgoing firing
+//     notifications, has passed (plus expiryGrace for clock skew). Absent on
+//     rows whose payload carried no ends_at, hence the second signal.
+//   - received_at is older than staleAfter. Alertmanager re-sends firing
+//     notifications every repeat_interval, making received_at a liveness
+//     heartbeat — provided staleAfter exceeds that interval.
+func expireStale(ctx context.Context, db *sql.DB, staleAfter time.Duration) {
+	now := time.Now()
+	res, err := db.ExecContext(ctx, `
+		UPDATE alerts
+		   SET status            = 'resolved',
+		       resolution_source = ?,
+		       ends_at           = COALESCE(ends_at, unixepoch())
+		 WHERE status = 'firing'
+		   AND archived_at IS NULL
+		   AND ((ends_at IS NOT NULL AND ends_at < ?) OR received_at < ?)`,
+		resolutionExpiry, now.Add(-expiryGrace).Unix(), now.Add(-staleAfter).Unix())
+	if err != nil {
+		log.Printf("sweeper: expire stale: %v", err)
+		return
+	}
+	if n, _ := res.RowsAffected(); n > 0 {
+		log.Printf("sweeper: expired %d stale firing alert(s)", n)
+	}
+}
+
+// archiveResolved hides resolved alerts that have been settled for archiveAfter.
+func archiveResolved(ctx context.Context, db *sql.DB, archiveAfter time.Duration) {
+	cutoff := time.Now().Add(-archiveAfter).Unix()
+	res, err := db.ExecContext(ctx,
+		`UPDATE alerts SET archived_at = unixepoch()
+		 WHERE status = 'resolved'
+		   AND archived_at IS NULL
+		   AND COALESCE(ends_at, received_at) < ?`, cutoff)
+	if err != nil {
+		log.Printf("archiver: %v", err)
+		return
+	}
+	if n, _ := res.RowsAffected(); n > 0 {
+		log.Printf("archiver: archived %d resolved alert(s)", n)
+	}
+}

internal/api/router.go

@@ -35,6 +35,8 @@ func NewRouter(db *sql.DB) http.Handler {
 		r.Get("/api/alerts/{id}", handleGetAlert(db))
 		r.Post("/api/alerts/{id}/acknowledge", handleAcknowledge(db))
 		r.Delete("/api/alerts/{id}/acknowledge", handleUnacknowledge(db))
+		r.Post("/api/alerts/{id}/archive", handleArchive(db))
+		r.Delete("/api/alerts/{id}/archive", handleUnarchive(db))
 		r.Get("/api/alerts/{id}/comments", handleListComments(db))
 		r.Post("/api/alerts/{id}/comments", handleCreateComment(db))
 		r.Delete("/api/alerts/{id}/comments/{commentID}", handleDeleteComment(db))

internal/api/stats.go

@@ -160,8 +160,9 @@ func handleStatsByDay(db *sql.DB) http.HandlerFunc {
 }
 
 // statsFilter builds a WHERE clause and args from optional ?from and ?to query params.
+// Archived alerts are always excluded, matching the default GET /api/alerts view.
 func statsFilter(q url.Values) (where string, args []any) {
-	clauses := []string{}
+	clauses := []string{"archived_at IS NULL"}
 	if from := q.Get("from"); from != "" {
 		if t, err := time.Parse("2006-01-02", from); err == nil {
 			clauses = append(clauses, "received_at >= ?")
@@ -174,8 +175,5 @@ func statsFilter(q url.Values) (where string, args []any) {
 			args = append(args, t.UTC().AddDate(0, 0, 1).Unix())
 		}
 	}
-	if len(clauses) == 0 {
-		return "1=1", args
-	}
 	return strings.Join(clauses, " AND "), args
 }

internal/config/config.go

@@ -1,10 +1,19 @@
 package config
 
-import "os"
+import (
+	"os"
+	"time"
+)
 
 type Config struct {
-	Addr   string
-	DBPath string
+	Addr         string
+	DBPath       string
+	ArchiveAfter time.Duration
+
+	// StaleAfter is how long a firing alert may go without a refreshing webhook
+	// before the sweeper treats it as resolved. It must exceed Alertmanager's
+	// repeat_interval (default 4h), which is what refreshes the alert.
+	StaleAfter time.Duration
 }
 
 func Load() Config {
@@ -16,5 +25,17 @@ func Load() Config {
 	if dbPath == "" {
 		dbPath = "terdut.db"
 	}
-	return Config{Addr: addr, DBPath: dbPath}
+	archiveAfter := 7 * 24 * time.Hour
+	if s := os.Getenv("TERDUT_ARCHIVE_AFTER"); s != "" {
+		if d, err := time.ParseDuration(s); err == nil {
+			archiveAfter = d
+		}
+	}
+	staleAfter := 6 * time.Hour
+	if s := os.Getenv("TERDUT_STALE_AFTER"); s != "" {
+		if d, err := time.ParseDuration(s); err == nil {
+			staleAfter = d
+		}
+	}
+	return Config{Addr: addr, DBPath: dbPath, ArchiveAfter: archiveAfter, StaleAfter: staleAfter}
 }

internal/db/migrations/006_alert_archive.sql

@@ -0,0 +1,2 @@
+ALTER TABLE alerts ADD COLUMN archived_at INTEGER;
+CREATE INDEX alerts_archived_at_idx ON alerts(archived_at);

internal/db/migrations/007_alert_resolution_source.sql

@@ -0,0 +1,4 @@
+-- Records why an alert left the firing state: 'alertmanager' when a resolved
+-- webhook set it, 'expiry' when the sweeper inferred it from staleness.
+-- NULL for firing alerts and for rows that predate this migration.
+ALTER TABLE alerts ADD COLUMN resolution_source TEXT;

internal/models/alert.go

@@ -18,4 +18,11 @@ type Alert struct {
 	AcknowledgedByID   *int64     `json:"acknowledged_by_id,omitempty"`
 	AcknowledgedByUser *string    `json:"acknowledged_by,omitempty"`
 	AcknowledgedAt     *time.Time `json:"acknowledged_at,omitempty"`
+
+	// ResolutionSource records why a resolved alert left the firing state:
+	// "alertmanager" for a real resolved webhook, "expiry" when the sweeper
+	// inferred it after the alert stopped being refreshed.
+	ResolutionSource *string `json:"resolution_source,omitempty"`
+
+	ArchivedAt *time.Time `json:"archived_at,omitempty"`
 }