Set the chart's placeholder version to 0.25.0

Cosmetic: make helm-package sets the published version and appVersion
from the tag, so these two fields decide nothing (see the comment
above them). Kept in step anyway, same as e8d45f9 and 3ee8583, so a
tree heading for v0.25.0 doesn't say 0.24.0.

README.md

@@ -48,11 +48,12 @@ curl -H "Authorization: Bearer $KEY" http://localhost:8080/api/users
 The server serves a web UI at `/`: the incident queue, each incident's alerts
 and timeline with every action (acknowledge, assign, snooze, note, resolve,
 archive), who is on call, the alert feed, and an *Account* tab for your own
-password and the ntfy topic your pages go to. It is built for a phone first. On a phone it has a bottom tab bar and a sticky action
+password and the ntfy topic your pages go to. It is built for a phone first. On a phone
-bar, it follows the system's dark mode, and it can be added to the home screen.
+it navigates through a hamburger menu and has a sticky action bar, it follows the
-From 900px wide it switches to a sidebar with the queue and the incident side by
+system's dark mode, and it can be added to the home screen. From 900px wide it switches
-side. Statistics remain in
+to a sidebar with the queue and the incident side by side. The Stats page shows
-[terdut-tui](https://github.com/yeniklas/terdut-tui) for now.
+incident counts, MTTA and MTTR, and alert frequency by name, hour and day over a
+chosen range.
 
 You sign in with a username and password. Users have no password until one is
 set, and a user without one can only use API keys:
@@ -512,19 +513,27 @@ nothing unless something downstream notices it stop. That is what
 `TERDUT_DEADMAN_MATCHERS` defaults to.
 
 **Switches belong to a team**, which decides which of its own alerts are
-heartbeats and how long a silence has to last. An owner sets them through
+heartbeats and how long a silence has to last. Each **switch** is a row of its
-`PUT /api/teams/{teamID}/deadman`; a missed heartbeat opens an incident in the
+own — a name, one matcher, a timeout and a severity — so switches in one team
-team whose integration received it.
+can have different deadlines. An owner adds and removes them on **Team →
+Switches**, which lists each with a status (**healthy**, **dead**, or
+**dormant** until its first heartbeat), when it was last heard from, and when it
+last opened an incident; a matcher that several clusters satisfy is broken down
+per cluster. The API is `POST`/`DELETE /api/teams/{teamID}/deadman/switches`. A
+missed heartbeat opens an incident in the team whose integration received it.
+Removing a switch stops the watching; an incident it already opened stays open
+until somebody resolves it.
 
-The environment variables are the starting point, not the setting: at startup
+The environment variables are the starting point, not the setting: the **first**
-every team **without** a configuration of its own is given one from them, and an
+time the server starts, every team is given a switch per default matcher from
-owner's later edit is never overwritten by a redeploy. A team created after
+them, once. After that a team's switches are its own — an owner's edit or
-that starts watching nothing until its owner says otherwise — inheriting an
+deletion is never put back by a redeploy. A team created later starts watching
-install-wide heartbeat would page a new team about a source it has never heard
+nothing until its owner says otherwise — inheriting an install-wide heartbeat
-of.
+would page a new team about a source it has never heard of.
 
 A matcher is a set of exact label conditions, one of which must be the
-`alertname`, in the same format the environment variable uses:
+`alertname`, in the format the environment variable uses (one matcher per switch; the
+variable takes several, separated by `;`):
 
 ```
 alertname=Watchdog,cluster=prod; alertname=EdgeHeartbeat
@@ -709,16 +718,18 @@ administrator who is not in the team gets the same `404` as anybody else.
 | `GET` | `/api/teams/{teamID}/members` | member | Who is in the team |
 | `POST` | `/api/teams/{teamID}/members` | **owner** | Add a member, or change their role `{"user_id","role"}` |
 | `DELETE` | `/api/teams/{teamID}/members/{userID}` | **owner** | Remove a member. `409` for the last owner |
-| `GET` | `/api/teams/{teamID}/integrations` | member | List integrations. Never returns keys |
+| `GET` | `/api/teams/{teamID}/integrations` | member | List integrations. Never returns keys. Each carries `status` (`active` if its key posted within 24h, `quiet` if it has but not lately, `never`), `last_used_at` (last webhook, usable or not), `last_alert_at` (when an alert last arrived on it) and `alerts_24h` (distinct alerts it refreshed in the last day). Alerts delivered before the source was recorded (migration 010) have none, so the last two fill in as Alertmanager re-sends them |
+| `PATCH` | `/api/teams/{teamID}/integrations/{integrationID}` | **owner** | Rename `{"name"}`. The key does not change |
 | `POST` | `/api/teams/{teamID}/integrations` | **owner** | Mint an integration `{"name","kind"}` — key and URL shown once |
-| `DELETE` | `/api/teams/{teamID}/integrations/{integrationID}` | **owner** | Revoke an integration |
+| `DELETE` | `/api/teams/{teamID}/integrations/{integrationID}` | **owner** | Revoke an integration. Alerts it delivered stay, unattributed |
 | `GET` | `/api/teams/{teamID}/invites` | **owner** | The team's invite links, with their uses and expiry. Never the tokens |
 | `POST` | `/api/teams/{teamID}/invites` | **owner** | Mint one `{"role","max_uses"}` — the full URL is returned once |
 | `DELETE` | `/api/teams/{teamID}/invites/{inviteID}` | **owner** | Revoke a link before it expires |
 | `GET` | `/api/teams/{teamID}/escalation` | member | The team's [escalation ladder](#escalation) `{repeat_count, fallback_topic, levels[]}`. Empty levels means the team has none |
 | `PUT` | `/api/teams/{teamID}/escalation` | **owner** | Replace it wholesale. `400` for a level with no targets or no timeout — a rung that pages nobody is a silence with a number on it |
-| `GET` | `/api/teams/{teamID}/deadman` | member | The team's [dead man's switch](#dead-mans-switch) configuration `{matchers, timeout_seconds, severity}` |
+| `GET` | `/api/teams/{teamID}/deadman/switches` | member | The team's [dead man's switches](#dead-mans-switch), each `{id, name, matcher, timeout_seconds, severity, status, last_heartbeat_at, last_triggered_at, open_incident_id, sources[]}`. `status` is `healthy`, `dead` or `dormant`; `sources` has one entry per heartbeat fingerprint. Empty when the team watches nothing |
-| `PUT` | `/api/teams/{teamID}/deadman` | **owner** | Replace it. `400` when no matcher names an `alertname`, because a switch that silently watches nothing is the failure this feature exists to prevent |
+| `POST` | `/api/teams/{teamID}/deadman/switches` | **owner** | Add one: `{name?, matcher, timeout_seconds, severity?}`. `400` when the matcher names no `alertname` or holds several, or the timeout is not positive — a switch that silently watches nothing is the failure this feature exists to prevent |
+| `DELETE` | `/api/teams/{teamID}/deadman/switches/{switchID}` | **owner** | Stop watching. An incident it opened stays open. `404` for a switch of another team |
 
 ### Notifications
 
@@ -963,8 +974,8 @@ What changes, and will need attention:
 
 **Dead man's switches moved too.** `TERDUT_DEADMAN_MATCHERS`, `_TIMEOUT` and
 `_SEVERITY` are no longer the setting; they are the default each existing team
-is seeded with at startup, after which an owner edits them per team through
+is seeded with at startup, after which an owner manages them per team through
-`PUT /api/teams/{teamID}/deadman` and a redeploy never overwrites that.
+`/api/teams/{teamID}/deadman/switches` and a redeploy never overwrites that.
 
 Nothing else about an incident changes, and incidents never move between teams:
 an alert belongs to whichever team's key it arrived on.

charts/terdut-server/Chart.yaml

@@ -15,5 +15,5 @@ type: application
 # appVersion and image.tag in values.yaml no longer agree, and that is not an oversight:
 # image.tag stays "latest", which is what a local install actually pulls. appVersion is
 # metadata and drives nothing.
-version: 0.19.0
+version: 0.25.0
-appVersion: "v0.19.0"
+appVersion: "v0.25.0"

internal/api/alertmanager.go

@@ -78,7 +78,7 @@ type ingested struct {
 // post, and which team the alerts belong to.
 func handleIntegrationWebhook(db *sql.DB, notify NotifyConfig) http.HandlerFunc {
 	return func(w http.ResponseWriter, r *http.Request) {
-		teamID, err := teamIDForKey(r.Context(), db, chi.URLParam(r, "key"))
+		src, err := sourceForKey(r.Context(), db, chi.URLParam(r, "key"))
 		if err != nil {
 			if errors.Is(err, errUnknownIntegration) {
 				// 401 and not 404: the path is real, the key is not, and a
@@ -90,11 +90,12 @@ func handleIntegrationWebhook(db *sql.DB, notify NotifyConfig) http.HandlerFunc
 			respond(w, http.StatusInternalServerError, errResp("internal error"))
 			return
 		}
-		receiveWebhook(w, r, db, notify, teamID)
+		receiveWebhook(w, r, db, notify, src)
 	}
 }
 
-func receiveWebhook(w http.ResponseWriter, r *http.Request, db *sql.DB, notify NotifyConfig, teamID int64) {
+func receiveWebhook(w http.ResponseWriter, r *http.Request, db *sql.DB, notify NotifyConfig, src alertSource) {
+	teamID := src.teamID
 	var payload amPayload
 	if err := decodeJSON(r, &payload); err != nil {
 		respond(w, http.StatusBadRequest, errResp("invalid payload"))
@@ -104,7 +105,7 @@ func receiveWebhook(w http.ResponseWriter, r *http.Request, db *sql.DB, notify N
 	// Alertmanager retries anything that is not 2xx, and a retry of a payload
 	// we failed to store is more useful than an error it cannot act on — so
 	// failures are logged, not surfaced.
-	if err := ingest(r.Context(), db, notify, teamID, payload); err != nil {
+	if err := ingest(r.Context(), db, notify, src, payload); err != nil {
 		log.Printf("webhook ingest (team %d, group %q): %v", teamID, payload.GroupKey, err)
 	}
 
@@ -114,7 +115,8 @@ func receiveWebhook(w http.ResponseWriter, r *http.Request, db *sql.DB, notify N
 // ingest stores a payload's alerts and reconciles the incident for its group.
 // The whole payload is one transaction: an incident that opened but whose alerts
 // failed to link would be a work item nobody could act on.
-func ingest(ctx context.Context, db *sql.DB, notify NotifyConfig, teamID int64, payload amPayload) error {
+func ingest(ctx context.Context, db *sql.DB, notify NotifyConfig, src alertSource, payload amPayload) error {
+	teamID := src.teamID
 	tx, err := db.BeginTx(ctx, nil)
 	if err != nil {
 		return err
@@ -124,12 +126,12 @@ func ingest(ctx context.Context, db *sql.DB, notify NotifyConfig, teamID int64,
 	// Which arriving alerts are heartbeats is the team's own answer, read
 	// inside the transaction so an owner editing it mid-payload cannot split
 	// one webhook across two interpretations.
-	deadman, err := deadmanConfigForTeam(ctx, tx, teamID)
+	deadman, err := deadmanSetForTeam(ctx, tx, teamID)
 	if err != nil {
 		return err
 	}
 
-	accepted, err := upsertAlerts(ctx, tx, deadman, teamID, payload.Alerts)
+	accepted, err := upsertAlerts(ctx, tx, deadman, src, payload.Alerts)
 	if err != nil {
 		return err
 	}
@@ -186,7 +188,8 @@ func ingest(ctx context.Context, db *sql.DB, notify NotifyConfig, teamID int64,
 
 // upsertAlerts stores each alert of a payload and reports what changed. Payloads
 // the ordering guard rejected are left out entirely.
-func upsertAlerts(ctx context.Context, tx *sql.Tx, deadman DeadmanConfig, teamID int64, alerts []amAlert) ([]ingested, error) {
+func upsertAlerts(ctx context.Context, tx *sql.Tx, deadman deadmanSet, src alertSource, alerts []amAlert) ([]ingested, error) {
+	teamID := src.teamID
 	now := time.Now().Unix()
 	accepted := make([]ingested, 0, len(alerts))
 
@@ -242,8 +245,8 @@ func upsertAlerts(ctx context.Context, tx *sql.Tx, deadman DeadmanConfig, teamID
 		if _, err := tx.ExecContext(ctx, `
 			INSERT INTO alerts
 				(team_id, fingerprint, name, status, labels, annotations, starts_at, ends_at,
-				 generator_url, received_at, resolution_source)
+				 generator_url, received_at, resolution_source, integration_id)
-			VALUES ($1, $2, $3, $4, $5::jsonb, $6::jsonb, $7, $8, $9, $10, $11)
+			VALUES ($1, $2, $3, $4, $5::jsonb, $6::jsonb, $7, $8, $9, $10, $11, $12)
 			ON CONFLICT (team_id, fingerprint) DO UPDATE SET
 				status            = excluded.status,
 				labels            = excluded.labels,
@@ -257,6 +260,8 @@ func upsertAlerts(ctx context.Context, tx *sql.Tx, deadman DeadmanConfig, teamID
 				-- is a breaking API change — see models.Alert.ReceivedAt.
 				received_at       = excluded.received_at,
 				resolution_source = excluded.resolution_source,
+				-- Last sender wins; see migration 010.
+				integration_id    = excluded.integration_id,
 				-- A re-fire makes the alert current again, so it leaves the archive.
 				archived_at       = CASE WHEN excluded.status = 'firing'
 				                        THEN NULL ELSE alerts.archived_at END
@@ -267,7 +272,7 @@ func upsertAlerts(ctx context.Context, tx *sql.Tx, deadman DeadmanConfig, teamID
 			teamID, a.Fingerprint, name, a.Status,
 			string(labelsJSON), string(annotationsJSON),
 			a.StartsAt.Unix(), endsAtUnix,
-			a.GeneratorURL, now, resolutionSource,
+			a.GeneratorURL, now, resolutionSource, src.integrationID,
 		); err != nil {
 			return nil, err
 		}
@@ -384,10 +389,10 @@ func openIncident(ctx context.Context, q querier, notify NotifyConfig, teamID in
 
 	var id int64
 	err = q.QueryRowContext(ctx, `
-		INSERT INTO incidents (team_id, group_key, title, group_labels, status, severity, triggered_at, assigned_to)
+		INSERT INTO incidents (team_id, group_key, title, group_labels, signature, status, severity, triggered_at, assigned_to)
-		VALUES ($1, $2, $3, $4::jsonb, 'triggered', $5, $6, $7)
+		VALUES ($1, $2, $3, $4::jsonb, $5, 'triggered', $6, $7, $8)
 		RETURNING id`,
-		teamID, groupKey, title, string(labelsJSON), severity,
+		teamID, groupKey, title, string(labelsJSON), incidentSignature(groupLabels, title), severity,
 		time.Now().Unix(), onCall).Scan(&id)
 	if err != nil {
 		return 0, err

internal/api/api_test.go

@@ -88,27 +88,25 @@ func newDeadmanTS(t *testing.T, deadman api.DeadmanConfig, notify ...api.NotifyC
 	return s
 }
 
-// setTeamDeadman configures the default team's switches over the API, rendering
+// setTeamDeadman gives the default team one switch per configured matcher, over
-// the matchers back into the string form the endpoint takes.
+// the API, the way an owner would add them.
 func setTeamDeadman(t *testing.T, s *ts, cfg api.DeadmanConfig) {
 	t.Helper()
-	matchers := make([]string, 0, len(cfg.Matchers))
 	for _, m := range cfg.Matchers {
 		parts := []string{"alertname=" + m.Name}
 		for k, v := range m.Labels {
 			parts = append(parts, k+"="+v)
 		}
 		sort.Strings(parts[1:])
-		matchers = append(matchers, strings.Join(parts, ","))
+		resp := s.req(t, http.MethodPost, "/api/teams/"+defaultTeam+"/deadman/switches", map[string]any{
-	}
+			"matcher":         strings.Join(parts, ","),
-	resp := s.req(t, http.MethodPut, "/api/teams/"+defaultTeam+"/deadman", map[string]any{
-		"matchers":        strings.Join(matchers, "; "),
 			"timeout_seconds": int64(cfg.Timeout.Seconds()),
 			"severity":        cfg.Severity,
 		})
-	defer resp.Body.Close()
+		resp.Body.Close()
-	if resp.StatusCode != http.StatusOK {
+		if resp.StatusCode != http.StatusCreated {
-		t.Fatalf("configure the team's dead man's switches: %d", resp.StatusCode)
+			t.Fatalf("add a dead man's switch: %d", resp.StatusCode)
+		}
 	}
 }
 

internal/api/deadman.go

@@ -4,6 +4,8 @@ import (
 	"context"
 	"database/sql"
 	"encoding/json"
+	"errors"
+	"fmt"
 	"log"
 	"sort"
 	"strings"
@@ -39,6 +41,17 @@ func (m DeadmanMatcher) String() string {
 	return m.Name + " (" + strings.Join(parts, ", ") + ")"
 }
 
+// config renders the matcher in the form parseDeadmanMatcher reads, which is
+// what a switch row stores: `alertname=Watchdog,cluster=prod`.
+func (m DeadmanMatcher) config() string {
+	parts := make([]string, 0, len(m.Labels))
+	for k, v := range m.Labels {
+		parts = append(parts, k+"="+v)
+	}
+	sort.Strings(parts)
+	return strings.Join(append([]string{"alertname=" + m.Name}, parts...), ",")
+}
+
 // matches reports whether an alert's labels satisfy every condition.
 func (m DeadmanMatcher) matches(labels map[string]string) bool {
 	if labels["alertname"] != m.Name {
@@ -52,12 +65,10 @@ func (m DeadmanMatcher) matches(labels map[string]string) bool {
 	return true
 }
 
-// DeadmanConfig inverts the handling of the alerts it matches: receiving one
+// DeadmanConfig is the server-wide default a team's switches are seeded from:
-// opens nothing, and the absence of one opens an incident.
+// the environment's matchers, timeout and severity. Switches themselves are rows
-//
+// of a team's own — see DeadmanSwitch — and this is only how a fresh install
-// The unit of monitoring is the fingerprint, not the matcher — two clusters
+// starts out.
-// sending the same heartbeat alertname are two independent switches, so one
-// healthy cluster cannot mask a dead one.
 type DeadmanConfig struct {
 	Matchers []DeadmanMatcher
 
@@ -76,41 +87,81 @@ type DeadmanConfig struct {
 // enabled reports whether there is anything to watch.
 func (c DeadmanConfig) enabled() bool { return c.Timeout > 0 && len(c.Matchers) > 0 }
 
-// match returns the first matcher an alert satisfies.
+// DeadmanSwitch inverts the handling of the alerts it matches: receiving one
-func (c DeadmanConfig) match(labels map[string]string) (DeadmanMatcher, bool) {
+// opens nothing, and the absence of one opens an incident.
-	if !c.enabled() {
+//
-		return DeadmanMatcher{}, false
+// The unit of monitoring is the fingerprint, not the switch — two clusters
-	}
+// sending the same heartbeat alertname are two independent heartbeats under one
-	for _, m := range c.Matchers {
+// switch, so one healthy cluster cannot mask a dead one.
-		if m.matches(labels) {
+type DeadmanSwitch struct {
-			return m, true
+	ID      int64
-		}
+	Name    string
-	}
+	Matcher DeadmanMatcher
-	return DeadmanMatcher{}, false
+
+	// Timeout is how long a heartbeat may go unheard before it is declared dead.
+	Timeout time.Duration
+
+	// Severity is what the incident opens at.
+	Severity string
 }
 
-// isDeadman is match without the matcher, for the ingest path.
+// deadmanSet is one team's switches.
-func (c DeadmanConfig) isDeadman(labels map[string]string) bool {
+type deadmanSet []DeadmanSwitch
-	_, ok := c.match(labels)
+
+// match returns the first switch an alert satisfies.
+func (d deadmanSet) match(labels map[string]string) (DeadmanSwitch, bool) {
+	for _, sw := range d {
+		if sw.Matcher.matches(labels) {
+			return sw, true
+		}
+	}
+	return DeadmanSwitch{}, false
+}
+
+// isDeadman is match without the switch, for the ingest path.
+func (d deadmanSet) isDeadman(labels map[string]string) bool {
+	_, ok := d.match(labels)
 	return ok
 }
 
 // names lists the distinct alertnames worth loading from the database.
-func (c DeadmanConfig) names() []string {
+func (d deadmanSet) names() []string {
 	seen := map[string]bool{}
-	out := make([]string, 0, len(c.Matchers))
+	out := make([]string, 0, len(d))
-	for _, m := range c.Matchers {
+	for _, sw := range d {
-		if !seen[m.Name] {
+		if !seen[sw.Matcher.Name] {
-			seen[m.Name] = true
+			seen[sw.Matcher.Name] = true
-			out = append(out, m.Name)
+			out = append(out, sw.Matcher.Name)
 		}
 	}
 	return out
 }
 
+// parseDeadmanMatcher reads one matcher from its configured form: "," separates
+// the conditions and "=" is exact label equality — `alertname=Watchdog,cluster=prod`.
+// The error says what is wrong with it, in words a form can show.
+func parseDeadmanMatcher(entry string) (DeadmanMatcher, error) {
+	m := DeadmanMatcher{Labels: map[string]string{}}
+	for _, cond := range strings.Split(strings.TrimSpace(entry), ",") {
+		k, v, ok := strings.Cut(cond, "=")
+		k, v = strings.TrimSpace(k), strings.TrimSpace(v)
+		if !ok || k == "" || v == "" {
+			return DeadmanMatcher{}, fmt.Errorf("%q is not label=value", strings.TrimSpace(cond))
+		}
+		if k == "alertname" {
+			m.Name = v
+			continue
+		}
+		m.Labels[k] = v
+	}
+	if m.Name == "" {
+		return DeadmanMatcher{}, errors.New("no alertname condition")
+	}
+	return m, nil
+}
+
 // ParseDeadmanConfig reads the matcher list from its configured form:
-// ";" separates matchers, "," separates the conditions within one, and "=" is
+// ";" separates matchers, and each is parsed as parseDeadmanMatcher does.
-// exact label equality — `alertname=Watchdog,cluster=prod; alertname=Heartbeat`.
 //
 // A malformed or alertname-less entry is dropped rather than fatal, following
 // config.duration's rule that one bad tuning knob should not take the server
@@ -125,28 +176,9 @@ func ParseDeadmanConfig(matchers string, timeout time.Duration, severity string)
 		if entry == "" {
 			continue
 		}
-
+		m, err := parseDeadmanMatcher(entry)
-		m := DeadmanMatcher{Labels: map[string]string{}}
+		if err != nil {
-		malformed := false
+			log.Printf("deadman: ignoring matcher %q: %v", entry, err)
-		for _, cond := range strings.Split(entry, ",") {
-			k, v, ok := strings.Cut(cond, "=")
-			k, v = strings.TrimSpace(k), strings.TrimSpace(v)
-			if !ok || k == "" || v == "" {
-				log.Printf("deadman: ignoring matcher %q: %q is not label=value", entry, strings.TrimSpace(cond))
-				malformed = true
-				break
-			}
-			if k == "alertname" {
-				m.Name = v
-				continue
-			}
-			m.Labels[k] = v
-		}
-		if malformed {
-			continue
-		}
-		if m.Name == "" {
-			log.Printf("deadman: ignoring matcher %q: no alertname condition", entry)
 			continue
 		}
 		cfg.Matchers = append(cfg.Matchers, m)
@@ -162,23 +194,35 @@ func ParseDeadmanConfig(matchers string, timeout time.Duration, severity string)
 		for _, m := range cfg.Matchers {
 			rendered = append(rendered, m.String())
 		}
-		log.Printf("deadman: watching %s, timeout %s, severity %s",
+		log.Printf("deadman: default for new teams: %s, timeout %s, severity %s",
 			strings.Join(rendered, "; "), timeout, severity)
 	}
 	return cfg
 }
 
-// deadmanAlert is one switch: the alert row carrying its last heartbeat.
+// deadmanAlert is one heartbeat: the alert row carrying its last sighting, and
+// the switch that claimed it.
 type deadmanAlert struct {
 	id          int64
 	teamID      int64
 	fingerprint string
 	labels      map[string]string
-	matcher     DeadmanMatcher
+	sw          DeadmanSwitch
 	resolved    bool
 	receivedAt  int64
 }
 
+// dead is the one rule for a silent heartbeat, shared by the sweeper that pages
+// on it and the status the Switches page shows, so the page cannot disagree
+// with the pager.
+//
+// An explicit resolved from Alertmanager is a stronger death signal than mere
+// absence: the sender is telling us the heartbeat stopped, so there is nothing
+// left to wait out.
+func (a deadmanAlert) dead(now time.Time) bool {
+	return a.resolved || a.receivedAt < now.Add(-a.sw.Timeout).Unix()
+}
+
 // groupKey is the switch's identity as an incident. Per fingerprint, so each
 // source is tracked on its own.
 func (a deadmanAlert) groupKey() string { return deadmanGroupPrefix + a.fingerprint }
@@ -189,13 +233,13 @@ func (a deadmanAlert) groupKey() string { return deadmanGroupPrefix + a.fingerpr
 // It returns the ids of the alerts it owns, because the generic staleness
 // expiry must leave them alone — staleAfter and ends_at would otherwise resolve
 // a heartbeat long before its own, much tighter, timeout ever fired.
-// Each team is swept against its own configuration: its own matchers, its own
+// Each team is swept against its own switches, each with its own matcher,
-// timeout, its own severity. A team watching nothing is skipped entirely, which
+// timeout and severity. A team watching nothing is skipped entirely, which is
-// is most of them.
+// most of them.
 func sweepDeadman(ctx context.Context, db *sql.DB, notify NotifyConfig) map[int64]bool {
 	owned := map[int64]bool{}
 
-	configs, err := deadmanConfigs(ctx, db)
+	configs, err := deadmanSets(ctx, db)
 	if err != nil {
 		log.Printf("deadman: load configs: %v", err)
 		return owned
@@ -203,39 +247,35 @@ func sweepDeadman(ctx context.Context, db *sql.DB, notify NotifyConfig) map[int6
 
 	now := time.Now()
 	for teamID, cfg := range configs {
-		switches, err := deadmanAlerts(ctx, db, teamID, cfg)
+		heartbeats, err := deadmanAlerts(ctx, db, teamID, cfg)
 		if err != nil {
-			log.Printf("deadman: load switches for team %d: %v", teamID, err)
+			log.Printf("deadman: load heartbeats for team %d: %v", teamID, err)
 			continue
 		}
-		cutoff := now.Add(-cfg.Timeout).Unix()
 
-		for _, sw := range switches {
+		for _, hb := range heartbeats {
-			owned[sw.id] = true
+			owned[hb.id] = true
 
-			// An explicit resolved from Alertmanager is a stronger death signal
+			if hb.dead(now) {
-			// than mere absence: the sender is telling us the heartbeat
+				if err := deadmanDied(ctx, db, notify, hb, now); err != nil {
-			// stopped, so there is nothing left to wait out.
+					log.Printf("deadman: open incident for %s: %v", hb.sw.Matcher.Name, err)
-			if sw.resolved || sw.receivedAt < cutoff {
-				if err := deadmanDied(ctx, db, cfg, notify, sw, now); err != nil {
-					log.Printf("deadman: open incident for %s: %v", sw.matcher.Name, err)
 				}
 				continue
 			}
-			if err := deadmanRecovered(ctx, db, sw); err != nil {
+			if err := deadmanRecovered(ctx, db, hb); err != nil {
-				log.Printf("deadman: resolve incident for %s: %v", sw.matcher.Name, err)
+				log.Printf("deadman: resolve incident for %s: %v", hb.sw.Matcher.Name, err)
 			}
 		}
 	}
 	return owned
 }
 
-// deadmanAlerts loads every alert row that a matcher claims. The candidate query
+// deadmanAlerts loads every alert row that one of a team's switches claims. The candidate query
 // is narrowed by alertname so it rides alerts_name_idx; the rest of the matching
 // happens in Go, which keeps one implementation of the rules. The rows are read
 // in full before the caller writes, so the writes do not run against an open
 // cursor over the same table.
-func deadmanAlerts(ctx context.Context, db *sql.DB, teamID int64, cfg DeadmanConfig) ([]deadmanAlert, error) {
+func deadmanAlerts(ctx context.Context, db *sql.DB, teamID int64, cfg deadmanSet) ([]deadmanAlert, error) {
 	names := cfg.names()
 	args := &sqlArgs{}
 	nameList := make([]any, len(names))
@@ -263,11 +303,11 @@ func deadmanAlerts(ctx context.Context, db *sql.DB, teamID int64, cfg DeadmanCon
 		}
 		json.Unmarshal([]byte(labelsJSON), &a.labels) //nolint:errcheck
 
-		m, ok := cfg.match(a.labels)
+		sw, ok := cfg.match(a.labels)
 		if !ok {
 			continue
 		}
-		a.matcher = m
+		a.sw = sw
 		a.resolved = status == "resolved"
 		out = append(out, a)
 	}
@@ -284,16 +324,16 @@ func deadmanAlerts(ctx context.Context, db *sql.DB, teamID int64, cfg DeadmanCon
 // incidentForGroup), and a source that is gone for good is a one-time page
 // rather than a nag. Only a heartbeat that comes back and dies again earns a new
 // incident.
-func deadmanDied(ctx context.Context, db *sql.DB, cfg DeadmanConfig, notify NotifyConfig, sw deadmanAlert, now time.Time) error {
+func deadmanDied(ctx context.Context, db *sql.DB, notify NotifyConfig, hb deadmanAlert, now time.Time) error {
 	var lastTriggered, open int64
 	if err := db.QueryRowContext(ctx, `
 		SELECT COALESCE(MAX(triggered_at), 0),
 		       COUNT(*) FILTER (WHERE resolved_at IS NULL)
 		FROM   incidents WHERE team_id = $1 AND group_key = $2`,
-		sw.teamID, sw.groupKey()).Scan(&lastTriggered, &open); err != nil {
+		hb.teamID, hb.groupKey()).Scan(&lastTriggered, &open); err != nil {
 		return err
 	}
-	if open > 0 || sw.receivedAt <= lastTriggered {
+	if open > 0 || hb.receivedAt <= lastTriggered {
 		return nil
 	}
 
@@ -306,18 +346,18 @@ func deadmanDied(ctx context.Context, db *sql.DB, cfg DeadmanConfig, notify Noti
 	// A heartbeat nobody has heard from is not firing, and saying otherwise in
 	// the alert list would be a lie. An Alertmanager-sourced resolution keeps its
 	// own source: it told us the truth first.
-	if !sw.resolved {
+	if !hb.resolved {
 		if _, err := tx.ExecContext(ctx, `
 			UPDATE alerts
 			   SET status            = 'resolved',
 			       resolution_source = $1,
 			       ends_at           = COALESCE(ends_at, `+nowEpoch+`)
-			 WHERE id = $2 AND status = 'firing'`, resolutionDeadman, sw.id); err != nil {
+			 WHERE id = $2 AND status = 'firing'`, resolutionDeadman, hb.id); err != nil {
 			return err
 		}
 	}
 
-	severity := cfg.Severity
+	severity := hb.sw.Severity
 	var sev *string
 	if severity != "" {
 		sev = &severity
@@ -325,14 +365,14 @@ func deadmanDied(ctx context.Context, db *sql.DB, cfg DeadmanConfig, notify Noti
 
 	// The incident opens in the team whose integration received the heartbeat:
 	// the switch belongs to whoever is watching that source, not to the install.
-	incidentID, err := openIncident(ctx, tx, notify, sw.teamID, sw.groupKey(),
+	incidentID, err := openIncident(ctx, tx, notify, hb.teamID, hb.groupKey(),
-		"No heartbeat from "+sw.matcher.String(), sw.labels, sev)
+		"No heartbeat from "+hb.sw.Matcher.String(), hb.labels, sev)
 	if err != nil {
 		return err
 	}
 
-	alertID := sw.id
+	alertID := hb.id
-	detail := "last heartbeat " + humanDuration(now.Sub(time.Unix(sw.receivedAt, 0))) + " ago"
+	detail := "last heartbeat " + humanDuration(now.Sub(time.Unix(hb.receivedAt, 0))) + " ago"
 	if err := logEvent(ctx, tx, incidentID, evDeadmanSilent, nil, &alertID, &detail); err != nil {
 		return err
 	}
@@ -340,7 +380,7 @@ func deadmanDied(ctx context.Context, db *sql.DB, cfg DeadmanConfig, notify Noti
 	if err := tx.Commit(); err != nil {
 		return err
 	}
-	log.Printf("deadman: %s went silent, opened incident %d", sw.matcher.String(), incidentID)
+	log.Printf("deadman: %s went silent, opened incident %d", hb.sw.Matcher.String(), incidentID)
 	return nil
 }
 
@@ -350,12 +390,12 @@ func deadmanDied(ctx context.Context, db *sql.DB, cfg DeadmanConfig, notify Noti
 // member alerts (linking the heartbeat would have the settled-incident cascade
 // close it on the very same sweep that opened it), so the alert-driven cascade
 // ignores it entirely and recovery is the only automatic way out.
-func deadmanRecovered(ctx context.Context, db *sql.DB, sw deadmanAlert) error {
+func deadmanRecovered(ctx context.Context, db *sql.DB, hb deadmanAlert) error {
 	var incidentID int64
 	switch err := db.QueryRowContext(ctx, `
 		SELECT id FROM incidents
 		 WHERE team_id = $1 AND group_key = $2 AND resolved_at IS NULL`,
-		sw.teamID, sw.groupKey()).Scan(&incidentID); {
+		hb.teamID, hb.groupKey()).Scan(&incidentID); {
 	case err == sql.ErrNoRows:
 		return nil
 	case err != nil:
@@ -387,116 +427,256 @@ func deadmanRecovered(ctx context.Context, db *sql.DB, sw deadmanAlert) error {
 	if err := tx.Commit(); err != nil {
 		return err
 	}
-	log.Printf("deadman: %s is back, resolved incident %d", sw.matcher.String(), incidentID)
+	log.Printf("deadman: %s is back, resolved incident %d", hb.sw.Matcher.String(), incidentID)
 	return nil
 }
 
 // ---------------------------------------------------------------------------
-// Per-team configuration
+// A team's switches
 // ---------------------------------------------------------------------------
 
-// deadmanConfigForTeam reads one team's switches. A team with no row, or with
+const deadmanSwitchColumns = "id, team_id, name, matcher, timeout_seconds, severity"
-// nothing configured, gets a disabled config — which is the right answer rather
-// than an error: most teams watch no heartbeat at all.
-func deadmanConfigForTeam(ctx context.Context, q querier, teamID int64) (DeadmanConfig, error) {
-	var matchers, severity string
-	var timeout int64
-	err := q.QueryRowContext(ctx,
-		"SELECT matchers, timeout_seconds, severity FROM deadman_configs WHERE team_id = $1",
-		teamID).Scan(&matchers, &timeout, &severity)
-	if err == sql.ErrNoRows {
-		return DeadmanConfig{}, nil
-	}
-	if err != nil {
-		return DeadmanConfig{}, err
-	}
-	return parseDeadmanQuietly(matchers, time.Duration(timeout)*time.Second, severity), nil
-}
 
-// deadmanConfigs reads every team's switches in one query, for the sweeper.
+// scanDeadmanSwitches reads switch rows into per-team sets. A row whose matcher
-func deadmanConfigs(ctx context.Context, db *sql.DB) (map[int64]DeadmanConfig, error) {
+// no longer parses is skipped rather than fatal: the API refuses to store one,
-	rows, err := db.QueryContext(ctx,
+// so it can only mean a hand edit, and one bad row must not stop the others
-		"SELECT team_id, matchers, timeout_seconds, severity FROM deadman_configs")
+// from being watched.
-	if err != nil {
+func scanDeadmanSwitches(rows *sql.Rows) (map[int64]deadmanSet, error) {
-		return nil, err
-	}
 	defer rows.Close()
-
+	out := map[int64]deadmanSet{}
-	out := map[int64]DeadmanConfig{}
 	for rows.Next() {
+		var sw DeadmanSwitch
 		var teamID, timeout int64
-		var matchers, severity string
+		var matcher string
-		if err := rows.Scan(&teamID, &matchers, &timeout, &severity); err != nil {
+		if err := rows.Scan(&sw.ID, &teamID, &sw.Name, &matcher, &timeout, &sw.Severity); err != nil {
 			return nil, err
 		}
-		cfg := parseDeadmanQuietly(matchers, time.Duration(timeout)*time.Second, severity)
+		m, err := parseDeadmanMatcher(matcher)
-		if cfg.enabled() {
+		if err != nil {
-			out[teamID] = cfg
+			log.Printf("deadman: switch %d has an unusable matcher %q: %v", sw.ID, matcher, err)
+			continue
 		}
+		sw.Matcher = m
+		sw.Timeout = time.Duration(timeout) * time.Second
+		out[teamID] = append(out[teamID], sw)
 	}
 	return out, rows.Err()
 }
 
-// SeedDeadmanConfigs gives every team without a row the server's environment
+// deadmanSetForTeam reads one team's switches. A team with none gets an empty
-// configuration, so the install that upgrades into per-team switches keeps
+// set — which is the right answer rather than an error: most teams watch no
-// watching exactly what it was watching before.
+// heartbeat at all.
+func deadmanSetForTeam(ctx context.Context, q querier, teamID int64) (deadmanSet, error) {
+	rows, err := q.QueryContext(ctx,
+		"SELECT "+deadmanSwitchColumns+" FROM deadman_switches WHERE team_id = $1 ORDER BY id", teamID)
+	if err != nil {
+		return nil, err
+	}
+	sets, err := scanDeadmanSwitches(rows)
+	return sets[teamID], err
+}
+
+// deadmanSets reads every team's switches in one query, for the sweeper.
+func deadmanSets(ctx context.Context, db *sql.DB) (map[int64]deadmanSet, error) {
+	rows, err := db.QueryContext(ctx,
+		"SELECT "+deadmanSwitchColumns+" FROM deadman_switches ORDER BY id")
+	if err != nil {
+		return nil, err
+	}
+	return scanDeadmanSwitches(rows)
+}
+
+// deadmanSeededKey is the settings row that records the environment defaults
+// were handed out. Without it, a team that deleted its last switch would get
+// the default back on the next restart.
+const deadmanSeededKey = "deadman_seeded"
+
+// SeedDeadmanConfigs gives every team the server's environment defaults as
+// switches, exactly once per install, so a fresh install watches Watchdog
+// without anybody setting it up.
 //
-// Idempotent, and never overwrites: once a team has a row it owns its own
+// Once seeded it never runs again: a team's switches are its own, and a redeploy
-// configuration, and a redeploy must not quietly put the environment's value
+// must not quietly put the environment's value back over an owner's edit or
-// back over an owner's edit.
+// deletion. Installs that upgraded from per-team configuration were already
+// seeded, which migration 009 records.
 //
-// A team created after startup gets no row and therefore watches nothing until
+// A team created after that gets none and watches nothing until its owner says
-// its owner says otherwise. That is deliberate: inheriting an install-wide
+// otherwise. That is deliberate: inheriting an install-wide heartbeat would page
-// heartbeat would page a new team about a source it has never heard of, and a
+// a new team about a source it has never heard of, and a switch nobody chose is
-// switch nobody chose is the kind that gets muted rather than fixed.
+// the kind that gets muted rather than fixed.
 func SeedDeadmanConfigs(ctx context.Context, db *sql.DB, cfg DeadmanConfig) error {
-	matchers := make([]string, 0, len(cfg.Matchers))
+	if !cfg.enabled() {
-	for _, m := range cfg.Matchers {
+		return nil
-		parts := []string{"alertname=" + m.Name}
-		for k, v := range m.Labels {
-			parts = append(parts, k+"="+v)
-		}
-		sort.Strings(parts[1:])
-		matchers = append(matchers, strings.Join(parts, ","))
 	}
 
-	_, err := db.ExecContext(ctx, `
+	tx, err := db.BeginTx(ctx, nil)
-		INSERT INTO deadman_configs (team_id, matchers, timeout_seconds, severity)
+	if err != nil {
-		SELECT id, $1, $2, $3 FROM teams
-		ON CONFLICT (team_id) DO NOTHING`,
-		strings.Join(matchers, "; "), int64(cfg.Timeout.Seconds()), cfg.Severity)
 		return err
+	}
+	defer tx.Rollback() //nolint:errcheck
+
+	res, err := tx.ExecContext(ctx,
+		"INSERT INTO settings (key, value) VALUES ($1, '1') ON CONFLICT (key) DO NOTHING",
+		deadmanSeededKey)
+	if err != nil {
+		return err
+	}
+	if n, _ := res.RowsAffected(); n == 0 {
+		return nil
+	}
+
+	for _, m := range cfg.Matchers {
+		if _, err := tx.ExecContext(ctx, `
+			INSERT INTO deadman_switches (team_id, name, matcher, timeout_seconds, severity)
+			SELECT id, $1, $1, $2, $3 FROM teams`,
+			m.config(), int64(cfg.Timeout.Seconds()), cfg.Severity); err != nil {
+			return err
+		}
+	}
+	return tx.Commit()
 }
 
-// parseDeadmanQuietly is ParseDeadmanConfig without the startup logging: a
+// ---------------------------------------------------------------------------
-// team's configuration is read on every sweep and every webhook, and logging it
+// Status
-// each time would bury everything else.
+// ---------------------------------------------------------------------------
-func parseDeadmanQuietly(matchers string, timeout time.Duration, severity string) DeadmanConfig {
+
-	cfg := DeadmanConfig{Timeout: timeout, Severity: severity}
+const (
-	for _, entry := range strings.Split(matchers, ";") {
+	switchHealthy = "healthy"
-		entry = strings.TrimSpace(entry)
+	switchDead    = "dead"
-		if entry == "" {
+	switchDormant = "dormant"
-			continue
+)
-		}
+
-		m := DeadmanMatcher{Labels: map[string]string{}}
+// deadmanSource is one heartbeat under a switch: a fingerprint that matched.
-		malformed := false
+type deadmanSource struct {
-		for _, cond := range strings.Split(entry, ",") {
+	Fingerprint     string            `json:"fingerprint"`
-			k, v, ok := strings.Cut(cond, "=")
+	Labels          map[string]string `json:"labels"`
-			k, v = strings.TrimSpace(k), strings.TrimSpace(v)
+	Status          string            `json:"status"`
-			if !ok || k == "" || v == "" {
+	LastHeartbeatAt time.Time         `json:"last_heartbeat_at"`
-				malformed = true
+	LastTriggeredAt *time.Time        `json:"last_triggered_at"`
-				break
+	IncidentID      *int64            `json:"incident_id"`
-			}
+}
-			if k == "alertname" {
+
-				m.Name = v
+// deadmanSwitchStatus is a switch as the Switches page shows it.
-				continue
+type deadmanSwitchStatus struct {
-			}
+	ID             int64  `json:"id"`
-			m.Labels[k] = v
+	Name           string `json:"name"`
-		}
+	Matcher        string `json:"matcher"`
-		if malformed || m.Name == "" {
+	TimeoutSeconds int64  `json:"timeout_seconds"`
-			continue
+	Severity       string `json:"severity"`
-		}
+
-		cfg.Matchers = append(cfg.Matchers, m)
+	// Status is dead when any source is, dormant when none has ever been heard
-	}
+	// from, healthy otherwise — a live cluster must not hide a dead one.
-	return cfg
+	Status          string          `json:"status"`
+	LastHeartbeatAt *time.Time      `json:"last_heartbeat_at"`
+	LastTriggeredAt *time.Time      `json:"last_triggered_at"`
+	OpenIncidentID  *int64          `json:"open_incident_id"`
+	Sources         []deadmanSource `json:"sources"`
+}
+
+// deadmanStatuses reports every switch of a team with what its heartbeats are
+// doing. The liveness verdict is deadmanAlert.dead, the sweeper's own.
+func deadmanStatuses(ctx context.Context, db *sql.DB, teamID int64, set deadmanSet, now time.Time) ([]deadmanSwitchStatus, error) {
+	out := make([]deadmanSwitchStatus, 0, len(set))
+	if len(set) == 0 {
+		return out, nil
+	}
+
+	heartbeats, err := deadmanAlerts(ctx, db, teamID, set)
+	if err != nil {
+		return nil, err
+	}
+
+	// One query for every switch's incident history, keyed the way the sweeper
+	// keys it.
+	type history struct {
+		triggeredAt int64
+		openID      int64
+	}
+	incidents := map[string]history{}
+	rows, err := db.QueryContext(ctx, `
+		SELECT group_key, MAX(triggered_at), COALESCE(MAX(id) FILTER (WHERE resolved_at IS NULL), 0)
+		FROM   incidents
+		WHERE  team_id = $1 AND group_key LIKE $2
+		GROUP  BY group_key`, teamID, deadmanGroupPrefix+"%")
+	if err != nil {
+		return nil, err
+	}
+	defer rows.Close()
+	for rows.Next() {
+		var key string
+		var h history
+		if err := rows.Scan(&key, &h.triggeredAt, &h.openID); err != nil {
+			return nil, err
+		}
+		incidents[key] = h
+	}
+	if err := rows.Err(); err != nil {
+		return nil, err
+	}
+
+	bySwitch := map[int64][]deadmanAlert{}
+	for _, hb := range heartbeats {
+		bySwitch[hb.sw.ID] = append(bySwitch[hb.sw.ID], hb)
+	}
+
+	later := func(cur *time.Time, unix int64) *time.Time {
+		t := time.Unix(unix, 0).UTC()
+		if cur == nil || t.After(*cur) {
+			return &t
+		}
+		return cur
+	}
+
+	for _, sw := range set {
+		st := deadmanSwitchStatus{
+			ID: sw.ID, Name: sw.Name, Matcher: sw.Matcher.config(),
+			TimeoutSeconds: int64(sw.Timeout.Seconds()), Severity: sw.Severity,
+			Status: switchDormant, Sources: []deadmanSource{},
+		}
+
+		for _, hb := range bySwitch[sw.ID] {
+			src := deadmanSource{
+				Fingerprint:     hb.fingerprint,
+				Labels:          hb.labels,
+				Status:          switchHealthy,
+				LastHeartbeatAt: time.Unix(hb.receivedAt, 0).UTC(),
+			}
+			if hb.dead(now) {
+				src.Status = switchDead
+			}
+			if h, ok := incidents[hb.groupKey()]; ok {
+				t := time.Unix(h.triggeredAt, 0).UTC()
+				src.LastTriggeredAt = &t
+				st.LastTriggeredAt = later(st.LastTriggeredAt, h.triggeredAt)
+				if h.openID != 0 {
+					id := h.openID
+					src.IncidentID = &id
+					if st.OpenIncidentID == nil || id > *st.OpenIncidentID {
+						st.OpenIncidentID = &id
+					}
+				}
+			}
+			st.LastHeartbeatAt = later(st.LastHeartbeatAt, hb.receivedAt)
+			st.Sources = append(st.Sources, src)
+
+			switch {
+			case src.Status == switchDead:
+				st.Status = switchDead
+			case st.Status == switchDormant:
+				st.Status = switchHealthy
+			}
+		}
+
+		// Dead ones first, then by fingerprint: what needs attention leads, and
+		// the order does not shuffle between refreshes.
+		sort.Slice(st.Sources, func(i, j int) bool {
+			a, b := st.Sources[i], st.Sources[j]
+			if (a.Status == switchDead) != (b.Status == switchDead) {
+				return a.Status == switchDead
+			}
+			return a.Fingerprint < b.Fingerprint
+		})
+		out = append(out, st)
+	}
+	return out, nil
 }

internal/api/deadman_test.go

@@ -1,6 +1,7 @@
 package api_test
 
 import (
+	"context"
 	"net/http"
 	"strings"
 	"testing"
@@ -483,13 +484,13 @@ func TestDeadman_ConfigurationIsPerTeam(t *testing.T) {
 	unwatched := newTeam(t, s, "unwatched")
 
 	// Only the first team calls Watchdog a heartbeat.
-	resp := s.req(t, http.MethodPut, "/api/teams/"+id64(watched.id)+"/deadman", map[string]any{
+	resp := s.req(t, http.MethodPost, "/api/teams/"+id64(watched.id)+"/deadman/switches", map[string]any{
-		"matchers":        "alertname=Watchdog",
+		"matcher":         "alertname=Watchdog",
 		"timeout_seconds": 3600,
 		"severity":        "critical",
 	})
 	resp.Body.Close()
-	if resp.StatusCode != http.StatusOK {
+	if resp.StatusCode != http.StatusCreated {
 		t.Fatalf("configure the watched team: %d", resp.StatusCode)
 	}
 
@@ -550,9 +551,9 @@ func TestDeadman_ConfigurationIsOwnerOnly(t *testing.T) {
 	decode(t, s.req(t, http.MethodPost, "/api/users/"+id64(user.ID)+"/api-keys",
 		map[string]string{"name": "test"}), &key)
 
-	req, _ := http.NewRequest(http.MethodPut,
+	req, _ := http.NewRequest(http.MethodPost,
-		s.URL+"/api/teams/"+id64(team.id)+"/deadman",
+		s.URL+"/api/teams/"+id64(team.id)+"/deadman/switches",
-		strings.NewReader(`{"matchers":"alertname=Watchdog","timeout_seconds":60}`))
+		strings.NewReader(`{"matcher":"alertname=Watchdog","timeout_seconds":60}`))
 	req.Header.Set("Authorization", "Bearer "+key.Key)
 	req.Header.Set("Content-Type", "application/json")
 	resp, err := http.DefaultClient.Do(req)
@@ -564,7 +565,7 @@ func TestDeadman_ConfigurationIsOwnerOnly(t *testing.T) {
 		t.Errorf("a member editing the switches: expected 403, got %d", resp.StatusCode)
 	}
 
-	read, _ := http.NewRequest(http.MethodGet, s.URL+"/api/teams/"+id64(team.id)+"/deadman", nil)
+	read, _ := http.NewRequest(http.MethodGet, s.URL+"/api/teams/"+id64(team.id)+"/deadman/switches", nil)
 	read.Header.Set("Authorization", "Bearer "+key.Key)
 	got, err := http.DefaultClient.Do(read)
 	if err != nil {
@@ -577,16 +578,165 @@ func TestDeadman_ConfigurationIsOwnerOnly(t *testing.T) {
 }
 
 // A matcher with no alertname watches nothing, silently, which is the failure
-// this feature exists to prevent — so it is refused at the door.
+// this feature exists to prevent — so it is refused at the door, along with the
-func TestDeadman_UnusableMatchersAreRejected(t *testing.T) {
+// other things that would make a switch unable to fire.
+func TestDeadman_UnusableSwitchesAreRejected(t *testing.T) {
 	s, _ := deadmanTS(t, deadmanCfg())
 
-	resp := s.req(t, http.MethodPut, "/api/teams/"+defaultTeam+"/deadman", map[string]any{
+	for name, body := range map[string]map[string]any{
-		"matchers":        "cluster=prod",
+		"no alertname":  {"matcher": "cluster=prod", "timeout_seconds": 900},
-		"timeout_seconds": 900,
+		"malformed":     {"matcher": "alertname=Watchdog,garbage", "timeout_seconds": 900},
-	})
+		"several":       {"matcher": "alertname=A; alertname=B", "timeout_seconds": 900},
+		"zero timeout":  {"matcher": "alertname=Watchdog", "timeout_seconds": 0},
+		"bad severity":  {"matcher": "alertname=Watchdog", "timeout_seconds": 900, "severity": "loud"},
+		"empty matcher": {"matcher": "", "timeout_seconds": 900},
+	} {
+		resp := s.req(t, http.MethodPost, "/api/teams/"+defaultTeam+"/deadman/switches", body)
 		resp.Body.Close()
 		if resp.StatusCode != http.StatusBadRequest {
-		t.Errorf("expected 400 for a matcher with no alertname, got %d", resp.StatusCode)
+			t.Errorf("%s: expected 400, got %d", name, resp.StatusCode)
+		}
+	}
+}
+
+// ---------------------------------------------------------------------------
+// The switch list
+// ---------------------------------------------------------------------------
+
+// listSwitches reads the default team's switches as the Switches page does.
+func listSwitches(t *testing.T, s *ts) []map[string]any {
+	t.Helper()
+	return list(t, s.req(t, http.MethodGet, "/api/teams/"+defaultTeam+"/deadman/switches", nil))
+}
+
+// A switch is healthy while its heartbeat is fresh, dead once it is silent, and
+// dormant until the first one arrives.
+func TestDeadman_ListReportsStatus(t *testing.T) {
+	s, _ := deadmanTS(t, api.ParseDeadmanConfig("alertname=Watchdog; alertname=NeverSent", time.Hour, "critical"))
+
+	got := listSwitches(t, s)
+	if len(got) != 2 {
+		t.Fatalf("expected 2 switches, got %d", len(got))
+	}
+	for _, sw := range got {
+		if sw["status"] != "dormant" || sw["last_heartbeat_at"] != nil || sw["last_triggered_at"] != nil {
+			t.Errorf("a switch nobody has heard from should be dormant and blank, got %v", sw)
+		}
+	}
+
+	heartbeat(t, s, "fp-watchdog", nil)
+	got = listSwitches(t, s)
+	if got[0]["status"] != "healthy" || got[0]["last_heartbeat_at"] == nil {
+		t.Errorf("a fresh heartbeat should be healthy with a timestamp, got %v", got[0])
+	}
+	if got[1]["status"] != "dormant" {
+		t.Errorf("the other switch is still dormant, got %v", got[1]["status"])
+	}
+
+	silence(t, s, "fp-watchdog", 2*time.Hour)
+	sweep(t, s, noArchive)
+	got = listSwitches(t, s)
+	if got[0]["status"] != "dead" {
+		t.Fatalf("a silent heartbeat should be dead, got %v", got[0]["status"])
+	}
+	if got[0]["last_triggered_at"] == nil || got[0]["open_incident_id"] == nil {
+		t.Errorf("a dead switch should show when it triggered and its open incident, got %v", got[0])
+	}
+}
+
+// One matcher, several clusters: the switch is as bad as its worst heartbeat and
+// each heartbeat is listed on its own.
+func TestDeadman_ListBreaksDownByFingerprint(t *testing.T) {
+	s, _ := deadmanTS(t, deadmanCfg())
+
+	heartbeat(t, s, "fp-a", map[string]string{"cluster": "a"})
+	heartbeat(t, s, "fp-b", map[string]string{"cluster": "b"})
+	silence(t, s, "fp-b", 2*time.Hour)
+
+	sw := listSwitches(t, s)[0]
+	if sw["status"] != "dead" {
+		t.Errorf("one dead cluster makes the switch dead, got %v", sw["status"])
+	}
+	sources := sw["sources"].([]any)
+	if len(sources) != 2 {
+		t.Fatalf("expected 2 sources, got %d", len(sources))
+	}
+	first, second := sources[0].(map[string]any), sources[1].(map[string]any)
+	if first["fingerprint"] != "fp-b" || first["status"] != "dead" || second["status"] != "healthy" {
+		t.Errorf("the dead source should lead, got %v then %v", first, second)
+	}
+}
+
+// Every switch keeps its own deadline.
+func TestDeadman_TimeoutsArePerSwitch(t *testing.T) {
+	s, _ := deadmanTS(t, deadmanCfg())
+	resp := s.req(t, http.MethodPost, "/api/teams/"+defaultTeam+"/deadman/switches", map[string]any{
+		"matcher": "alertname=Edge", "timeout_seconds": 300,
+	})
+	resp.Body.Close()
+
+	heartbeat(t, s, "fp-watchdog", nil)
+	postWebhook(t, s, []map[string]any{
+		amAlert("fp-edge", "Edge", "firing", "2026-05-20T10:00:00Z", zeroTime, nil),
+	}, `{}:{alertname="Edge"}`)
+
+	// Ten minutes of silence: past the Edge switch's five, inside Watchdog's hour.
+	silence(t, s, "fp-watchdog", 10*time.Minute)
+	silence(t, s, "fp-edge", 10*time.Minute)
+
+	got := listSwitches(t, s)
+	if got[0]["status"] != "healthy" || got[1]["status"] != "dead" {
+		t.Errorf("want Watchdog healthy and Edge dead, got %v and %v", got[0]["status"], got[1]["status"])
+	}
+}
+
+// Deleting is an owner's, is scoped to the team, and leaves what the switch
+// already opened alone.
+func TestDeadman_DeleteIsScopedToTheTeam(t *testing.T) {
+	s, _ := deadmanTS(t, deadmanCfg())
+	other := newTeam(t, s, "other")
+
+	id := int64(listSwitches(t, s)[0]["id"].(float64))
+
+	// Another team's owner cannot reach it.
+	resp := other.call(http.MethodDelete, "/api/teams/"+id64(other.id)+"/deadman/switches/"+id64(id), nil)
+	resp.Body.Close()
+	if resp.StatusCode != http.StatusNotFound {
+		t.Errorf("deleting another team's switch: expected 404, got %d", resp.StatusCode)
+	}
+	if got := len(listSwitches(t, s)); got != 1 {
+		t.Fatalf("the switch should have survived, %d left", got)
+	}
+
+	resp = s.req(t, http.MethodDelete, "/api/teams/"+defaultTeam+"/deadman/switches/"+id64(id), nil)
+	resp.Body.Close()
+	if resp.StatusCode != http.StatusNoContent {
+		t.Fatalf("deleting: expected 204, got %d", resp.StatusCode)
+	}
+	if got := len(listSwitches(t, s)); got != 0 {
+		t.Errorf("expected no switches, got %d", got)
+	}
+}
+
+// The environment's defaults are handed out once and then belong to the teams.
+func TestDeadman_SeedRunsOnce(t *testing.T) {
+	s := newTS(t)
+	cfg := api.ParseDeadmanConfig("alertname=Watchdog", time.Hour, "critical")
+
+	if err := api.SeedDeadmanConfigs(context.Background(), s.db, cfg); err != nil {
+		t.Fatalf("seed: %v", err)
+	}
+	if got := len(listSwitches(t, s)); got != 1 {
+		t.Fatalf("the first seed should add the default, got %d switches", got)
+	}
+
+	// The owner deletes it; a restart must not put it back.
+	id := int64(listSwitches(t, s)[0]["id"].(float64))
+	s.req(t, http.MethodDelete, "/api/teams/"+defaultTeam+"/deadman/switches/"+id64(id), nil).Body.Close()
+	if err := api.SeedDeadmanConfigs(context.Background(), s.db, cfg); err != nil {
+		t.Fatalf("seed again: %v", err)
+	}
+	if got := len(listSwitches(t, s)); got != 0 {
+		t.Errorf("a second seed resurrected %d switch(es)", got)
 	}
 }

internal/api/incident_store.go

@@ -36,6 +36,10 @@ const (
 	evUnsnoozed      = "unsnoozed"
 	evResolved       = "resolved"
 	evNote           = "note"
+	// evResolutionNote is the note worth finding again: what fixed it. The
+	// similar-incidents lookup and the page lead with these; plain notes are
+	// the working chatter and stay one click away.
+	evResolutionNote = "resolution_note"
 	evDeadmanSilent  = "deadman_silent"
 )
 
@@ -295,6 +299,34 @@ func openIncidentForAlert(ctx context.Context, q querier, alertID int64) (int64,
 	return id, err
 }
 
+// volatileLabels say where a problem ran this time, not what the problem is, so
+// they stay out of the signature. Migration 008's backfill lists the same set.
+var volatileLabels = map[string]bool{
+	"instance": true, "pod": true, "pod_name": true, "pod_ip": true,
+	"container": true, "container_name": true, "endpoint": true,
+}
+
+// incidentSignature identifies "the same problem" across incidents: the alert
+// name plus the stable group labels, sorted. Incidents in one team with equal
+// signatures are what the similar-incidents lookup returns. title stands in for
+// the name when the payload carried no alertname (groupless and dead man's
+// switch incidents).
+func incidentSignature(groupLabels map[string]string, title string) string {
+	name := groupLabels["alertname"]
+	if name == "" {
+		name = title
+	}
+	rest := make([]string, 0, len(groupLabels))
+	for k, v := range groupLabels {
+		if k == "alertname" || volatileLabels[k] {
+			continue
+		}
+		rest = append(rest, k+"="+v)
+	}
+	sort.Strings(rest)
+	return name + "|" + strings.Join(rest, ",")
+}
+
 // incidentTitle renders a human-readable title from Alertmanager's groupLabels,
 // leading with the alert name and appending whatever else the operator grouped
 // by. Falls back to the alert's own name when the payload carried no groupLabels.

internal/api/incidents.go

@@ -3,6 +3,7 @@ package api
 import (
 	"database/sql"
 	"fmt"
+	"io"
 	"net/http"
 	"strconv"
 	"strings"
@@ -237,6 +238,15 @@ func handleIncidentResolve(db *sql.DB) http.HandlerFunc {
 			return
 		}
 		user, _ := userFromContext(r.Context())
+		// The body is optional: clients that predate resolution notes send none.
+		var req struct {
+			Resolution string `json:"resolution"`
+		}
+		if err := decodeJSON(r, &req); err != nil && err != io.EOF {
+			respond(w, http.StatusBadRequest, errResp("invalid request body"))
+			return
+		}
+		req.Resolution = strings.TrimSpace(req.Resolution)
 		if !updateOpenIncident(w, r, db, id,
 			`UPDATE incidents SET status = 'resolved', resolved_at = $1, resolution_source = $2
 			  WHERE id = $3 AND resolved_at IS NULL`,
@@ -252,6 +262,12 @@ func handleIncidentResolve(db *sql.DB) http.HandlerFunc {
 			respond(w, http.StatusInternalServerError, errResp("internal error"))
 			return
 		}
+		if req.Resolution != "" {
+			if err := logEvent(r.Context(), db, id, evResolutionNote, &user.ID, nil, &req.Resolution); err != nil {
+				respond(w, http.StatusInternalServerError, errResp("internal error"))
+				return
+			}
+		}
 		respondIncident(w, r, db, id)
 	}
 }
@@ -421,11 +437,17 @@ func handleCreateNote(db *sql.DB) http.HandlerFunc {
 		}
 		var req struct {
 			Content string `json:"content"`
+			// Pinned files the note as the resolution note: what fixed it.
+			Pinned bool `json:"pinned"`
 		}
 		if err := decodeJSON(r, &req); err != nil {
 			respond(w, http.StatusBadRequest, errResp("invalid request body"))
 			return
 		}
+		noteType := evNote
+		if req.Pinned {
+			noteType = evResolutionNote
+		}
 		if req.Content == "" {
 			respond(w, http.StatusBadRequest, errResp("content is required"))
 			return
@@ -440,7 +462,7 @@ func handleCreateNote(db *sql.DB) http.HandlerFunc {
 		err := db.QueryRowContext(r.Context(), `
 			INSERT INTO incident_events (incident_id, type, user_id, detail, created_at)
 			VALUES ($1, $2, $3, $4, $5)
-			RETURNING id`, id, evNote, user.ID, req.Content, now.Unix()).Scan(&eventID)
+			RETURNING id`, id, noteType, user.ID, req.Content, now.Unix()).Scan(&eventID)
 		if err != nil {
 			respond(w, http.StatusInternalServerError, errResp("internal error"))
 			return
@@ -449,7 +471,7 @@ func handleCreateNote(db *sql.DB) http.HandlerFunc {
 		respond(w, http.StatusCreated, models.IncidentEvent{
 			ID:         eventID,
 			IncidentID: id,
-			Type:       evNote,
+			Type:       noteType,
 			UserID:     &user.ID,
 			Username:   &user.Username,
 			Detail:     &req.Content,
@@ -475,8 +497,8 @@ func handleDeleteNote(db *sql.DB) http.HandlerFunc {
 		user, _ := userFromContext(r.Context())
 		res, err := db.ExecContext(r.Context(), `
 			DELETE FROM incident_events
-			 WHERE id = $1 AND incident_id = $2 AND type = $3 AND user_id = $4`,
+			 WHERE id = $1 AND incident_id = $2 AND type IN ($3, $4) AND user_id = $5`,
-			eventID, id, evNote, user.ID)
+			eventID, id, evNote, evResolutionNote, user.ID)
 		if err != nil {
 			respond(w, http.StatusInternalServerError, errResp("internal error"))
 			return

internal/api/notifier.go

@@ -330,6 +330,16 @@ func deliver(ctx context.Context, db *sql.DB, cfg NotifyConfig, n outboxRow) err
 
 	msg := renderNotification(inc, n, firing, cfg)
 
+	// The page that opens an incident carries what fixed it last time, so the
+	// person woken up starts from that. Best effort: a failed lookup must not
+	// hold back the page itself.
+	if n.kind == notifyTriggered {
+		if sim, err := similarIncidents(ctx, db, n.incidentID, 1); err == nil && len(sim) > 0 && len(sim[0].ResolutionNotes) > 0 {
+			notes := sim[0].ResolutionNotes
+			msg.Message += "\nLast time: " + shorten(derefString(notes[len(notes)-1].Detail), 160)
+		}
+	}
+
 	// An Acknowledge button needs both a user to attribute the acknowledgement
 	// to and a URL the phone can reach. Minted per delivery, so every push
 	// carries its own short-lived token rather than reusing one.
@@ -573,6 +583,17 @@ func plural(n int) string {
 	return "s"
 }
 
+// shorten cuts s to at most n runes, marking the cut, and flattens newlines so
+// a multi-line note stays one line in a push.
+func shorten(s string, n int) string {
+	s = strings.Join(strings.Fields(s), " ")
+	r := []rune(s)
+	if len(r) <= n {
+		return s
+	}
+	return string(r[:n-1]) + "…"
+}
+
 // derefString reads a nullable text column as a plain string.
 func derefString(s *string) string {
 	if s == nil {

internal/api/router.go

@@ -112,6 +112,7 @@ func NewRouter(db *sql.DB, notify NotifyConfig, cfg config.Config) http.Handler
 		r.Get("/api/incidents/{id}", handleGetIncident(db))
 		r.Get("/api/incidents/{id}/alerts", handleIncidentAlerts(db))
 		r.Get("/api/incidents/{id}/timeline", handleIncidentTimeline(db))
+		r.Get("/api/incidents/{id}/similar", handleIncidentSimilar(db))
 		r.Post("/api/incidents/{id}/acknowledge", handleIncidentAcknowledge(db))
 		r.Delete("/api/incidents/{id}/acknowledge", handleIncidentUnacknowledge(db))
 		r.Post("/api/incidents/{id}/resolve", handleIncidentResolve(db))
@@ -143,12 +144,14 @@ func NewRouter(db *sql.DB, notify NotifyConfig, cfg config.Config) http.Handler
 
 		// A team's own dead man's switches: which of its alerts are heartbeats,
 		// and how long a silence has to last before somebody is paged.
-		r.Get("/api/teams/{teamID}/deadman", handleGetTeamDeadman(db))
+		r.Get("/api/teams/{teamID}/deadman/switches", handleListTeamDeadman(db))
-		r.Put("/api/teams/{teamID}/deadman", handleSetTeamDeadman(db))
+		r.Post("/api/teams/{teamID}/deadman/switches", handleCreateTeamDeadman(db))
+		r.Delete("/api/teams/{teamID}/deadman/switches/{switchID}", handleDeleteTeamDeadman(db))
 
 		// Integrations: where a team's alerts come in, and the key that says so.
 		r.Get("/api/teams/{teamID}/integrations", handleListIntegrations(db))
 		r.Post("/api/teams/{teamID}/integrations", handleCreateIntegration(db, notify.PublicURL))
+		r.Patch("/api/teams/{teamID}/integrations/{integrationID}", handleRenameIntegration(db))
 		r.Delete("/api/teams/{teamID}/integrations/{integrationID}", handleDeleteIntegration(db))
 
 		// The rota is per team. /api/schedule/current is the exception: it

internal/api/similar.go

@@ -0,0 +1,113 @@
+package api
+
+import (
+	"context"
+	"database/sql"
+	"net/http"
+	"strconv"
+	"time"
+
+	"git.ryuvia.com/niklas/terdut-server/internal/models"
+)
+
+const (
+	similarDefaultLimit = 5
+	similarMaxLimit     = 20
+)
+
+// handleIncidentSimilar lists earlier, resolved incidents in the same team with
+// the same signature that someone left notes on, incidents with a resolution
+// note first. This is the "have we seen this before" answer for a responder
+// looking at a fresh incident; the plain notes are one timeline fetch away.
+func handleIncidentSimilar(db *sql.DB) http.HandlerFunc {
+	return func(w http.ResponseWriter, r *http.Request) {
+		id, ok := incidentIDParam(w, r, db)
+		if !ok {
+			return
+		}
+		limit := similarDefaultLimit
+		if v := r.URL.Query().Get("limit"); v != "" {
+			n, err := strconv.Atoi(v)
+			if err != nil || n < 1 {
+				respond(w, http.StatusBadRequest, errResp("invalid limit"))
+				return
+			}
+			limit = min(n, similarMaxLimit)
+		}
+
+		out, err := similarIncidents(r.Context(), db, id, limit)
+		if err != nil {
+			respond(w, http.StatusInternalServerError, errResp("internal error"))
+			return
+		}
+		respond(w, http.StatusOK, out)
+	}
+}
+
+func similarIncidents(ctx context.Context, q querier, id int64, limit int) ([]models.SimilarIncident, error) {
+	rows, err := q.QueryContext(ctx, `
+		SELECT o.id, o.title, o.triggered_at, o.resolved_at,
+		       (SELECT COUNT(*) FROM incident_events e
+		         WHERE e.incident_id = o.id AND e.type = $3)
+		FROM   incidents i
+		JOIN   incidents o ON o.team_id = i.team_id AND o.signature = i.signature
+		WHERE  i.id = $1 AND o.id <> i.id AND o.resolved_at IS NOT NULL
+		  AND  EXISTS (SELECT 1 FROM incident_events e
+		                WHERE e.incident_id = o.id AND e.type IN ($3, $4))
+		ORDER  BY EXISTS (SELECT 1 FROM incident_events e
+		                   WHERE e.incident_id = o.id AND e.type = $4) DESC,
+		          o.triggered_at DESC
+		LIMIT  $2`, id, limit, evNote, evResolutionNote)
+	if err != nil {
+		return nil, err
+	}
+	defer rows.Close()
+
+	out := []models.SimilarIncident{}
+	ids := []int64{}
+	for rows.Next() {
+		var s models.SimilarIncident
+		var triggered, resolved int64
+		if err := rows.Scan(&s.ID, &s.Title, &triggered, &resolved, &s.NoteCount); err != nil {
+			return nil, err
+		}
+		s.TriggeredAt = time.Unix(triggered, 0).UTC()
+		s.ResolvedAt = time.Unix(resolved, 0).UTC()
+		s.ResolutionNotes = []models.IncidentEvent{}
+		out = append(out, s)
+		ids = append(ids, s.ID)
+	}
+	if err := rows.Err(); err != nil {
+		return nil, err
+	}
+	if len(out) == 0 {
+		return out, nil
+	}
+
+	nrows, err := q.QueryContext(ctx, `
+		SELECT e.id, e.incident_id, e.type, e.user_id, u.username, e.detail, e.created_at
+		FROM   incident_events e
+		LEFT   JOIN users u ON u.id = e.user_id
+		WHERE  e.incident_id = ANY($1) AND e.type = $2
+		ORDER  BY e.created_at, e.id`, ids, evResolutionNote)
+	if err != nil {
+		return nil, err
+	}
+	defer nrows.Close()
+
+	byID := make(map[int64]*models.SimilarIncident, len(out))
+	for i := range out {
+		byID[out[i].ID] = &out[i]
+	}
+	for nrows.Next() {
+		var e models.IncidentEvent
+		var ts int64
+		if err := nrows.Scan(&e.ID, &e.IncidentID, &e.Type, &e.UserID, &e.Username, &e.Detail, &ts); err != nil {
+			return nil, err
+		}
+		e.CreatedAt = time.Unix(ts, 0).UTC()
+		s := byID[e.IncidentID]
+		s.ResolutionNotes = append(s.ResolutionNotes, e)
+	}
+	return out, nrows.Err()
+}

internal/api/similar_test.go

@@ -0,0 +1,99 @@
+package api_test
+
+import (
+	"bytes"
+	"encoding/json"
+	"net/http"
+	"strconv"
+	"testing"
+)
+
+// postGrouped posts a firing webhook whose group labels are exactly the given
+// map, unlike postWebhook, which only ever groups by alertname.
+func postGrouped(t *testing.T, s *ts, fingerprint, startsAt string, groupLabels map[string]string) {
+	t.Helper()
+	labels := map[string]string{}
+	for k, v := range groupLabels {
+		labels[k] = v
+	}
+	payload := map[string]any{
+		"version": "4", "status": "firing",
+		"groupKey":    fingerprint,
+		"groupLabels": groupLabels,
+		"alerts": []map[string]any{
+			amAlert(fingerprint, groupLabels["alertname"], "firing", startsAt, zeroTime, labels),
+		},
+	}
+	data, _ := json.Marshal(payload)
+	resp, err := http.Post(s.URL+"/api/integrations/"+s.ingestKey+"/alertmanager",
+		"application/json", bytes.NewReader(data))
+	if err != nil {
+		t.Fatalf("post webhook: %v", err)
+	}
+	resp.Body.Close()
+}
+
+func similar(t *testing.T, s *ts, id int) []map[string]any {
+	t.Helper()
+	var out []map[string]any
+	decode(t, s.req(t, http.MethodGet, "/api/incidents/"+strconv.Itoa(id)+"/similar", nil), &out)
+	return out
+}
+
+// Same alert on another instance is the same problem; a resolution note left on
+// the first one is what the second one should be shown.
+func TestSimilar_IgnoresVolatileLabelsAndLeadsWithResolutionNote(t *testing.T) {
+	s := newTS(t)
+	postGrouped(t, s, "fp-a", "2026-05-20T10:00:00Z",
+		map[string]string{"alertname": "DiskFull", "instance": "web-1", "job": "node"})
+	s.req(t, http.MethodPost, "/api/incidents/1/resolve",
+		map[string]string{"resolution": "rotated the logs"}).Body.Close()
+
+	postGrouped(t, s, "fp-b", "2026-05-21T10:00:00Z",
+		map[string]string{"alertname": "DiskFull", "instance": "web-2", "job": "node"})
+
+	got := similar(t, s, 2)
+	if len(got) != 1 || int(got[0]["id"].(float64)) != 1 {
+		t.Fatalf("expected incident 1 as the only similar one, got %v", got)
+	}
+	notes := got[0]["resolution_notes"].([]any)
+	if len(notes) != 1 || notes[0].(map[string]any)["detail"] != "rotated the logs" {
+		t.Fatalf("expected the resolution note, got %v", notes)
+	}
+}
+
+// A different stable label (job) is a different problem, and an incident nobody
+// wrote a note on has nothing to show.
+func TestSimilar_DifferentSignatureOrNoNotesIsExcluded(t *testing.T) {
+	s := newTS(t)
+	postGrouped(t, s, "fp-1", "2026-05-20T10:00:00Z",
+		map[string]string{"alertname": "DiskFull", "job": "node"})
+	s.req(t, http.MethodPost, "/api/incidents/1/notes", map[string]string{"content": "checked"}).Body.Close()
+	s.req(t, http.MethodPost, "/api/incidents/1/resolve", nil).Body.Close()
+
+	postGrouped(t, s, "fp-2", "2026-05-20T11:00:00Z",
+		map[string]string{"alertname": "DiskFull", "job": "db"})
+	s.req(t, http.MethodPost, "/api/incidents/2/resolve", nil).Body.Close()
+
+	postGrouped(t, s, "fp-3", "2026-05-21T10:00:00Z",
+		map[string]string{"alertname": "DiskFull", "job": "db"})
+
+	// Incident 3 matches 2 by signature, but 2 has no notes.
+	if got := similar(t, s, 3); len(got) != 0 {
+		t.Fatalf("expected nothing similar to incident 3, got %v", got)
+	}
+}
+
+// An open incident is not "earlier experience" yet, and the incident itself is
+// never its own match.
+func TestSimilar_OpenIncidentsAreNotListed(t *testing.T) {
+	s := newTS(t)
+	postGrouped(t, s, "fp-o1", "2026-05-20T10:00:00Z", map[string]string{"alertname": "Flap"})
+	s.req(t, http.MethodPost, "/api/incidents/1/notes",
+		map[string]any{"content": "still open", "pinned": true}).Body.Close()
+	postGrouped(t, s, "fp-o2", "2026-05-21T10:00:00Z", map[string]string{"alertname": "Flap"})
+
+	if got := similar(t, s, 2); len(got) != 0 {
+		t.Fatalf("expected an open incident not to be listed, got %v", got)
+	}
+}

internal/api/sources_test.go

@@ -0,0 +1,171 @@
+package api_test
+
+import (
+	"bytes"
+	"net/http"
+	"strings"
+	"testing"
+	"time"
+)
+
+// listSources reads a team's alert sources as the Sources page does.
+func listSources(t *testing.T, tm teamFixture) []map[string]any {
+	t.Helper()
+	return list(t, tm.call(http.MethodGet, "/api/teams/"+id64(tm.id)+"/integrations", nil))
+}
+
+// addSource mints a second source in a team and returns its key.
+func addSource(t *testing.T, tm teamFixture, name string) string {
+	t.Helper()
+	var out struct {
+		Key string `json:"key"`
+	}
+	decode(t, tm.call(http.MethodPost, "/api/teams/"+id64(tm.id)+"/integrations",
+		map[string]string{"name": name}), &out)
+	return out.Key
+}
+
+// A source that has never posted is "never", with nothing to say about alerts.
+func TestSources_NeverUsedIsBlank(t *testing.T) {
+	s := newTS(t)
+	tm := newTeam(t, s, "red")
+
+	got := listSources(t, tm)
+	if len(got) != 1 {
+		t.Fatalf("expected 1 source, got %d", len(got))
+	}
+	src := got[0]
+	if src["status"] != "never" || src["last_used_at"] != nil || src["last_alert_at"] != nil {
+		t.Errorf("a source nobody has posted on should be blank, got %v", src)
+	}
+	if src["alerts_24h"].(float64) != 0 {
+		t.Errorf("alerts_24h = %v, want 0", src["alerts_24h"])
+	}
+}
+
+// Each source is credited with what arrived on its own key, and only that.
+func TestSources_AlertsAreAttributedToTheirSource(t *testing.T) {
+	s := newTS(t)
+	tm := newTeam(t, s, "red")
+	second := addSource(t, tm, "staging")
+
+	postToIntegration(t, s, tm.key, "fp-1", "DiskFull")
+	postToIntegration(t, s, tm.key, "fp-2", "CPUHot")
+
+	got := listSources(t, tm)
+	first, other := got[0], got[1]
+	if first["status"] != "active" || first["last_used_at"] == nil || first["last_alert_at"] == nil {
+		t.Errorf("the source that posted should be active with timestamps, got %v", first)
+	}
+	if first["alerts_24h"].(float64) != 2 {
+		t.Errorf("alerts_24h = %v, want 2", first["alerts_24h"])
+	}
+	if other["status"] != "never" || other["alerts_24h"].(float64) != 0 {
+		t.Errorf("the other source should be untouched, got %v", other)
+	}
+
+	// Re-sending the same alert on the other key moves it: last sender wins.
+	postToIntegration(t, s, second, "fp-1", "DiskFull")
+	got = listSources(t, tm)
+	if got[0]["alerts_24h"].(float64) != 1 || got[1]["alerts_24h"].(float64) != 1 {
+		t.Errorf("fp-1 should have moved to the second source, got %v and %v",
+			got[0]["alerts_24h"], got[1]["alerts_24h"])
+	}
+}
+
+// A payload with no alerts in it is a webhook, not an alert: the source was
+// heard from, and nothing arrived.
+func TestSources_EmptyPayloadStampsUseButNotAlert(t *testing.T) {
+	s := newTS(t)
+	tm := newTeam(t, s, "red")
+
+	resp, err := http.Post(s.URL+"/api/integrations/"+tm.key+"/alertmanager",
+		"application/json", bytes.NewReader([]byte(`{"version":"4","status":"firing","alerts":[]}`)))
+	if err != nil {
+		t.Fatalf("post: %v", err)
+	}
+	resp.Body.Close()
+
+	src := listSources(t, tm)[0]
+	if src["status"] != "active" || src["last_alert_at"] != nil {
+		t.Errorf("want active with no alert yet, got %v", src)
+	}
+}
+
+// Quiet is "has posted, not lately"; the alert counter forgets after a day but
+// the last alert's timestamp is kept.
+func TestSources_QuietAfterADay(t *testing.T) {
+	s := newTS(t)
+	tm := newTeam(t, s, "red")
+	postToIntegration(t, s, tm.key, "fp-1", "DiskFull")
+
+	old := time.Now().Add(-48 * time.Hour).Unix()
+	s.exec(t, "UPDATE integrations SET last_used_at = $1", old)
+	s.exec(t, "UPDATE alerts SET received_at = $1 WHERE fingerprint = 'fp-1'", old)
+
+	src := listSources(t, tm)[0]
+	if src["status"] != "quiet" {
+		t.Errorf("status = %v, want quiet", src["status"])
+	}
+	if src["alerts_24h"].(float64) != 0 {
+		t.Errorf("alerts_24h = %v, want 0", src["alerts_24h"])
+	}
+	if src["last_alert_at"] == nil {
+		t.Error("last_alert_at should survive the day")
+	}
+}
+
+// Revoking a source does not take its alerts with it.
+func TestSources_RevokeKeepsTheAlerts(t *testing.T) {
+	s := newTS(t)
+	tm := newTeam(t, s, "red")
+	postToIntegration(t, s, tm.key, "fp-1", "DiskFull")
+
+	id := int64(listSources(t, tm)[0]["id"].(float64))
+	resp := tm.call(http.MethodDelete, "/api/teams/"+id64(tm.id)+"/integrations/"+id64(id), nil)
+	resp.Body.Close()
+	if resp.StatusCode != http.StatusNoContent {
+		t.Fatalf("revoke: %d", resp.StatusCode)
+	}
+	if got := len(list(t, tm.call(http.MethodGet, "/api/alerts", nil))); got != 1 {
+		t.Errorf("the alert should outlive its source, got %d alerts", got)
+	}
+}
+
+// Renaming is an owner's, scoped to the team, and does not touch the key.
+func TestSources_Rename(t *testing.T) {
+	s := newTS(t)
+	tm := newTeam(t, s, "red")
+	other := newTeam(t, s, "blue")
+	id := int64(listSources(t, tm)[0]["id"].(float64))
+	path := "/api/teams/" + id64(tm.id) + "/integrations/" + id64(id)
+
+	resp := tm.call(http.MethodPatch, path, map[string]string{"name": "  prod  "})
+	resp.Body.Close()
+	if resp.StatusCode != http.StatusNoContent {
+		t.Fatalf("rename: %d", resp.StatusCode)
+	}
+	if name := listSources(t, tm)[0]["name"]; name != "prod" {
+		t.Errorf("name = %q, want it trimmed to prod", name)
+	}
+	postToIntegration(t, s, tm.key, "fp-1", "DiskFull") // the old key still works
+
+	for name, body := range map[string]map[string]string{
+		"empty":    {"name": "  "},
+		"too long": {"name": strings.Repeat("x", 101)},
+	} {
+		resp := tm.call(http.MethodPatch, path, body)
+		resp.Body.Close()
+		if resp.StatusCode != http.StatusBadRequest {
+			t.Errorf("%s name: expected 400, got %d", name, resp.StatusCode)
+		}
+	}
+
+	// Another team's owner cannot reach it.
+	resp = other.call(http.MethodPatch, "/api/teams/"+id64(other.id)+"/integrations/"+id64(id),
+		map[string]string{"name": "mine now"})
+	resp.Body.Close()
+	if resp.StatusCode != http.StatusNotFound {
+		t.Errorf("renaming another team's source: expected 404, got %d", resp.StatusCode)
+	}
+}

internal/api/teams.go

@@ -355,8 +355,42 @@ func isLastTeamOwner(ctx context.Context, db *sql.DB, teamID, userID int64) (boo
 // Integrations
 // ---------------------------------------------------------------------------
 
-// handleListIntegrations lists a team's integrations. Never the keys: those
+// sourceQuietAfter is how long a source may go without posting before the
-// exist in plaintext only in the response that created them.
+// Sources page calls it quiet rather than active. A day is longer than any
+// repeat_interval worth having, so an Alertmanager that is up and has anything
+// firing never crosses it; a source with nothing firing may, and that is a
+// reason to look, not proof of a fault — which is why this is a colour and not
+// an alarm. Dead man's switches are where silence pages.
+const sourceQuietAfter = 24 * time.Hour
+
+const (
+	sourceActive = "active"
+	sourceQuiet  = "quiet"
+	sourceNever  = "never"
+)
+
+// integrationStatus is an integration as the Sources page shows it.
+type integrationStatus struct {
+	models.Integration
+
+	// Status is active when the key posted within sourceQuietAfter, quiet when
+	// it has posted but not lately, never when it has not posted at all.
+	Status string `json:"status"`
+
+	// LastAlertAt is when an alert last arrived on this source, which is not the
+	// same as when it last posted: a payload with nothing usable in it stamps
+	// last_used_at and not this. Absent until an alert has arrived since
+	// migration 010 started recording it.
+	LastAlertAt *time.Time `json:"last_alert_at,omitempty"`
+
+	// Alerts24h counts the distinct alerts this source refreshed in the last
+	// day. An alert re-sent every few hours counts once, not once per re-send.
+	Alerts24h int64 `json:"alerts_24h"`
+}
+
+// handleListIntegrations lists a team's integrations with what each has been
+// delivering. Never the keys: those exist in plaintext only in the response that
+// created them.
 func handleListIntegrations(db *sql.DB) http.HandlerFunc {
 	return func(w http.ResponseWriter, r *http.Request) {
 		teamID, ok := teamParam(w, r)
@@ -367,28 +401,45 @@ func handleListIntegrations(db *sql.DB) http.HandlerFunc {
 			return
 		}
 
+		now := time.Now()
 		rows, err := db.QueryContext(r.Context(), `
-			SELECT id, team_id, kind, name, created_at, last_used_at
+			SELECT i.id, i.team_id, i.kind, i.name, i.created_at, i.last_used_at,
-			FROM   integrations
+			       -- Scalar subqueries, not a join and GROUP BY: each is a
-			WHERE  team_id = $1
+			       -- single range over alerts_integration_idx, where the join
-			ORDER  BY id`, teamID)
+			       -- would read every alert a source ever delivered.
+			       (SELECT MAX(received_at) FROM alerts WHERE integration_id = i.id),
+			       (SELECT COUNT(*) FROM alerts
+			         WHERE integration_id = i.id AND received_at >= $2)
+			FROM   integrations i
+			WHERE  i.team_id = $1
+			ORDER  BY i.id`, teamID, now.Add(-sourceQuietAfter).Unix())
 		if err != nil {
 			respond(w, http.StatusInternalServerError, errResp("internal error"))
 			return
 		}
 		defer rows.Close()
 
-		integrations := []models.Integration{}
+		integrations := []integrationStatus{}
 		for rows.Next() {
-			var i models.Integration
+			var i integrationStatus
 			var created int64
-			var lastUsed *int64
+			var lastUsed, lastAlert *int64
-			if err := rows.Scan(&i.ID, &i.TeamID, &i.Kind, &i.Name, &created, &lastUsed); err != nil {
+			if err := rows.Scan(&i.ID, &i.TeamID, &i.Kind, &i.Name, &created, &lastUsed,
+				&lastAlert, &i.Alerts24h); err != nil {
 				respond(w, http.StatusInternalServerError, errResp("internal error"))
 				return
 			}
 			i.CreatedAt = time.Unix(created, 0).UTC()
 			i.LastUsedAt = unixPtr(lastUsed)
+			i.LastAlertAt = unixPtr(lastAlert)
+			switch {
+			case i.LastUsedAt == nil:
+				i.Status = sourceNever
+			case now.Sub(*i.LastUsedAt) > sourceQuietAfter:
+				i.Status = sourceQuiet
+			default:
+				i.Status = sourceActive
+			}
 			integrations = append(integrations, i)
 		}
 		if err := rows.Err(); err != nil {
@@ -457,6 +508,54 @@ func handleCreateIntegration(db *sql.DB, publicURL string) http.HandlerFunc {
 	}
 }
 
+// handleRenameIntegration renames a source. The key is untouched, so nothing
+// posting with it notices.
+func handleRenameIntegration(db *sql.DB) http.HandlerFunc {
+	return func(w http.ResponseWriter, r *http.Request) {
+		teamID, ok := teamParam(w, r)
+		if !ok {
+			return
+		}
+		if !requireTeamOwner(w, r, teamID) {
+			return
+		}
+		id, err := strconv.ParseInt(chi.URLParam(r, "integrationID"), 10, 64)
+		if err != nil {
+			respond(w, http.StatusBadRequest, errResp("invalid integration id"))
+			return
+		}
+
+		var req struct {
+			Name string `json:"name"`
+		}
+		if err := decodeJSON(r, &req); err != nil {
+			respond(w, http.StatusBadRequest, errResp("invalid request body"))
+			return
+		}
+		req.Name = strings.TrimSpace(req.Name)
+		if req.Name == "" {
+			respond(w, http.StatusBadRequest, errResp("name is required"))
+			return
+		}
+		if len(req.Name) > 100 {
+			respond(w, http.StatusBadRequest, errResp("name is too long"))
+			return
+		}
+
+		res, err := db.ExecContext(r.Context(),
+			"UPDATE integrations SET name = $1 WHERE id = $2 AND team_id = $3", req.Name, id, teamID)
+		if err != nil {
+			respond(w, http.StatusInternalServerError, errResp("internal error"))
+			return
+		}
+		if n, _ := res.RowsAffected(); n == 0 {
+			respond(w, http.StatusNotFound, errResp("not found"))
+			return
+		}
+		w.WriteHeader(http.StatusNoContent)
+	}
+}
+
 func handleDeleteIntegration(db *sql.DB) http.HandlerFunc {
 	return func(w http.ResponseWriter, r *http.Request) {
 		teamID, ok := teamParam(w, r)
@@ -493,24 +592,32 @@ func integrationPath(key, kind string) string {
 	return "/api/integrations/" + key + "/" + kind
 }
 
-// teamIDForKey resolves an integration key to its team, and stamps the key's
+// alertSource is who an arriving webhook is from: the integration whose key it
+// used, and the team that integration puts its alerts in.
+type alertSource struct {
+	integrationID int64
+	teamID        int64
+}
+
+// sourceForKey resolves an integration key to its source, and stamps the key's
 // last use. An unknown key is not an error worth distinguishing: the caller is
 // told nothing beyond "no".
-func teamIDForKey(ctx context.Context, db *sql.DB, key string) (int64, error) {
+func sourceForKey(ctx context.Context, db *sql.DB, key string) (alertSource, error) {
-	var teamID int64
+	var src alertSource
 	err := db.QueryRowContext(ctx,
-		"SELECT team_id FROM integrations WHERE key_hash = $1", hashToken(key)).Scan(&teamID)
+		"SELECT id, team_id FROM integrations WHERE key_hash = $1", hashToken(key)).
+		Scan(&src.integrationID, &src.teamID)
 	if errors.Is(err, sql.ErrNoRows) {
-		return 0, errUnknownIntegration
+		return alertSource{}, errUnknownIntegration
 	}
 	if err != nil {
-		return 0, err
+		return alertSource{}, err
 	}
 	// Best effort, like an API key's: a failed stamp must not reject an alert.
 	db.ExecContext(ctx, //nolint:errcheck
-		"UPDATE integrations SET last_used_at = $1 WHERE key_hash = $2",
+		"UPDATE integrations SET last_used_at = $1 WHERE id = $2",
-		time.Now().Unix(), hashToken(key))
+		time.Now().Unix(), src.integrationID)
-	return teamID, nil
+	return src, nil
 }
 
 var errUnknownIntegration = errors.New("unknown integration key")
@@ -539,17 +646,24 @@ func defaultTeamID(ctx context.Context, db *sql.DB) (int64, error) {
 // A team's dead man's switches
 // ---------------------------------------------------------------------------
 
-// deadmanResponse is the wire shape of a team's switch configuration. The
+// deadmanSwitchRequest is what creating a switch takes. The timeout is seconds,
-// timeout is seconds rather than a duration string, because that is what the
+// because that is what the column holds and what arithmetic is done on; a client
-// column holds and what arithmetic is done on; a client renders it.
+// renders it.
-type deadmanResponse struct {
+type deadmanSwitchRequest struct {
-	TeamID         int64  `json:"team_id"`
+	Name           string `json:"name"`
-	Matchers       string `json:"matchers"`
+	Matcher        string `json:"matcher"`
 	TimeoutSeconds int64  `json:"timeout_seconds"`
 	Severity       string `json:"severity"`
 }
 
-func handleGetTeamDeadman(db *sql.DB) http.HandlerFunc {
+// deadmanSeverities are the severities an incident can open at.
+var deadmanSeverities = map[string]bool{"critical": true, "error": true, "warning": true, "info": true}
+
+// handleListTeamDeadman lists a team's switches with what each one's heartbeats
+// are doing. A team with none gets an empty list, which is a configuration and
+// not an absence: answering 404 would make "off" indistinguishable from "this
+// server does not do this".
+func handleListTeamDeadman(db *sql.DB) http.HandlerFunc {
 	return func(w http.ResponseWriter, r *http.Request) {
 		teamID, ok := teamParam(w, r)
 		if !ok {
@@ -559,27 +673,26 @@ func handleGetTeamDeadman(db *sql.DB) http.HandlerFunc {
 			return
 		}
 
-		out := deadmanResponse{TeamID: teamID, Severity: "critical"}
+		set, err := deadmanSetForTeam(r.Context(), db, teamID)
-		err := db.QueryRowContext(r.Context(),
+		if err != nil {
-			"SELECT matchers, timeout_seconds, severity FROM deadman_configs WHERE team_id = $1",
+			respond(w, http.StatusInternalServerError, errResp("internal error"))
-			teamID).Scan(&out.Matchers, &out.TimeoutSeconds, &out.Severity)
+			return
-		if err != nil && !errors.Is(err, sql.ErrNoRows) {
+		}
+		out, err := deadmanStatuses(r.Context(), db, teamID, set, time.Now())
+		if err != nil {
 			respond(w, http.StatusInternalServerError, errResp("internal error"))
 			return
 		}
-		// A team with no row watches nothing, which is a configuration and not
-		// an absence: answering 404 would make "off" indistinguishable from
-		// "this server does not do this".
 		respond(w, http.StatusOK, out)
 	}
 }
 
-// handleSetTeamDeadman replaces a team's switch configuration.
+// handleCreateTeamDeadman adds one switch.
 //
-// Validated by parsing: a matcher string that survives ParseDeadmanConfig with
+// Validated by parsing: a matcher with no alertname is rejected rather than
-// nothing usable in it is rejected rather than stored, because a switch that
+// stored, because a switch that silently watches nothing is the failure this
-// silently watches nothing is the failure this feature exists to prevent.
+// feature exists to prevent.
-func handleSetTeamDeadman(db *sql.DB) http.HandlerFunc {
+func handleCreateTeamDeadman(db *sql.DB) http.HandlerFunc {
 	return func(w http.ResponseWriter, r *http.Request) {
 		teamID, ok := teamParam(w, r)
 		if !ok {
@@ -589,50 +702,87 @@ func handleSetTeamDeadman(db *sql.DB) http.HandlerFunc {
 			return
 		}
 
-		var req struct {
+		var req deadmanSwitchRequest
-			Matchers       string `json:"matchers"`
-			TimeoutSeconds int64  `json:"timeout_seconds"`
-			Severity       string `json:"severity"`
-		}
 		if err := decodeJSON(r, &req); err != nil {
 			respond(w, http.StatusBadRequest, errResp("invalid request body"))
 			return
 		}
-		req.Matchers = strings.TrimSpace(req.Matchers)
+		req.Matcher = strings.TrimSpace(req.Matcher)
+		req.Name = strings.TrimSpace(req.Name)
 		if req.Severity == "" {
 			req.Severity = "critical"
 		}
-		if req.TimeoutSeconds < 0 {
+		if !deadmanSeverities[req.Severity] {
-			respond(w, http.StatusBadRequest, errResp("timeout_seconds must not be negative"))
+			respond(w, http.StatusBadRequest, errResp("severity must be critical, error, warning or info"))
 			return
 		}
-		if req.Matchers != "" {
+		if req.TimeoutSeconds <= 0 {
-			parsed := parseDeadmanQuietly(req.Matchers, time.Duration(req.TimeoutSeconds)*time.Second, req.Severity)
+			respond(w, http.StatusBadRequest, errResp("timeout_seconds must be positive"))
-			if len(parsed.Matchers) == 0 {
+			return
+		}
+		if strings.Contains(req.Matcher, ";") {
+			respond(w, http.StatusBadRequest, errResp("one matcher per switch: add another switch instead of separating with ;"))
+			return
+		}
+		m, err := parseDeadmanMatcher(req.Matcher)
+		if err != nil {
 			respond(w, http.StatusBadRequest, errResp(
-					"no usable matchers: each must name an alertname, as in alertname=Watchdog,cluster=prod"))
+				"unusable matcher ("+err.Error()+"): each must name an alertname, as in alertname=Watchdog,cluster=prod"))
 			return
 		}
+		if req.Name == "" {
+			req.Name = m.config()
+		}
+		if len(req.Name) > 100 {
+			respond(w, http.StatusBadRequest, errResp("name is too long"))
+			return
 		}
 
-		if _, err := db.ExecContext(r.Context(), `
+		var id int64
-			INSERT INTO deadman_configs (team_id, matchers, timeout_seconds, severity, updated_at)
+		if err := db.QueryRowContext(r.Context(), `
-			VALUES ($1, $2, $3, $4, `+nowEpoch+`)
+			INSERT INTO deadman_switches (team_id, name, matcher, timeout_seconds, severity)
-			ON CONFLICT (team_id) DO UPDATE SET
+			VALUES ($1, $2, $3, $4, $5) RETURNING id`,
-				matchers        = excluded.matchers,
+			teamID, req.Name, m.config(), req.TimeoutSeconds, req.Severity).Scan(&id); err != nil {
-				timeout_seconds = excluded.timeout_seconds,
-				severity        = excluded.severity,
-				updated_at      = excluded.updated_at`,
-			teamID, req.Matchers, req.TimeoutSeconds, req.Severity); err != nil {
 			respond(w, http.StatusInternalServerError, errResp("internal error"))
 			return
 		}
 
-		respond(w, http.StatusOK, deadmanResponse{
+		respond(w, http.StatusCreated, deadmanSwitchStatus{
-			TeamID:         teamID,
+			ID: id, Name: req.Name, Matcher: m.config(),
-			Matchers:       req.Matchers,
+			TimeoutSeconds: req.TimeoutSeconds, Severity: req.Severity,
-			TimeoutSeconds: req.TimeoutSeconds,
+			Status: switchDormant, Sources: []deadmanSource{},
-			Severity:       req.Severity,
 		})
 	}
 }
+
+// handleDeleteTeamDeadman removes a switch. An incident it already opened stays
+// open until somebody resolves it: deleting the switch says "stop watching", not
+// "the problem is gone".
+func handleDeleteTeamDeadman(db *sql.DB) http.HandlerFunc {
+	return func(w http.ResponseWriter, r *http.Request) {
+		teamID, ok := teamParam(w, r)
+		if !ok {
+			return
+		}
+		if !requireTeamOwner(w, r, teamID) {
+			return
+		}
+		switchID, err := strconv.ParseInt(chi.URLParam(r, "switchID"), 10, 64)
+		if err != nil {
+			respond(w, http.StatusBadRequest, errResp("invalid switch id"))
+			return
+		}
+
+		res, err := db.ExecContext(r.Context(),
+			"DELETE FROM deadman_switches WHERE id = $1 AND team_id = $2", switchID, teamID)
+		if err != nil {
+			respond(w, http.StatusInternalServerError, errResp("internal error"))
+			return
+		}
+		if n, _ := res.RowsAffected(); n == 0 {
+			respond(w, http.StatusNotFound, errResp("switch not found"))
+			return
+		}
+		w.WriteHeader(http.StatusNoContent)
+	}
+}

internal/db/db.go

@@ -5,6 +5,7 @@ import (
 	"embed"
 	"fmt"
 	"io/fs"
+	"log"
 	"sort"
 	"strings"
 	"time"
@@ -15,6 +16,19 @@ import (
 //go:embed migrations
 var migrationsFS embed.FS
 
+// pingAttempts and pingRetryDelay bound the retry on the first connection.
+// This pod's own IP can reach the Postgres pod's node before that node's
+// NetworkPolicy enforcement (kube-router, reacting to the pod's creation
+// event) has added it to the allowed-source set, which fails the ping with
+// "connection refused" rather than a timeout. That race resolves within
+// several seconds in practice; five attempts two seconds apart give it
+// comfortable room without turning a genuinely absent database into a long
+// hang.
+const (
+	pingAttempts   = 5
+	pingRetryDelay = 2 * time.Second
+)
+
 // Open connects to Postgres. dsn is a libpq connection string or URL, e.g.
 // postgres://terdut:secret@localhost:5432/terdut?sslmode=disable.
 //
@@ -33,11 +47,19 @@ func Open(dsn string) (*sql.DB, error) {
 	db.SetMaxOpenConns(10)
 	db.SetMaxIdleConns(5)
 	db.SetConnMaxLifetime(time.Hour)
-	if err := db.Ping(); err != nil {
+
+	for attempt := 1; ; attempt++ {
+		err = db.Ping()
+		if err == nil {
+			return db, nil
+		}
+		if attempt == pingAttempts {
 			db.Close()
 			return nil, fmt.Errorf("ping: %w", err)
 		}
-	return db, nil
+		log.Printf("open db: ping attempt %d/%d failed, retrying in %s: %v", attempt, pingAttempts, pingRetryDelay, err)
+		time.Sleep(pingRetryDelay)
+	}
 }
 
 // Migrate applies every embedded migration that has not been applied yet, in

internal/db/migrations/008_incident_signature.sql

@@ -0,0 +1,23 @@
+-- Similar incidents: a signature per incident, so "has this happened before"
+-- is an indexed equality instead of a search.
+--
+-- The signature is the alert name plus the group labels that identify WHAT is
+-- broken, minus the ones that only say WHERE it happened to run this time
+-- (instance, pod, ...). Two incidents with the same signature in the same team
+-- are the same problem for a responder's purposes.
+--
+-- Computed in Go for new incidents (incidentSignature in incident_store.go).
+-- The backfill below MUST produce the same string; keep the volatile list in
+-- both places in step.
+ALTER TABLE incidents ADD COLUMN signature TEXT NOT NULL DEFAULT '';
+
+UPDATE incidents SET signature =
+    COALESCE(NULLIF(group_labels->>'alertname', ''), title) || '|' ||
+    COALESCE((
+        SELECT string_agg(e.k || '=' || e.v, ',' ORDER BY e.k)
+          FROM jsonb_each_text(incidents.group_labels) AS e(k, v)
+         WHERE e.k <> 'alertname'
+           AND e.k NOT IN ('instance', 'pod', 'pod_name', 'pod_ip', 'container', 'container_name', 'endpoint')
+    ), '');
+
+CREATE INDEX incidents_signature_idx ON incidents(team_id, signature, triggered_at DESC);

internal/db/migrations/009_deadman_switches.sql

@@ -0,0 +1,54 @@
+-- Dead man's switches become rows of their own.
+--
+-- 004 kept a team's switches in one string with one timeout and one severity,
+-- which was enough to configure them and not enough to show them: there was no
+-- thing to list, nothing to hang a status on, and every switch in a team had to
+-- share a deadline. A row per switch gives each its own name, matcher, timeout
+-- and severity, and gives the Team → Switches page something to be a list of.
+--
+-- The matcher keeps the syntax the string used, one matcher per row:
+-- `alertname=Watchdog,cluster=prod`. The unit of monitoring is still the
+-- fingerprint, so a matcher that many clusters satisfy is still one switch row
+-- watching several independent heartbeats.
+CREATE TABLE deadman_switches (
+    id      BIGSERIAL PRIMARY KEY,
+    team_id BIGINT NOT NULL REFERENCES teams(id) ON DELETE CASCADE,
+
+    -- What the owner calls it. Defaults to the matcher when they do not say.
+    name TEXT NOT NULL,
+
+    -- "," separates the label conditions, "=" is exact equality, and alertname is
+    -- mandatory: it is what keeps the sweeper's candidate query on an index.
+    matcher TEXT NOT NULL,
+
+    -- Seconds of silence before the switch is declared dead. Never zero: a switch
+    -- that cannot fire is deleted, not disabled.
+    timeout_seconds BIGINT NOT NULL CHECK (timeout_seconds > 0),
+
+    -- The severity its incidents open at. See 004 for why they carry their own.
+    severity TEXT NOT NULL DEFAULT 'critical',
+
+    created_at BIGINT NOT NULL DEFAULT FLOOR(EXTRACT(EPOCH FROM now()))::bigint
+);
+
+CREATE INDEX deadman_switches_team_idx ON deadman_switches (team_id);
+
+-- Carry every team's configuration over, one row per matcher. A team whose
+-- timeout was zero had switches turned off, which is now "no rows".
+INSERT INTO deadman_switches (team_id, name, matcher, timeout_seconds, severity)
+SELECT c.team_id, btrim(m), btrim(m), c.timeout_seconds, c.severity
+FROM   deadman_configs c,
+       LATERAL regexp_split_to_table(c.matchers, ';') AS m
+WHERE  c.timeout_seconds > 0
+  AND  btrim(m) <> ''
+ORDER  BY c.team_id;
+
+-- The server seeds environment defaults into teams once, and remembers that it
+-- did. An install that had a row per team was already seeded; without this
+-- marker the first start after upgrading would seed teams that had switched
+-- theirs off.
+INSERT INTO settings (key, value)
+SELECT 'deadman_seeded', '1'
+WHERE  EXISTS (SELECT 1 FROM deadman_configs);
+
+DROP TABLE deadman_configs;

internal/db/migrations/010_alert_source.sql

@@ -0,0 +1,21 @@
+-- Which alert source an alert last arrived on.
+--
+-- Team -> Sources shows when each source last posted, which integrations
+-- already knew (last_used_at, stamped on every webhook). What it could not say
+-- was what a source delivered: an alert never recorded the key it came in on, so
+-- "prod alertmanager" and "staging alertmanager" were indistinguishable once
+-- inside. This column is that link, and lets the page show each source's last
+-- alert and how many alerts it has kept fresh over the past day.
+--
+-- Last sender wins: every accepted payload restamps it, the way it advances
+-- received_at. Two sources posting the same fingerprint into one team is
+-- already one alert, and it is attributed to whichever spoke last.
+--
+-- Nullable, and not backfilled. Alerts that arrived before this migration have
+-- no source, and NULL says so honestly rather than guessing. It heals by itself:
+-- Alertmanager re-sends every alert each repeat_interval, and each re-send is an
+-- accepted payload. Deleting a source keeps its alerts, unattributed.
+ALTER TABLE alerts ADD COLUMN integration_id BIGINT REFERENCES integrations(id) ON DELETE SET NULL;
+
+CREATE INDEX alerts_integration_idx ON alerts (integration_id, received_at)
+    WHERE integration_id IS NOT NULL;

internal/models/incident.go

@@ -79,3 +79,15 @@ type IncidentEvent struct {
 	Detail     *string   `json:"detail,omitempty"`
 	CreatedAt  time.Time `json:"created_at"`
 }
+
+// SimilarIncident is an earlier, resolved incident with the same signature as
+// the one being looked at. ResolutionNotes are the "what fixed it" notes;
+// NoteCount counts the plain working notes, which live on the timeline.
+type SimilarIncident struct {
+	ID              int64           `json:"id"`
+	Title           string          `json:"title"`
+	TriggeredAt     time.Time       `json:"triggered_at"`
+	ResolvedAt      time.Time       `json:"resolved_at"`
+	NoteCount       int             `json:"note_count"`
+	ResolutionNotes []IncidentEvent `json:"resolution_notes"`
+}

internal/web/incident_test.go

@@ -0,0 +1,26 @@
+package web
+
+import (
+	"io/fs"
+	"strings"
+	"testing"
+)
+
+func TestCopyIncidentIsEmbedded(t *testing.T) {
+	sub, err := fs.Sub(files, "static")
+	if err != nil {
+		t.Fatal(err)
+	}
+	for file, want := range map[string]string{
+		"js/incident.js": "copyIncident",
+		"js/ui.js":       "copy:",
+	} {
+		b, err := fs.ReadFile(sub, file)
+		if err != nil {
+			t.Fatal(err)
+		}
+		if !strings.Contains(string(b), want) {
+			t.Errorf("%s lacks %s", file, want)
+		}
+	}
+}

internal/web/static/app.css

@@ -48,7 +48,11 @@
   --mono: ui-monospace, SFMono-Regular, Menlo, Consolas, monospace;
 
   --topbar-h: 52px;
-  --tabbar-h: 58px;
+  /* No bottom tab bar on any breakpoint any more — mobile uses the hamburger
+     menu in the topbar, desktop the sidebar — so this stays 0. Kept as a
+     variable rather than deleted since .view, .toast and .nav's own height
+     calc still read it. */
+  --tabbar-h: 0px;
   --safe-top: env(safe-area-inset-top, 0px);
   --safe-bottom: env(safe-area-inset-bottom, 0px);
 }
@@ -196,7 +200,13 @@ input:focus, textarea:focus { outline: none; border-color: var(--accent); box-sh
   -webkit-backdrop-filter: saturate(1.4) blur(12px);
   border-bottom: 1px solid var(--border);
 }
-.topbar-title { font-size: 18px; font-weight: 700; letter-spacing: -0.01em; }
+.topbar-left { display: flex; align-items: center; gap: 8px; min-width: 0; }
+.topbar-title {
+  font-size: 18px; font-weight: 700; letter-spacing: -0.01em;
+  overflow: hidden; text-overflow: ellipsis; white-space: nowrap; min-width: 0;
+}
+#menu-btn { position: relative; }
+.nav-badge.menu-btn-badge { top: 2px; left: auto; right: 2px; }
 
 .open-pill {
   display: inline-flex; align-items: center; gap: 6px;
@@ -209,15 +219,12 @@ input:focus, textarea:focus { outline: none; border-color: var(--accent); box-sh
 .open-pill.has-triggered::before { background: var(--crit); }
 .open-pill.all-acked::before { background: var(--warn); }
 
-/* Bottom tab bar on phones. */
+/* Hidden on phones — mobile navigates through the hamburger menu in the
+   topbar instead (see #menu-btn / openNavMenu in app.js). Reappears as the
+   left sidebar from 900px, where the desktop block below redeclares display. */
 .nav {
+  display: none;
   position: fixed; left: 0; right: 0; bottom: 0; z-index: 20;
-  /* One column per link, however many there are. This was repeat(4, 1fr) when
-     there were four tabs; Team and Admin arriving pushed six items into four
-     columns, which on a phone is how they stopped fitting. Auto columns mean
-     the next tab cannot break the row either — and Admin is only rendered for
-     an administrator, so the count genuinely varies between viewers. */
-  display: grid; grid-auto-flow: column; grid-auto-columns: 1fr;
   height: calc(var(--tabbar-h) + var(--safe-bottom));
   padding-bottom: var(--safe-bottom);
   background: color-mix(in srgb, var(--surface) 92%, transparent);
@@ -239,16 +246,6 @@ input:focus, textarea:focus { outline: none; border-color: var(--accent); box-sh
 }
 .nav-link svg { width: 24px; height: 24px; flex: none; fill: none; stroke: currentColor; stroke-width: 1.8; stroke-linecap: round; stroke-linejoin: round; }
 
-/* Narrow phones, where six tabs each get about 55-65px. Tightening is enough:
-   the widest label, "On-call", is about 38px at this size, so nothing has to
-   be hidden. The ellipsis above is the backstop if a future tab is named
-   something longer. */
-@media (max-width: 420px) {
-  .nav-link { font-size: 10px; gap: 1px; }
-  .nav-link svg { width: 21px; height: 21px; }
-  .nav-badge { left: calc(50% + 4px); min-width: 16px; height: 16px; font-size: 10px; line-height: 16px; }
-}
-
 .nav-link[aria-current="page"] { color: var(--accent); }
 .nav-badge {
   position: absolute; top: 6px; left: calc(50% + 6px);
@@ -362,7 +359,13 @@ input:focus, textarea:focus { outline: none; border-color: var(--accent); box-sh
 .badge.st-triggered, .badge.st-firing { background: var(--crit-soft); color: var(--crit); }
 .badge.st-acknowledged { background: var(--warn-soft); color: var(--warn); }
 .badge.st-snoozed { background: var(--snooze-soft); color: var(--snooze); }
-.badge.st-resolved { background: var(--ok-soft); color: var(--ok); }
+.badge.st-resolved, .badge.st-healthy { background: var(--ok-soft); color: var(--ok); }
+.badge.st-dead { background: var(--crit-soft); color: var(--crit); }
+/* Dormant is the plain badge on purpose: nothing has gone wrong and nothing has
+   gone right, which is what the muted default already says. */
+.badge.st-dormant, .badge.st-never { background: var(--surface-2); color: var(--muted); }
+.badge.st-active { background: var(--ok-soft); color: var(--ok); }
+.badge.st-quiet { background: var(--warn-soft); color: var(--warn); }
 .badge.sev-critical { background: var(--crit-soft); color: var(--crit); }
 .badge.sev-warning { background: var(--warn-soft); color: var(--warn); }
 .badge.sev-info { background: var(--info-soft); color: var(--info); }
@@ -380,6 +383,8 @@ input:focus, textarea:focus { outline: none; border-color: var(--accent); box-sh
   -webkit-backdrop-filter: saturate(1.4) blur(12px);
   border-bottom: 1px solid var(--border);
 }
+.detail-head .copy { margin-left: auto; }
+.clip-buffer { position: fixed; top: 0; left: 0; opacity: 0; pointer-events: none; }
 .detail-head .crumb { font-weight: 600; color: var(--muted); font-size: 14px; }
 .detail-title { font-size: 21px; font-weight: 750; letter-spacing: -0.01em; margin: 16px 0 8px; overflow-wrap: anywhere; }
 .detail-badges { display: flex; flex-wrap: wrap; gap: 6px; margin-bottom: 14px; }
@@ -453,11 +458,16 @@ details[open] > summary { margin-bottom: 8px; }
 .tl-text { overflow-wrap: anywhere; }
 .tl-text .who { font-weight: 650; }
 .tl-time { color: var(--faint); font-size: 12px; }
-.tl-note .note {
+.note {
   margin-top: 6px; padding: 10px 12px;
   background: var(--surface-2); border-radius: var(--radius-sm);
   white-space: pre-wrap; overflow-wrap: anywhere;
 }
+.note-fix { background: var(--ok-soft); border-left: 3px solid var(--ok); }
+.similar { list-style: none; margin: 0; padding: 0; }
+.similar-item { padding: 10px 0; }
+.similar-item + .similar-item { border-top: 1px solid var(--border, var(--surface-2)); }
+.check { display: flex; align-items: center; gap: 8px; font-size: 14px; color: var(--muted); }
 .note-actions { display: flex; justify-content: flex-end; }
 .note-actions .btn { color: var(--muted); }
 
@@ -601,8 +611,6 @@ kbd {
 /* ---------- desktop ---------- */
 
 @media (min-width: 900px) {
-  :root { --tabbar-h: 0px; }
-
   .app { display: grid; grid-template-columns: 220px 1fr; height: 100dvh; }
 
   .nav {
@@ -638,6 +646,10 @@ kbd {
   .view-queue .pane { overflow: auto; height: 100dvh; }
   .pane-list { border-right: 1px solid var(--border); }
   .pane-list .chips { position: sticky; top: 0; z-index: 2; background: var(--bg); padding-top: 16px; }
+  /* The pane is 340-420px wide and a mouse cannot scroll a row whose scrollbar
+     is hidden, so the chips wrap here instead: Archived stays reachable. */
+  .pane-list .chips { flex-wrap: wrap; overflow-x: visible; }
+  .pane-list .chip-sep { display: none; }
   .view-queue:not(.has-detail) .pane-detail { display: block; }
 
   /* On desktop the list stays visible next to the detail. */
@@ -685,6 +697,18 @@ kbd {
 .disabled-row td { opacity: 0.55; }
 .btn-sm.danger { color: var(--crit); border-color: var(--crit-soft); }
 
+/* --- status lists: alert sources and dead man's switches -----------------
+   Six columns do not fit a phone, so the table scrolls inside its card rather
+   than the page. A heartbeat under a switch with several is indented, the way
+   the escalation ladder indents its levels. */
+.card-head { display: flex; align-items: center; justify-content: space-between; gap: 12px; flex-wrap: wrap; }
+.table-scroll { overflow-x: auto; margin-top: 12px; }
+.status-table th, .status-table td { white-space: nowrap; }
+.status-table td:nth-child(2) { white-space: normal; min-width: 12em; }
+.status-table .source-row td { border-bottom-style: dashed; }
+.status-table .source-row td:first-child { padding-left: 16px; }
+.source-labels { display: flex; flex-wrap: wrap; gap: 4px; align-items: center; }
+
 .inline-form { display: flex; gap: 8px; margin-top: 12px; }
 .inline-form input { flex: 1; min-width: 0; }
 
@@ -872,3 +896,40 @@ button.rota-day:hover { background: var(--surface-2); }
 .overview-head { display: flex; align-items: baseline; gap: 8px; }
 .overview-count { margin-left: auto; color: var(--muted); font-size: 18px; font-weight: 700; }
 .overview-item p { margin: 4px 0 0; }
+
+/* ---------- stats page ---------- */
+
+#view-stats .chips { padding-left: 0; padding-right: 0; }
+.stats { display: grid; gap: 14px; padding-bottom: 16px; }
+.stat-tiles { display: grid; grid-template-columns: repeat(2, 1fr); gap: 8px; }
+.stat-tile {
+  --sev: var(--border-strong);
+  background: var(--surface); border: 1px solid var(--border);
+  border-left: 3px solid var(--sev); border-radius: var(--radius);
+  box-shadow: var(--shadow); padding: 12px;
+}
+.stat-tile.st-triggered { --sev: var(--crit); }
+.stat-tile.st-acknowledged { --sev: var(--warn); }
+.stat-tile.st-resolved { --sev: var(--ok); }
+.stat-value { font-size: 24px; font-weight: 700; font-variant-numeric: tabular-nums; }
+.stat-label { margin-top: 2px; font-size: 12px; color: var(--muted); }
+.chart-card + .chart-card { margin-top: 0; }
+.chart-title {
+  margin-bottom: 10px; font-size: 13px; font-weight: 700;
+  text-transform: uppercase; letter-spacing: 0.06em; color: var(--muted);
+}
+.hbars { list-style: none; display: grid; gap: 6px; }
+.hbar { display: grid; grid-template-columns: minmax(80px, 34%) 1fr auto; align-items: center; gap: 8px; font-size: 13px; }
+.hbar-name { overflow: hidden; text-overflow: ellipsis; white-space: nowrap; }
+.hbar-track { height: 10px; border-radius: 5px; background: var(--surface-2); overflow: hidden; }
+.hbar-fill { display: block; height: 100%; border-radius: 5px; background: var(--ok); }
+.hbar-count { min-width: 2ch; text-align: right; color: var(--muted); font-variant-numeric: tabular-nums; }
+.columns { display: block; width: 100%; height: auto; }
+.columns .axis { stroke: var(--border-strong); stroke-width: 1; }
+.columns .col-hit { fill: transparent; }
+.columns .col-bar { fill: var(--accent); }
+.columns .col:hover .col-bar { opacity: 0.75; }
+.columns .col-label { fill: var(--faint); font-size: 10px; font-family: var(--font); }
+@media (min-width: 900px) {
+  .stat-tiles { grid-template-columns: repeat(3, 1fr); }
+}

internal/web/static/index.html

@@ -90,6 +90,10 @@
         <svg viewBox="0 0 24 24" aria-hidden="true"><path d="M6 16V11a6 6 0 0 1 12 0v5l1.5 2h-15z"/><path d="M10 20.5a2 2 0 0 0 4 0"/></svg>
         <span class="nav-label">Alerts</span>
       </a>
+      <a class="nav-link" href="/stats" data-section="stats" aria-label="Stats">
+        <svg viewBox="0 0 24 24" aria-hidden="true"><path d="M4 20h16M7 20v-7M12 20V6M17 20v-10"/></svg>
+        <span class="nav-label">Stats</span>
+      </a>
       <a class="nav-link" href="/team" data-section="team" aria-label="Team">
         <svg viewBox="0 0 24 24" aria-hidden="true"><circle cx="9" cy="8" r="3"/><circle cx="17" cy="9" r="2.5"/><path d="M3 19a6 6 0 0 1 12 0M15 19a5 5 0 0 1 6-4"/></svg>
         <span class="nav-label">Team</span>
@@ -108,7 +112,13 @@
     </nav>
 
     <header class="topbar">
+      <div class="topbar-left">
+        <button class="btn btn-ghost btn-icon" id="menu-btn" type="button" aria-label="Menu" aria-haspopup="menu">
+          <svg viewBox="0 0 24 24" aria-hidden="true"><path d="M4 7h16M4 12h16M4 17h16"/></svg>
+          <span class="nav-badge menu-btn-badge" data-badge hidden></span>
+        </button>
         <h1 class="topbar-title" id="topbar-title">Queue</h1>
+      </div>
       <span class="open-pill" id="open-pill" hidden></span>
     </header>
 
@@ -122,6 +132,7 @@
 
     <section id="view-oncall" class="view view-page" data-view="oncall" hidden></section>
     <section id="view-alerts" class="view view-page" data-view="alerts" hidden></section>
+    <section id="view-stats" class="view view-page" data-view="stats" hidden></section>
     <section id="view-team" class="view view-page" data-view="team" hidden></section>
     <section id="view-admin" class="view view-page" data-view="admin" hidden></section>
     <!-- One person, at /admin/users/{id}: reached from the Admin tab's user

internal/web/static/js/account.js

@@ -35,7 +35,6 @@ function render() {
 
     h('div', { class: 'page-head' }),
     h('button', { class: 'btn btn-block', type: 'button', onclick: signOut }, icon('logout'), 'Sign out'),
-    h('p', { class: 'foot-note', text: 'Statistics are in terdut-tui for now.' }),
   );
 }
 

internal/web/static/js/api.js

@@ -83,15 +83,22 @@ export const timeline = (id) => call('GET', `/incidents/${id}/timeline`);
 
 export const acknowledge = (id) => call('POST', `/incidents/${id}/acknowledge`);
 export const unacknowledge = (id) => call('DELETE', `/incidents/${id}/acknowledge`);
-export const resolve = (id) => call('POST', `/incidents/${id}/resolve`);
+export const resolve = (id, resolution) => call('POST', `/incidents/${id}/resolve`, resolution ? { body: { resolution } } : {});
 export const assign = (id, userID) => call('POST', `/incidents/${id}/assign`, { body: { user_id: userID } });
 export const snooze = (id, spec) => call('POST', `/incidents/${id}/snooze`, { body: spec });
 export const unsnooze = (id) => call('DELETE', `/incidents/${id}/snooze`);
 export const archive = (id) => call('POST', `/incidents/${id}/archive`);
 export const unarchive = (id) => call('DELETE', `/incidents/${id}/archive`);
-export const addNote = (id, content) => call('POST', `/incidents/${id}/notes`, { body: { content } });
+export const addNote = (id, content, pinned = false) => call('POST', `/incidents/${id}/notes`, { body: { content, pinned } });
+export const similar = (id) => call('GET', `/incidents/${id}/similar`);
 export const deleteNote = (id, eventID) => call('DELETE', `/incidents/${id}/notes/${eventID}`);
 
+// stats
+export const statsIncidents = (query) => call('GET', '/stats/incidents', { query });
+export const statsTop = (query) => call('GET', '/stats/alerts/top', { query });
+export const statsByHour = (query) => call('GET', '/stats/alerts/by-hour', { query });
+export const statsByDay = (query) => call('GET', '/stats/alerts/by-day', { query });
+
 // alerts
 export const alerts = (query, opts) => call('GET', '/alerts', { query, ...opts });
 
@@ -126,11 +133,16 @@ export const removeTeamMember = (id, userID) => call('DELETE', `/teams/${id}/mem
 export const integrations = (id) => call('GET', `/teams/${id}/integrations`);
 export const createIntegration = (id, name) =>
   call('POST', `/teams/${id}/integrations`, { body: { name } });
+export const renameIntegration = (id, integrationID, name) =>
+  call('PATCH', `/teams/${id}/integrations/${integrationID}`, { body: { name } });
 export const deleteIntegration = (id, integrationID) =>
   call('DELETE', `/teams/${id}/integrations/${integrationID}`);
 
-export const deadman = (id) => call('GET', `/teams/${id}/deadman`);
+export const deadmanSwitches = (id) => call('GET', `/teams/${id}/deadman/switches`);
-export const setDeadman = (id, body) => call('PUT', `/teams/${id}/deadman`, { body });
+export const createDeadmanSwitch = (id, body) =>
+  call('POST', `/teams/${id}/deadman/switches`, { body });
+export const deleteDeadmanSwitch = (id, switchID) =>
+  call('DELETE', `/teams/${id}/deadman/switches/${switchID}`);
 
 export const escalation = (id) => call('GET', `/teams/${id}/escalation`);
 export const setEscalation = (id, body) => call('PUT', `/teams/${id}/escalation`, { body });

internal/web/static/js/app.js

@@ -8,6 +8,7 @@ import * as queue from './queue.js';
 import * as incident from './incident.js';
 import * as oncall from './oncall.js';
 import * as alerts from './alerts.js';
+import * as stats from './stats.js';
 import * as account from './account.js';
 import * as team from './team.js';
 import * as admin from './admin.js';
@@ -23,6 +24,7 @@ const SECTIONS = {
   queue: { title: 'Queue', view: queue },
   oncall: { title: 'On-call', view: oncall },
   alerts: { title: 'Alerts', view: alerts },
+  stats: { title: 'Stats', view: stats },
   team: { title: 'Team', view: team },
   admin: { title: 'Admin', view: admin },
   adminuser: { title: 'User', view: adminuser, nav: 'admin' },
@@ -30,6 +32,18 @@ const SECTIONS = {
   more: { title: 'Account', view: account },
 };
 
+// The mobile hamburger menu's contents — the same sections the desktop
+// sidebar's .nav-link list carries in index.html, in the same order.
+const NAV_ITEMS = [
+  { path: '/', section: 'queue', label: 'Queue', icon: 'queueList' },
+  { path: '/oncall', section: 'oncall', label: 'On-call', icon: 'calendar' },
+  { path: '/alerts', section: 'alerts', label: 'Alerts', icon: 'bell' },
+  { path: '/stats', section: 'stats', label: 'Stats', icon: 'chart' },
+  { path: '/team', section: 'team', label: 'Team', icon: 'team' },
+  { path: '/admin', section: 'admin', label: 'Admin', icon: 'shield', adminOnly: true },
+  { path: '/more', section: 'more', label: 'Account', icon: 'user' },
+];
+
 function parseRoute(pathname) {
   const m = pathname.match(/^\/incidents\/(\d+)\/?$/);
   if (m) return { section: 'queue', incident: Number(m[1]) };
@@ -47,7 +61,7 @@ function parseRoute(pathname) {
   if (t) return { section: 'admin', tab: t.tab };
   const tt = team.TABS.find((x) => x.path === `/${name}`);
   if (tt) return { section: 'team', tab: tt.tab };
-  if (name === 'oncall' || name === 'alerts' || name === 'more') return { section: name };
+  if (name === 'oncall' || name === 'alerts' || name === 'stats' || name === 'more') return { section: name };
   return { section: 'queue', incident: null };
 }
 
@@ -132,6 +146,35 @@ function render() {
   updateTitle();
 }
 
+// ---------- nav menu ----------
+
+// The mobile hamburger menu: same shape as the sheet-based action menus in
+// incident.js (openSheet + a <ul class="menu"> of menu-item buttons), one
+// item per NAV_ITEMS entry, resolving with a path for navigate() to use.
+function openNavMenu() {
+  const current = SECTIONS[route.section].nav || route.section;
+  const triggered = state.open.filter((i) => i.status === 'triggered').length;
+  const items = NAV_ITEMS.filter((n) => !n.adminOnly || state.me?.user?.is_admin);
+  ui.openSheet(() => [
+    ui.h('h2', { class: 'sheet-title', text: 'Sections' }),
+    ui.h('ul', { class: 'menu', role: 'menu' }, items.map((n) =>
+      ui.h('li', {}, ui.h('button', {
+        class: 'menu-item',
+        type: 'button',
+        role: 'menuitemradio',
+        'aria-checked': String(n.section === current),
+        onclick: () => ui.closeSheet(n.path),
+      },
+        ui.icon(n.icon),
+        n.label,
+        n.section === 'queue' && triggered > 0 && ui.badge(String(triggered), 'st-triggered menu-sub'),
+      ))),
+    ),
+  ]).then((path) => {
+    if (path) navigate(path);
+  });
+}
+
 // ---------- refresh + badges ----------
 
 async function refresh() {
@@ -158,9 +201,12 @@ function updateBadges() {
   pill.classList.toggle('has-triggered', triggered > 0);
   pill.classList.toggle('all-acked', open > 0 && triggered === 0);
 
-  const badge = document.querySelector('[data-badge]');
+  // Two badges carry this count: the sidebar's Queue tab (desktop) and the
+  // hamburger button (mobile) — only one of the two is ever visible at once.
+  for (const badge of document.querySelectorAll('[data-badge]')) {
     badge.hidden = triggered === 0;
     badge.textContent = String(triggered);
+  }
   updateTitle();
 }
 
@@ -180,6 +226,7 @@ async function boot() {
   document.addEventListener('keydown', onKey);
   $('login-form').addEventListener('submit', onLogin);
   $('signup-form').addEventListener('submit', onSignup);
+  $('menu-btn').addEventListener('click', openNavMenu);
 
   // /signup is the one route that works without a session.
   if (location.pathname.replace(/\/$/, '') === '/signup') {

internal/web/static/js/incident.js

@@ -17,6 +17,7 @@ const pane = () => document.getElementById('detail');
 let currentID = null;
 let inc = null;
 let events = [];
+let similarList = [];
 let error = null;
 let busy = false;
 
@@ -38,10 +39,15 @@ export async function refresh() {
   const id = currentID;
   if (id == null) return;
   try {
-    const [i, t] = await Promise.all([api.incident(id), api.timeline(id)]);
+    // Similar incidents are a courtesy: an older server answers 404 and a
+    // failure here must not hide the incident itself.
+    const [i, t, sim] = await Promise.all([
+      api.incident(id), api.timeline(id), api.similar(id).catch(() => []),
+    ]);
     if (id !== currentID) return;
     inc = i;
     events = t;
+    similarList = sim;
     error = null;
   } catch (err) {
     if (id !== currentID) return;
@@ -60,6 +66,8 @@ function render() {
     h('button', { class: 'btn btn-ghost btn-icon back', type: 'button', 'aria-label': 'Back to queue', onclick: back },
       icon('back')),
     h('span', { class: 'crumb', text: currentID != null ? `Incident #${currentID}` : '' }),
+    inc && h('button', { class: 'btn btn-ghost btn-icon copy', type: 'button', 'aria-label': 'Copy incident', title: 'Copy incident (y)', onclick: copyIncident },
+      icon('copy')),
   );
 
   if (!inc) {
@@ -81,6 +89,7 @@ function render() {
       facts(),
       groupLabels(),
       alertsSection(),
+      similarSection(),
       timelineSection(),
     ),
     actionBar(),
@@ -179,8 +188,9 @@ function alertItem(a) {
 
 // ---------- timeline ----------
 
-function eventText(ev) {
+// named spells users out instead of "you", for text that leaves this page.
-  const person = ev.user_id != null ? who(ev.user_id, ev.username) : null;
+function eventText(ev, named = false) {
+  const person = ev.user_id != null ? (named ? ev.username || 'someone' : who(ev.user_id, ev.username)) : null;
   const strong = (t) => h('span', { class: 'who', text: t || 'someone' });
   const alertName = () => {
     const a = (inc.alerts || []).find((x) => x.id === ev.alert_id);
@@ -197,6 +207,7 @@ function eventText(ev) {
     case 'unsnoozed': return [strong(person), ' ended the snooze'];
     case 'resolved': return person ? [strong(person), ' resolved the incident'] : ['Resolved: every alert stopped firing'];
     case 'note': return [strong(person), ' added a note'];
+    case 'resolution_note': return [strong(person), ' noted what fixed it'];
     case 'notified': {
       const to = person ? strong(person) : 'the fallback topic';
       if (ev.detail === 'reminder') return ['Reminder sent to ', to];
@@ -209,6 +220,22 @@ function eventText(ev) {
   }
 }
 
+// Earlier incidents with the same signature that someone left notes on, the
+// ones that recorded what fixed it first. Plain notes are on that incident's
+// own page.
+function similarSection() {
+  if (!similarList.length) return null;
+  return h('section', { class: 'section' },
+    h('h2', { class: 'section-title' }, h('span', { text: 'Seen before' })),
+    h('div', { class: 'card' },
+      h('ul', { class: 'similar' }, similarList.map((s) => h('li', { class: 'similar-item' },
+        h('a', { href: `/incidents/${s.id}`, text: `#${s.id} ${s.title}` }),
+        h('div', { class: 'sub', text: `${when(s.resolved_at)} · ${ago(s.resolved_at)}${s.note_count ? ` · ${s.note_count} note${s.note_count === 1 ? '' : 's'}` : ''}` }),
+        ...s.resolution_notes.map((n) => h('div', { class: 'note note-fix', text: n.detail || '' })),
+      )))),
+  );
+}
+
 function timelineSection() {
   const sorted = [...events].sort((a, b) => Date.parse(a.created_at) - Date.parse(b.created_at) || a.id - b.id);
   return h('section', { class: 'section' },
@@ -223,20 +250,116 @@ function timelineSection() {
   );
 }
 
+const isNote = (ev) => ev.type === 'note' || ev.type === 'resolution_note';
+
 function timelineItem(ev) {
-  const mine = ev.type === 'note' && ev.user_id === myID();
+  const mine = isNote(ev) && ev.user_id === myID();
   return h('li', { class: `tl-item tl-${ev.type}` },
     h('span', { class: 'tl-dot' }),
     h('div', { class: 'tl-body' },
       h('div', { class: 'tl-text' }, eventText(ev)),
       h('div', { class: 'tl-time', title: ev.created_at, text: `${when(ev.created_at)} · ${ago(ev.created_at)}` }),
-      ev.type === 'note' && h('div', { class: 'note', text: ev.detail || '' }),
+      isNote(ev) && h('div', { class: ev.type === 'resolution_note' ? 'note note-fix' : 'note', text: ev.detail || '' }),
       mine && h('div', { class: 'note-actions' },
         h('button', { class: 'btn btn-ghost btn-sm', type: 'button', onclick: () => deleteNote(ev) }, icon('trash'), 'Delete')),
     ),
   );
 }
 
+// ---------- copy ----------
+
+const fence = (rows) => ['```', ...rows, '```'];
+const pairs = (obj) => Object.entries(obj || {}).sort(([a], [b]) => a.localeCompare(b)).map(([k, v]) => `${k}=${v}`);
+
+// incidentMarkdown is everything on this page as text that reads well in a chat
+// or an agent prompt. Times are ISO 8601, since "3 min ago" means nothing once
+// it has been pasted somewhere else.
+function incidentMarkdown() {
+  const out = [`# Incident #${inc.id}: ${inc.title}`, ''];
+  const add = (k, v) => { if (v != null && v !== '') out.push(`- ${k}: ${v}`); };
+  add('Status', inc.status);
+  add('Severity', inc.severity);
+  add('Team', inc.team_name);
+  add('Assigned to', inc.assigned_to_id != null ? inc.assigned_to || 'someone' : 'unassigned');
+  add('Triggered', inc.triggered_at);
+  if (inc.acknowledged_at) add('Acknowledged', `${inc.acknowledged_at} by ${inc.acknowledged_by || 'someone'}`);
+  if (isOpen() && isFuture(inc.snoozed_until)) add('Snoozed until', inc.snoozed_until);
+  if (inc.escalation_level > 0) add('Escalation level', inc.escalation_level);
+  if (inc.resolved_at) add('Resolved', `${inc.resolved_at} (${inc.resolution_source === 'manual' ? 'manually' : 'all alerts stopped firing'})`);
+  if (inc.archived_at) add('Archived', inc.archived_at);
+  const group = pairs(inc.group_labels);
+  if (group.length) out.push('- Grouped by:', ...group.map((g) => `  - ${g}`));
+
+  const alerts = inc.alerts || [];
+  out.push('', `## Alerts (${alerts.length})`);
+  for (const a of alerts) {
+    out.push('', `### ${a.name} (${a.status})`);
+    out.push(`- Started: ${a.starts_at}`);
+    if (a.status === 'resolved' && a.ends_at) out.push(`- Ended: ${a.ends_at}`);
+    if (a.generator_url) out.push(`- Source: ${a.generator_url}`);
+    const labels = pairs(a.labels);
+    if (labels.length) out.push('', 'Labels:', ...fence(labels));
+    const annotations = Object.entries(a.annotations || {}).sort(([x], [y]) => x.localeCompare(y));
+    if (annotations.length) out.push('', 'Annotations:', ...fence(annotations.map(([k, v]) => `${k}: ${v}`)));
+  }
+
+  const sorted = [...events].sort((a, b) => Date.parse(a.created_at) - Date.parse(b.created_at) || a.id - b.id);
+  if (sorted.length) {
+    out.push('', '## Timeline', '');
+    for (const ev of sorted) {
+      const text = eventText(ev, true).map((f) => (f instanceof Node ? f.textContent : f)).join('');
+      out.push(`- ${ev.created_at} ${text}`);
+      if (isNote(ev) && ev.detail) {
+        const label = ev.type === 'resolution_note' ? '  (what fixed it)' : '';
+        out.push(...(label ? [label] : []), ...ev.detail.split('\n').map((l) => `  > ${l}`));
+      }
+    }
+  }
+
+  if (similarList.length) {
+    out.push('', '## Seen before', '', 'Earlier incidents with the same signature:');
+    for (const s of similarList) {
+      out.push(`- #${s.id} ${s.title} (resolved ${s.resolved_at})`);
+      for (const n of s.resolution_notes || []) {
+        out.push('  - What fixed it:', ...(n.detail || '').split('\n').map((l) => `    > ${l}`));
+      }
+    }
+  }
+
+  out.push('', `_Copied from Terminal Duty at ${new Date().toISOString()}_`, '');
+  return out.join('\n');
+}
+
+// writeClipboard falls back to execCommand: the async API needs a secure
+// context, and this server is often reached over plain HTTP.
+async function writeClipboard(text) {
+  try {
+    await navigator.clipboard.writeText(text);
+    return;
+  } catch {
+    // fall through
+  }
+  const ta = h('textarea', { readonly: true, 'aria-hidden': 'true', class: 'clip-buffer' });
+  ta.value = text;
+  document.body.append(ta);
+  ta.select();
+  try {
+    if (!document.execCommand('copy')) throw new Error('copy refused');
+  } finally {
+    ta.remove();
+  }
+}
+
+async function copyIncident() {
+  if (!inc) return;
+  try {
+    await writeClipboard(incidentMarkdown());
+    toast('Copied incident');
+  } catch {
+    toast('Could not copy', 'error');
+  }
+}
+
 // ---------- actions ----------
 
 const isOpen = () => inc.status !== 'resolved';
@@ -296,15 +419,35 @@ function unacknowledge() {
 
 async function resolve() {
   const id = inc.id;
-  const ok = await confirm({
+  const res = await openSheet(() => {
-    title: 'Resolve this incident?',
+    const textarea = h('textarea', {
+      name: 'resolution', autofocus: true, maxlength: '10000',
+      placeholder: 'What fixed it? Optional, shown on the next similar incident.',
+    });
+    const form = h('form', {
+      class: 'sheet-form',
+      onsubmit: (e) => {
+        e.preventDefault();
+        closeSheet({ resolution: textarea.value.trim() });
+      },
+    },
+      h('h2', { class: 'sheet-title', text: 'Resolve this incident?' }),
+      h('p', {
         text: 'Resolving is final. If these alerts fire again they open a new incident, '
           + 'and if any are still firing this one stays closed regardless. '
           + 'Use snooze if you only need it out of the way.',
-    confirmLabel: 'Resolve',
+      }),
-    danger: true,
+      textarea,
+      h('div', { class: 'sheet-actions' },
+        h('button', { class: 'btn', type: 'button', onclick: () => closeSheet(null), text: 'Cancel' }),
+        h('button', { class: 'btn btn-danger', type: 'submit', text: 'Resolve' })),
+    );
+    textarea.addEventListener('keydown', (e) => {
+      if (e.key === 'Enter' && (e.ctrlKey || e.metaKey)) form.requestSubmit();
     });
-  if (ok) await run(() => api.resolve(id), 'Resolved');
+    return form;
+  });
+  if (res) await run(() => api.resolve(id, res.resolution), 'Resolved');
 }
 
 function archive() {
@@ -382,16 +525,18 @@ async function addNote() {
     const textarea = h('textarea', {
       name: 'content', required: true, autofocus: true, placeholder: 'What did you find? What did you do?', maxlength: '10000',
     });
+    const fix = h('input', { type: 'checkbox', name: 'fix' });
     const form = h('form', {
       class: 'sheet-form',
       onsubmit: (e) => {
         e.preventDefault();
         const v = textarea.value.trim();
-        if (v) closeSheet(v);
+        if (v) closeSheet({ content: v, pinned: fix.checked });
       },
     },
       h('h2', { class: 'sheet-title', text: 'Add note' }),
       textarea,
+      h('label', { class: 'check' }, fix, ' This is what fixed it (shown on similar incidents)'),
       h('div', { class: 'sheet-actions' },
         h('button', { class: 'btn', type: 'button', onclick: () => closeSheet(null), text: 'Cancel' }),
         h('button', { class: 'btn btn-primary', type: 'submit', text: 'Save note' })),
@@ -402,7 +547,7 @@ async function addNote() {
     });
     return form;
   });
-  if (content) await run(() => api.addNote(id, content), 'Note added');
+  if (content) await run(() => api.addNote(id, content.content, content.pinned), 'Note added');
 }
 
 async function deleteNote(ev) {
@@ -422,10 +567,12 @@ async function moreMenu() {
     items.push(item('user', 'Assign…', assign));
     items.push(isSnoozed() ? item('bell', 'End snooze', unsnooze) : item('clock', 'Snooze…', snooze));
     items.push(item('note', 'Add note…', addNote));
+    items.push(item('copy', 'Copy incident', copyIncident));
     items.push(h('li', { class: 'menu-sep', role: 'separator' }));
     items.push(item('checkCircle', 'Resolve…', resolve, 'danger'));
   } else {
     items.push(item('note', 'Add note…', addNote));
+    items.push(item('copy', 'Copy incident', copyIncident));
     items.push(inc.archived_at ? item('undo', 'Unarchive', unarchive) : item('archive', 'Archive', archive));
   }
 
@@ -454,6 +601,7 @@ export function key(e) {
     case 'z': if (isOpen() && !isSnoozed()) snooze(); return true;
     case 'Z': if (isSnoozed()) unsnooze(); return true;
     case 'c': addNote(); return true;
+    case 'y': copyIncident(); return true;
     case 'x': if (!isOpen()) (inc.archived_at ? unarchive() : archive()); return true;
     default: return false;
   }

internal/web/static/js/stats.js

@@ -0,0 +1,168 @@
+// Statistics: how many incidents, how fast they are answered, and when and
+// what the alerts are. The same figures the TUI's Stats tab shows, over a
+// range picked with the chips. The server scopes them to the caller's teams.
+
+import * as api from './api.js';
+import { h, clear, emptyState, spinner } from './ui.js';
+import { duration } from './format.js';
+
+const DAY_MS = 24 * 60 * 60 * 1000;
+
+// `days` counts back from today, inclusive; the server reads from/to as UTC
+// dates, so these are too.
+const RANGES = [
+  { id: 'today', label: 'Today', days: 1 },
+  { id: '7d', label: '7d', days: 7 },
+  { id: '30d', label: '30d', days: 30 },
+  { id: '90d', label: '90d', days: 90 },
+  { id: 'all', label: 'All', days: null },
+];
+
+const view = () => document.getElementById('view-stats');
+
+let range = '30d';
+let data = null;
+let error = null;
+
+const utcDate = (ms) => new Date(ms).toISOString().slice(0, 10);
+
+function query(r) {
+  if (!r.days) return {};
+  const now = Date.now();
+  return { from: utcDate(now - (r.days - 1) * DAY_MS), to: utcDate(now) };
+}
+
+export function show() {
+  render();
+  refresh();
+}
+
+export async function refresh() {
+  const requested = range;
+  const q = query(RANGES.find((x) => x.id === range));
+  try {
+    const [incidents, top, byHour, byDay] = await Promise.all([
+      api.statsIncidents(q),
+      api.statsTop({ ...q, limit: 10 }),
+      api.statsByHour(q),
+      api.statsByDay(q),
+    ]);
+    if (requested !== range) return;
+    data = { incidents, top, byHour, byDay };
+    error = null;
+  } catch (err) {
+    if (requested !== range) return;
+    error = err.message;
+  }
+  render();
+}
+
+function setRange(id) {
+  if (id === range) return;
+  range = id;
+  data = null;
+  render();
+  refresh();
+}
+
+function render() {
+  const chips = h('div', { class: 'chips', role: 'tablist', 'aria-label': 'Time range' },
+    RANGES.map((r) => h('button', {
+      class: 'chip',
+      type: 'button',
+      role: 'tab',
+      'aria-selected': String(r.id === range),
+      onclick: () => setRange(r.id),
+      text: r.label,
+    })));
+
+  let body;
+  if (error && !data) body = h('div', { class: 'load-error', text: error });
+  else if (!data) body = spinner();
+  else if (!data.incidents.total && !data.byHour.some((x) => x.count)) {
+    body = emptyState('No data in this range', '', 'chart');
+  } else {
+    body = h('div', { class: 'stats' },
+      error && h('div', { class: 'load-error', text: `Showing older data: ${error}` }),
+      tiles(data.incidents),
+      data.top.length > 0 && card('Top alerts', topAlerts(data.top)),
+      card('Alerts by hour (UTC)', columns(
+        data.byHour.map((x) => ({ label: String(x.hour), value: x.count, tick: x.hour % 6 === 0 })),
+        'Alerts by hour of day')),
+      card('Alerts by day', columns(
+        data.byDay.map((x) => ({ label: x.day_name.slice(0, 3), value: x.count, tick: true })),
+        'Alerts by day of week')));
+  }
+  clear(view(), h('div', {}, chips, body));
+}
+
+// A missing mean means nothing has been acknowledged or resolved yet.
+const mean = (s) => (s == null ? '—' : duration(s * 1000));
+
+function tiles(s) {
+  const tile = (label, value, cls = '') => h('div', { class: `stat-tile ${cls}` },
+    h('div', { class: 'stat-value', text: String(value) }),
+    h('div', { class: 'stat-label', text: label }));
+  return h('div', { class: 'stat-tiles' },
+    tile('Incidents', s.total),
+    tile('Triggered', s.triggered, 'st-triggered'),
+    tile('Acknowledged', s.acknowledged, 'st-acknowledged'),
+    tile('Resolved', s.resolved, 'st-resolved'),
+    tile('Mean time to acknowledge', mean(s.mtta_seconds)),
+    tile('Mean time to resolve', mean(s.mttr_seconds)));
+}
+
+function card(title, content) {
+  return h('section', { class: 'chart-card card card-pad' },
+    h('h3', { class: 'chart-title', text: title }), content);
+}
+
+// Ranked names with a bar scaled to the busiest one.
+function topAlerts(items) {
+  const max = Math.max(...items.map((x) => x.count), 1);
+  return h('ol', { class: 'hbars' }, items.map((x) => {
+    const fill = h('span', { class: 'hbar-fill' });
+    fill.style.width = `${Math.max(2, (x.count / max) * 100)}%`;
+    return h('li', { class: 'hbar' },
+      h('span', { class: 'hbar-name', title: x.name, text: x.name }),
+      h('span', { class: 'hbar-track' }, fill),
+      h('span', { class: 'hbar-count', text: String(x.count) }));
+  }));
+}
+
+const SVG_NS = 'http://www.w3.org/2000/svg';
+
+function svg(tag, attrs = {}, text) {
+  const el = document.createElementNS(SVG_NS, tag);
+  for (const [k, v] of Object.entries(attrs)) el.setAttribute(k, String(v));
+  if (text != null) el.textContent = text;
+  return el;
+}
+
+// A column chart: one bar per item, the value in a tooltip, and a label under
+// the items marked `tick`.
+function columns(items, label) {
+  const W = 480;
+  const H = 140;
+  const base = H - 18;
+  const step = W / items.length;
+  const max = Math.max(...items.map((x) => x.value), 1);
+  const root = svg('svg', {
+    class: 'columns', viewBox: `0 0 ${W} ${H}`, role: 'img', 'aria-label': label,
+  });
+  root.appendChild(svg('line', { class: 'axis', x1: 0, x2: W, y1: base, y2: base }));
+  items.forEach((it, i) => {
+    const bh = it.value ? Math.max(2, (it.value / max) * (base - 6)) : 0;
+    const x = i * step + step * 0.15;
+    const g = svg('g', { class: 'col' });
+    g.appendChild(svg('title', {}, `${it.label}: ${it.value}`));
+    // A full-height transparent hit area, so a tiny bar is still hoverable.
+    g.appendChild(svg('rect', { class: 'col-hit', x: i * step, y: 0, width: step, height: base }));
+    if (bh) g.appendChild(svg('rect', { class: 'col-bar', x, y: base - bh, width: step * 0.7, height: bh, rx: 2 }));
+    root.appendChild(g);
+    if (it.tick) {
+      root.appendChild(svg('text', { class: 'col-label', x: i * step + step / 2, y: H - 4, 'text-anchor': 'middle' }, it.label));
+    }
+  });
+  return root;
+}

internal/web/static/js/team.js

@@ -18,9 +18,9 @@
 // than no form, but it is not the thing enforcing anything.
 
 import * as api from './api.js';
-import { h, clear, spinner, confirm, icon, openSheet, closeSheet, menuCard } from './ui.js';
+import { h, clear, spinner, confirm, icon, openSheet, closeSheet, menuCard, badge, labelChip } from './ui.js';
 import { state, currentTeam, users as allUsers, myID } from './state.js';
-import { isoDate, addDays, mondayOf, initial } from './format.js';
+import { isoDate, addDays, mondayOf, initial, ago, when, duration } from './format.js';
 
 const view = () => document.getElementById('view-team');
 
@@ -111,14 +111,14 @@ async function load(id) {
     return { members, escalation };
   }
   if (tab === 'sources') return { integrations: await api.integrations(id) };
-  if (tab === 'deadman') return { deadman: await api.deadman(id) };
+  if (tab === 'deadman') return { deadman: await api.deadmanSwitches(id) };
 
   const grid = gridDays();
   const [members, integrations, escalation, deadman, schedule] = await Promise.all([
     api.teamMembers(id),
     api.integrations(id),
     api.escalation(id),
-    api.deadman(id),
+    api.deadmanSwitches(id),
     api.schedule(id, isoDate(grid.start), isoDate(addDays(grid.start, grid.count - 1))),
   ]);
   return { members, integrations, escalation, deadman, schedule };
@@ -203,8 +203,8 @@ function overview() {
   const levels = (data.escalation?.levels || []).length;
   const keys = (data.integrations || []).length;
   const unused = (data.integrations || []).filter((i) => !i.last_used_at).length;
-  const switches = (data.deadman?.matchers || '')
+  const switches = (data.deadman || []).length;
-    .split(';').map((x) => x.trim()).filter(Boolean).length;
+  const dead = (data.deadman || []).filter((s) => s.status === 'dead').length;
 
   return h('div', { class: 'overview-menu' },
     menuCard('/team/rota', 'Rota', null,
@@ -220,7 +220,9 @@ function overview() {
         ? (unused ? `${unused} of them never used.` : 'All in use.')
         : 'No key yet, so nothing can reach this team.'),
     menuCard('/team/deadman', 'Dead man’s switches', switches || null,
-      switches ? 'Alerts whose absence opens an incident.' : 'Nothing watched.'),
+      switches
+        ? (dead ? `${dead} of them silent.` : 'All quiet, as they should be.')
+        : 'Nothing watched.'),
   );
 }
 
@@ -574,33 +576,54 @@ function minutesInput(seconds, onChange) {
 function integrationsCard() {
   const rows = (data.integrations || []).map((i) =>
     h('tr', {},
-      h('td', {}, h('strong', { text: i.name })),
+      h('td', {}, sourceBadge(i.status)),
-      h('td', { class: 'muted small', text: i.kind }),
+      h('td', {},
-      h('td', { class: 'muted small', text: i.last_used_at ? 'in use' : 'never used' }),
+        h('strong', { text: i.name }),
-      h('td', {}, isOwner() && h('button', {
+        h('div', { class: 'muted small', text: i.kind })),
+      // When the key last posted, and when an alert last arrived on it. They
+      // differ: a payload with nothing usable in it stamps only the first.
+      h('td', { class: 'muted small' }, timeCell(i.last_used_at)),
+      h('td', { class: 'muted small' }, timeCell(i.last_alert_at)),
+      h('td', { class: 'muted small num', title: 'Distinct alerts refreshed in the last 24 hours',
+        text: String(i.alerts_24h ?? 0) }),
+      h('td', { class: 'muted small' }, h('span', { title: when(i.created_at), text: ago(i.created_at) })),
+      h('td', {}, isOwner() && h('div', { class: 'row-actions' },
+        h('button', {
+          class: 'btn-sm', type: 'button', text: 'Rename', onclick: () => openRenameSource(i),
+        }),
+        h('button', {
           class: 'btn-sm danger', type: 'button', text: 'Revoke',
           onclick: async () => {
             if (!(await confirm({
               title: `Revoke ${i.name}?`,
-            text: 'Anything posting with this key stops delivering immediately.',
+              text: 'Anything posting with this key stops delivering immediately. Alerts it already delivered stay.',
               confirmLabel: 'Revoke',
               danger: true,
             }))) return;
             act(() => api.deleteIntegration(teamID, i.id));
           },
-      })),
+        }))),
     ));
 
   return h('div', { class: 'card' },
+    h('div', { class: 'card-head' },
       h('h2', { text: 'Alert sources' }),
+      isOwner() && h('button', {
+        class: 'btn', type: 'button', text: 'New source', onclick: openNewSource,
+      })),
     h('p', { class: 'muted small' },
       'Alerts arrive on an integration key, which says both that the sender may ',
       'post and which team the alerts belong to.'),
-    rows.length
-      ? h('table', { class: 'admin-table' }, h('tbody', {}, rows))
-      : h('p', { class: 'muted', text: 'No alert source yet, so nothing can reach this team.' }),
     freshKey && newKeyPanel(),
-    isOwner() && !freshKey && newIntegrationForm(),
+    rows.length
+      ? h('div', { class: 'table-scroll' },
+        h('table', { class: 'admin-table status-table' },
+          h('thead', {}, h('tr', {},
+            h('th', { text: 'Status' }), h('th', { text: 'Source' }),
+            h('th', { text: 'Last webhook' }), h('th', { text: 'Last alert' }),
+            h('th', { class: 'num', text: 'Alerts 24h' }), h('th', { text: 'Created' }), h('th'))),
+          h('tbody', {}, rows)))
+      : h('p', { class: 'muted', text: 'No alert source yet, so nothing can reach this team.' }),
   );
 }
 
@@ -631,66 +654,204 @@ function newKeyPanel() {
   );
 }
 
-function newIntegrationForm() {
+// A sheet with one name field, for adding a source and for renaming one: the two
-  const name = h('input', { type: 'text', placeholder: 'prod alertmanager', required: true });
+// differ only in what they call and what they put in the box.
-  const form = h('form', { class: 'inline-form' }, name,
+function openNameSheet({ title, submit, value, run }) {
-    h('button', { class: 'btn', type: 'submit', text: 'Add' }));
+  const name = h('input', {
+    type: 'text', placeholder: 'prod alertmanager', required: true, value, autofocus: true,
+  });
+  const problem = h('p', { class: 'load-error', hidden: true });
+  const form = h('form', { class: 'stacked-form' },
+    h('label', {}, 'Name ', name),
+    problem,
+    h('div', { class: 'sheet-actions' },
+      h('button', { class: 'btn', type: 'button', text: 'Cancel', onclick: () => closeSheet(false) }),
+      h('button', { class: 'btn btn-primary', type: 'submit', text: submit })));
+
   form.addEventListener('submit', async (e) => {
     e.preventDefault();
     try {
-      freshKey = await api.createIntegration(teamID, name.value.trim());
+      await run(name.value.trim());
-      await refresh();
     } catch (err) {
-      error = err.message;
+      problem.textContent = err.message;
-      render();
+      problem.hidden = false;
+      return;
     }
+    closeSheet(true);
+    refresh();
+  });
+  openSheet(() => [h('h2', { class: 'sheet-title', text: title }), form]);
+}
+
+function openNewSource() {
+  openNameSheet({
+    title: 'New source', submit: 'Add source', value: '',
+    // The key comes back once, and the card shows it until dismissed.
+    run: async (name) => { freshKey = await api.createIntegration(teamID, name); },
+  });
+}
+
+function openRenameSource(i) {
+  openNameSheet({
+    title: `Rename ${i.name}`, submit: 'Rename', value: i.name,
+    run: (name) => api.renameIntegration(teamID, i.id, name),
   });
-  return form;
 }
 
 // --- dead man's switches ---------------------------------------------------
 
+// Status badges, shared by the Sources and Switches lists: a table of label and
+// hint per status, and one function to draw it. Module-level, so the two cards
+// can be defined in either order.
+const SWITCH_STATUS = {
+  healthy: { label: 'Healthy', hint: 'Heard from within its timeout.' },
+  dead: { label: 'Dead', hint: 'Silent for longer than its timeout.' },
+  dormant: { label: 'Dormant', hint: 'Nothing has matched yet, so there is nothing to lose.' },
+};
+
+const SOURCE_STATUS = {
+  active: { label: 'Active', hint: 'Posted within the last day.' },
+  quiet: { label: 'Quiet', hint: 'Has posted, but not in the last day. Nothing firing is a fine reason.' },
+  never: { label: 'Never used', hint: 'Nothing has been posted with this key yet.' },
+};
+
+function statusBadge(table, status, fallback) {
+  const s = table[status] || table[fallback];
+  const el = badge(s.label, `st-${status}`);
+  el.title = s.hint;
+  return el;
+}
+
+const switchBadge = (status) => statusBadge(SWITCH_STATUS, status, 'dormant');
+const sourceBadge = (status) => statusBadge(SOURCE_STATUS, status, 'never');
+
+const timeCell = (iso) => iso
+  ? h('span', { title: when(iso), text: ago(iso) })
+  : h('span', { class: 'muted', text: 'never' });
+
+// When it last opened an incident. An incident that is still open is a link,
+// because that is the thing somebody looking at a red row wants next.
+const triggeredCell = (iso, incidentID) => {
+  if (!iso) return h('span', { class: 'muted', text: 'never' });
+  return incidentID
+    ? h('a', { href: `/incidents/${incidentID}`, title: when(iso) }, `#${incidentID} · ${ago(iso)}`)
+    : h('span', { title: when(iso), text: ago(iso) });
+};
+
+function switchRows(sw) {
+  const main = h('tr', {},
+    h('td', {}, switchBadge(sw.status)),
+    h('td', {},
+      h('strong', { text: sw.name }),
+      sw.name !== sw.matcher && h('div', { class: 'muted small' }, h('code', { text: sw.matcher }))),
+    h('td', { class: 'muted small' }, timeCell(sw.last_heartbeat_at)),
+    h('td', { class: 'muted small' }, triggeredCell(sw.last_triggered_at, sw.open_incident_id)),
+    h('td', { class: 'muted small', text: duration(sw.timeout_seconds * 1000) }),
+    h('td', {}, isOwner() && h('button', {
+      class: 'btn-sm danger', type: 'button', text: 'Remove',
+      onclick: async () => {
+        if (!(await confirm({
+          title: `Remove ${sw.name}?`,
+          text: 'It stops being watched. An incident it already opened stays open until it is resolved.',
+          confirmLabel: 'Remove',
+          danger: true,
+        }))) return;
+        act(() => api.deleteDeadmanSwitch(teamID, sw.id));
+      },
+    })),
+  );
+
+  // One heartbeat is the switch's own times; several are worth telling apart,
+  // since a live cluster must not hide a dead one.
+  const sources = sw.sources.length > 1
+    ? sw.sources.map((src) => h('tr', { class: 'source-row' },
+      h('td', {}, switchBadge(src.status)),
+      h('td', { class: 'source-labels' },
+        ...Object.entries(src.labels || {})
+          .filter(([k]) => k !== 'alertname')
+          .map(([k, v]) => labelChip(k, v)),
+        !Object.keys(src.labels || {}).some((k) => k !== 'alertname')
+          && h('code', { class: 'small', text: src.fingerprint })),
+      h('td', { class: 'muted small' }, timeCell(src.last_heartbeat_at)),
+      h('td', { class: 'muted small' }, triggeredCell(src.last_triggered_at, src.incident_id)),
+      h('td'), h('td')))
+    : [];
+  return [main, ...sources];
+}
+
 function deadmanCard() {
-  const d = data.deadman || {};
+  const switches = data.deadman || [];
-  const matchers = h('input', {
-    type: 'text', value: d.matchers || '', placeholder: 'alertname=Watchdog',
-    class: 'wide',
-  });
-  const timeout = h('input', {
-    type: 'number', min: '0', class: 'setting-value',
-    value: String(Math.round((d.timeout_seconds || 0) / 60)),
-  });
-  const severity = h('select', {},
-    ...['critical', 'error', 'warning', 'info'].map((s) =>
-      h('option', { value: s, text: s, selected: (d.severity || 'critical') === s })));
-
-  const form = h('form', { class: 'stacked-form' },
-    h('label', {}, 'Heartbeat alerts ', matchers),
-    h('label', {}, 'Declare dead after ', timeout, ' minutes of silence'),
-    h('label', {}, 'Open the incident at severity ', severity),
-    h('button', { class: 'btn', type: 'submit', text: 'Save switches' }));
-
-  form.addEventListener('submit', (e) => {
-    e.preventDefault();
-    act(() => api.setDeadman(teamID, {
-      matchers: matchers.value.trim(),
-      timeout_seconds: Number(timeout.value) * 60,
-      severity: severity.value,
-    }));
-  });
 
   return h('div', { class: 'card' },
+    h('div', { class: 'card-head' },
       h('h2', { text: 'Dead man’s switches' }),
+      isOwner() && h('button', {
+        class: 'btn', type: 'button', text: 'New switch', onclick: openNewSwitch,
+      })),
     h('p', { class: 'muted small' },
       'Alerts whose ABSENCE is the signal. Receiving one opens nothing; going ',
-      'quiet for longer than the timeout opens an incident. ',
+      'quiet for longer than the switch’s timeout opens an incident.'),
-      h('code', { text: 'alertname=Watchdog,cluster=prod; alertname=EdgeHeartbeat' }),
+    switches.length
-      ' — semicolons separate switches, commas separate conditions, and every ',
+      ? h('div', { class: 'table-scroll' },
-      'switch must name an alertname. Leave empty to watch nothing.'),
+        h('table', { class: 'admin-table status-table' },
-    isOwner() ? form : h('p', { class: 'muted', text: d.matchers || 'Nothing watched.' }),
+          h('thead', {}, h('tr', {},
+            h('th', { text: 'Status' }), h('th', { text: 'Switch' }),
+            h('th', { text: 'Last heartbeat' }), h('th', { text: 'Last triggered' }),
+            h('th', { text: 'Silent after' }), h('th'))),
+          h('tbody', {}, switches.flatMap(switchRows))))
+      : h('p', { class: 'muted', text: 'Nothing watched.' }),
   );
 }
 
+// The form lives in the sheet, not on the page: most visits are to look at the
+// list, and a form that is always open is the page this replaced.
+function openNewSwitch() {
+  const name = h('input', { type: 'text', placeholder: 'Prod Watchdog', autofocus: true });
+  const matcher = h('input', {
+    type: 'text', placeholder: 'alertname=Watchdog,cluster=prod', class: 'wide', required: true,
+  });
+  const timeout = h('input', {
+    type: 'number', min: '1', value: '15', class: 'setting-value', required: true,
+  });
+  const severity = h('select', {},
+    ...['critical', 'error', 'warning', 'info'].map((s) => h('option', { value: s, text: s })));
+  const problem = h('p', { class: 'load-error', hidden: true });
+
+  const form = h('form', { class: 'stacked-form' },
+    h('label', {}, 'Name (optional) ', name),
+    h('label', {}, 'Heartbeat alert ', matcher),
+    h('p', { class: 'muted small' },
+      'Conditions are ', h('code', { text: 'label=value' }), ' separated by commas, and one ',
+      'must be ', h('code', { text: 'alertname' }), '. Every distinct label set that ',
+      'matches is watched on its own.'),
+    h('label', {}, 'Declare dead after ', timeout, ' minutes of silence'),
+    h('label', {}, 'Open the incident at severity ', severity),
+    problem,
+    h('div', { class: 'sheet-actions' },
+      h('button', { class: 'btn', type: 'button', text: 'Cancel', onclick: () => closeSheet(false) }),
+      h('button', { class: 'btn btn-primary', type: 'submit', text: 'Add switch' })));
+
+  form.addEventListener('submit', async (e) => {
+    e.preventDefault();
+    try {
+      await api.createDeadmanSwitch(teamID, {
+        name: name.value.trim(),
+        matcher: matcher.value.trim(),
+        timeout_seconds: Math.round(Number(timeout.value) * 60),
+        severity: severity.value,
+      });
+    } catch (err) {
+      problem.textContent = err.message;
+      problem.hidden = false;
+      return;
+    }
+    closeSheet(true);
+    refresh();
+  });
+
+  openSheet(() => [h('h2', { class: 'sheet-title', text: 'New switch' }), form]);
+}
+
 // --- members ---------------------------------------------------------------
 
 function membersCard() {

internal/web/static/js/ui.js

@@ -34,15 +34,21 @@ export function clear(el, ...children) {
 const ICONS = {
   back: ['M15 18l-6-6 6-6'],
   more: ['M5 12h.01M12 12h.01M19 12h.01'],
+  queueList: ['M4 6h16M4 12h16M4 18h10'],
+  calendar: ['rect:3.5,5,17,15,2', 'M3.5 10h17M8 3v4M16 3v4'],
+  team: ['circle:9,8,3', 'circle:17,9,2.5', 'M3 19a6 6 0 0 1 12 0M15 19a5 5 0 0 1 6-4'],
+  shield: ['M12 3l7 3v6c0 4-3 7-7 9-4-2-7-5-7-9V6z'],
   check: ['M5 12.5l4.5 4.5L19 7'],
   checkCircle: ['M8 12.5l3 3 5-6', 'circle:12,12,9'],
   undo: ['M9 14L4 9l5-5', 'M4 9h10a6 6 0 0 1 0 12h-3'],
   user: ['circle:12,8,3.5', 'M5 20a7 7 0 0 1 14 0'],
   clock: ['circle:12,12,9', 'M12 7v5l3 2'],
+  chart: ['M4 20h16M7 20v-7M12 20V6M17 20v-10'],
   bell: ['M6 16V11a6 6 0 0 1 12 0v5l1.5 2h-15z', 'M10 20.5a2 2 0 0 0 4 0'],
   note: ['M5 4h14v12l-4 4H5z', 'M15 20v-4h4', 'M9 9h6M9 13h4'],
   archive: ['M3.5 5h17v4h-17z', 'M5 9v10h14V9', 'M10 13h4'],
   flag: ['M5 21V4', 'M5 4h11l-2 4 2 4H5'],
+  copy: ['rect:9,9,11,11,2', 'M5 15V6a2 2 0 0 1 2-2h9'],
   trash: ['M4 7h16', 'M9 7V4h6v3', 'M6 7l1 13h10l1-13'],
   chevronLeft: ['M15 18l-6-6 6-6'],
   chevronRight: ['M9 6l6 6-6 6'],
@@ -64,6 +70,14 @@ export function icon(name, cls = 'icon') {
       node.setAttribute('cx', cx);
       node.setAttribute('cy', cy);
       node.setAttribute('r', r);
+    } else if (d.startsWith('rect:')) {
+      const [x, y, w, hgt, rx] = d.slice(5).split(',');
+      node = document.createElementNS(SVG, 'rect');
+      node.setAttribute('x', x);
+      node.setAttribute('y', y);
+      node.setAttribute('width', w);
+      node.setAttribute('height', hgt);
+      if (rx) node.setAttribute('rx', rx);
     } else {
       node = document.createElementNS(SVG, 'path');
       node.setAttribute('d', d);

internal/web/stats_test.go

@@ -0,0 +1,26 @@
+package web
+
+import (
+	"io/fs"
+	"strings"
+	"testing"
+)
+
+func TestStatsPageIsEmbedded(t *testing.T) {
+	sub, err := fs.Sub(files, "static")
+	if err != nil {
+		t.Fatal(err)
+	}
+	index, err := fs.ReadFile(sub, "index.html")
+	if err != nil {
+		t.Fatal(err)
+	}
+	for _, want := range []string{`id="view-stats"`, `data-section="stats"`} {
+		if !strings.Contains(string(index), want) {
+			t.Errorf("index.html lacks %s", want)
+		}
+	}
+	if _, err := fs.Stat(sub, "js/stats.js"); err != nil {
+		t.Errorf("js/stats.js not embedded: %v", err)
+	}
+}