Set the chart's placeholder version to 0.25.0

Cosmetic: make helm-package sets the published version and appVersion
from the tag, so these two fields decide nothing (see the comment
above them). Kept in step anyway, same as e8d45f9 and 3ee8583, so a
tree heading for v0.25.0 doesn't say 0.24.0.

README.md

@@ -513,19 +513,27 @@ nothing unless something downstream notices it stop. That is what
 `TERDUT_DEADMAN_MATCHERS` defaults to.
 
 **Switches belong to a team**, which decides which of its own alerts are
-heartbeats and how long a silence has to last. An owner sets them through
+heartbeats and how long a silence has to last. Each **switch** is a row of its
-`PUT /api/teams/{teamID}/deadman`; a missed heartbeat opens an incident in the
+own — a name, one matcher, a timeout and a severity — so switches in one team
-team whose integration received it.
+can have different deadlines. An owner adds and removes them on **Team →
+Switches**, which lists each with a status (**healthy**, **dead**, or
+**dormant** until its first heartbeat), when it was last heard from, and when it
+last opened an incident; a matcher that several clusters satisfy is broken down
+per cluster. The API is `POST`/`DELETE /api/teams/{teamID}/deadman/switches`. A
+missed heartbeat opens an incident in the team whose integration received it.
+Removing a switch stops the watching; an incident it already opened stays open
+until somebody resolves it.
 
-The environment variables are the starting point, not the setting: at startup
+The environment variables are the starting point, not the setting: the **first**
-every team **without** a configuration of its own is given one from them, and an
+time the server starts, every team is given a switch per default matcher from
-owner's later edit is never overwritten by a redeploy. A team created after
+them, once. After that a team's switches are its own — an owner's edit or
-that starts watching nothing until its owner says otherwise — inheriting an
+deletion is never put back by a redeploy. A team created later starts watching
-install-wide heartbeat would page a new team about a source it has never heard
+nothing until its owner says otherwise — inheriting an install-wide heartbeat
-of.
+would page a new team about a source it has never heard of.
 
 A matcher is a set of exact label conditions, one of which must be the
-`alertname`, in the same format the environment variable uses:
+`alertname`, in the format the environment variable uses (one matcher per switch; the
+variable takes several, separated by `;`):
 
 ```
 alertname=Watchdog,cluster=prod; alertname=EdgeHeartbeat
@@ -710,16 +718,18 @@ administrator who is not in the team gets the same `404` as anybody else.
 | `GET` | `/api/teams/{teamID}/members` | member | Who is in the team |
 | `POST` | `/api/teams/{teamID}/members` | **owner** | Add a member, or change their role `{"user_id","role"}` |
 | `DELETE` | `/api/teams/{teamID}/members/{userID}` | **owner** | Remove a member. `409` for the last owner |
-| `GET` | `/api/teams/{teamID}/integrations` | member | List integrations. Never returns keys |
+| `GET` | `/api/teams/{teamID}/integrations` | member | List integrations. Never returns keys. Each carries `status` (`active` if its key posted within 24h, `quiet` if it has but not lately, `never`), `last_used_at` (last webhook, usable or not), `last_alert_at` (when an alert last arrived on it) and `alerts_24h` (distinct alerts it refreshed in the last day). Alerts delivered before the source was recorded (migration 010) have none, so the last two fill in as Alertmanager re-sends them |
+| `PATCH` | `/api/teams/{teamID}/integrations/{integrationID}` | **owner** | Rename `{"name"}`. The key does not change |
 | `POST` | `/api/teams/{teamID}/integrations` | **owner** | Mint an integration `{"name","kind"}` — key and URL shown once |
-| `DELETE` | `/api/teams/{teamID}/integrations/{integrationID}` | **owner** | Revoke an integration |
+| `DELETE` | `/api/teams/{teamID}/integrations/{integrationID}` | **owner** | Revoke an integration. Alerts it delivered stay, unattributed |
 | `GET` | `/api/teams/{teamID}/invites` | **owner** | The team's invite links, with their uses and expiry. Never the tokens |
 | `POST` | `/api/teams/{teamID}/invites` | **owner** | Mint one `{"role","max_uses"}` — the full URL is returned once |
 | `DELETE` | `/api/teams/{teamID}/invites/{inviteID}` | **owner** | Revoke a link before it expires |
 | `GET` | `/api/teams/{teamID}/escalation` | member | The team's [escalation ladder](#escalation) `{repeat_count, fallback_topic, levels[]}`. Empty levels means the team has none |
 | `PUT` | `/api/teams/{teamID}/escalation` | **owner** | Replace it wholesale. `400` for a level with no targets or no timeout — a rung that pages nobody is a silence with a number on it |
-| `GET` | `/api/teams/{teamID}/deadman` | member | The team's [dead man's switch](#dead-mans-switch) configuration `{matchers, timeout_seconds, severity}` |
+| `GET` | `/api/teams/{teamID}/deadman/switches` | member | The team's [dead man's switches](#dead-mans-switch), each `{id, name, matcher, timeout_seconds, severity, status, last_heartbeat_at, last_triggered_at, open_incident_id, sources[]}`. `status` is `healthy`, `dead` or `dormant`; `sources` has one entry per heartbeat fingerprint. Empty when the team watches nothing |
-| `PUT` | `/api/teams/{teamID}/deadman` | **owner** | Replace it. `400` when no matcher names an `alertname`, because a switch that silently watches nothing is the failure this feature exists to prevent |
+| `POST` | `/api/teams/{teamID}/deadman/switches` | **owner** | Add one: `{name?, matcher, timeout_seconds, severity?}`. `400` when the matcher names no `alertname` or holds several, or the timeout is not positive — a switch that silently watches nothing is the failure this feature exists to prevent |
+| `DELETE` | `/api/teams/{teamID}/deadman/switches/{switchID}` | **owner** | Stop watching. An incident it opened stays open. `404` for a switch of another team |
 
 ### Notifications
 
@@ -964,8 +974,8 @@ What changes, and will need attention:
 
 **Dead man's switches moved too.** `TERDUT_DEADMAN_MATCHERS`, `_TIMEOUT` and
 `_SEVERITY` are no longer the setting; they are the default each existing team
-is seeded with at startup, after which an owner edits them per team through
+is seeded with at startup, after which an owner manages them per team through
-`PUT /api/teams/{teamID}/deadman` and a redeploy never overwrites that.
+`/api/teams/{teamID}/deadman/switches` and a redeploy never overwrites that.
 
 Nothing else about an incident changes, and incidents never move between teams:
 an alert belongs to whichever team's key it arrived on.

charts/terdut-server/Chart.yaml

@@ -15,5 +15,5 @@ type: application
 # appVersion and image.tag in values.yaml no longer agree, and that is not an oversight:
 # image.tag stays "latest", which is what a local install actually pulls. appVersion is
 # metadata and drives nothing.
-version: 0.22.1
+version: 0.25.0
-appVersion: "v0.22.1"
+appVersion: "v0.25.0"

internal/api/alertmanager.go

@@ -78,7 +78,7 @@ type ingested struct {
 // post, and which team the alerts belong to.
 func handleIntegrationWebhook(db *sql.DB, notify NotifyConfig) http.HandlerFunc {
 	return func(w http.ResponseWriter, r *http.Request) {
-		teamID, err := teamIDForKey(r.Context(), db, chi.URLParam(r, "key"))
+		src, err := sourceForKey(r.Context(), db, chi.URLParam(r, "key"))
 		if err != nil {
 			if errors.Is(err, errUnknownIntegration) {
 				// 401 and not 404: the path is real, the key is not, and a
@@ -90,11 +90,12 @@ func handleIntegrationWebhook(db *sql.DB, notify NotifyConfig) http.HandlerFunc
 			respond(w, http.StatusInternalServerError, errResp("internal error"))
 			return
 		}
-		receiveWebhook(w, r, db, notify, teamID)
+		receiveWebhook(w, r, db, notify, src)
 	}
 }
 
-func receiveWebhook(w http.ResponseWriter, r *http.Request, db *sql.DB, notify NotifyConfig, teamID int64) {
+func receiveWebhook(w http.ResponseWriter, r *http.Request, db *sql.DB, notify NotifyConfig, src alertSource) {
+	teamID := src.teamID
 	var payload amPayload
 	if err := decodeJSON(r, &payload); err != nil {
 		respond(w, http.StatusBadRequest, errResp("invalid payload"))
@@ -104,7 +105,7 @@ func receiveWebhook(w http.ResponseWriter, r *http.Request, db *sql.DB, notify N
 	// Alertmanager retries anything that is not 2xx, and a retry of a payload
 	// we failed to store is more useful than an error it cannot act on — so
 	// failures are logged, not surfaced.
-	if err := ingest(r.Context(), db, notify, teamID, payload); err != nil {
+	if err := ingest(r.Context(), db, notify, src, payload); err != nil {
 		log.Printf("webhook ingest (team %d, group %q): %v", teamID, payload.GroupKey, err)
 	}
 
@@ -114,7 +115,8 @@ func receiveWebhook(w http.ResponseWriter, r *http.Request, db *sql.DB, notify N
 // ingest stores a payload's alerts and reconciles the incident for its group.
 // The whole payload is one transaction: an incident that opened but whose alerts
 // failed to link would be a work item nobody could act on.
-func ingest(ctx context.Context, db *sql.DB, notify NotifyConfig, teamID int64, payload amPayload) error {
+func ingest(ctx context.Context, db *sql.DB, notify NotifyConfig, src alertSource, payload amPayload) error {
+	teamID := src.teamID
 	tx, err := db.BeginTx(ctx, nil)
 	if err != nil {
 		return err
@@ -124,12 +126,12 @@ func ingest(ctx context.Context, db *sql.DB, notify NotifyConfig, teamID int64,
 	// Which arriving alerts are heartbeats is the team's own answer, read
 	// inside the transaction so an owner editing it mid-payload cannot split
 	// one webhook across two interpretations.
-	deadman, err := deadmanConfigForTeam(ctx, tx, teamID)
+	deadman, err := deadmanSetForTeam(ctx, tx, teamID)
 	if err != nil {
 		return err
 	}
 
-	accepted, err := upsertAlerts(ctx, tx, deadman, teamID, payload.Alerts)
+	accepted, err := upsertAlerts(ctx, tx, deadman, src, payload.Alerts)
 	if err != nil {
 		return err
 	}
@@ -186,7 +188,8 @@ func ingest(ctx context.Context, db *sql.DB, notify NotifyConfig, teamID int64,
 
 // upsertAlerts stores each alert of a payload and reports what changed. Payloads
 // the ordering guard rejected are left out entirely.
-func upsertAlerts(ctx context.Context, tx *sql.Tx, deadman DeadmanConfig, teamID int64, alerts []amAlert) ([]ingested, error) {
+func upsertAlerts(ctx context.Context, tx *sql.Tx, deadman deadmanSet, src alertSource, alerts []amAlert) ([]ingested, error) {
+	teamID := src.teamID
 	now := time.Now().Unix()
 	accepted := make([]ingested, 0, len(alerts))
 
@@ -242,8 +245,8 @@ func upsertAlerts(ctx context.Context, tx *sql.Tx, deadman DeadmanConfig, teamID
 		if _, err := tx.ExecContext(ctx, `
 			INSERT INTO alerts
 				(team_id, fingerprint, name, status, labels, annotations, starts_at, ends_at,
-				 generator_url, received_at, resolution_source)
+				 generator_url, received_at, resolution_source, integration_id)
-			VALUES ($1, $2, $3, $4, $5::jsonb, $6::jsonb, $7, $8, $9, $10, $11)
+			VALUES ($1, $2, $3, $4, $5::jsonb, $6::jsonb, $7, $8, $9, $10, $11, $12)
 			ON CONFLICT (team_id, fingerprint) DO UPDATE SET
 				status            = excluded.status,
 				labels            = excluded.labels,
@@ -257,6 +260,8 @@ func upsertAlerts(ctx context.Context, tx *sql.Tx, deadman DeadmanConfig, teamID
 				-- is a breaking API change — see models.Alert.ReceivedAt.
 				received_at       = excluded.received_at,
 				resolution_source = excluded.resolution_source,
+				-- Last sender wins; see migration 010.
+				integration_id    = excluded.integration_id,
 				-- A re-fire makes the alert current again, so it leaves the archive.
 				archived_at       = CASE WHEN excluded.status = 'firing'
 				                        THEN NULL ELSE alerts.archived_at END
@@ -267,7 +272,7 @@ func upsertAlerts(ctx context.Context, tx *sql.Tx, deadman DeadmanConfig, teamID
 			teamID, a.Fingerprint, name, a.Status,
 			string(labelsJSON), string(annotationsJSON),
 			a.StartsAt.Unix(), endsAtUnix,
-			a.GeneratorURL, now, resolutionSource,
+			a.GeneratorURL, now, resolutionSource, src.integrationID,
 		); err != nil {
 			return nil, err
 		}

internal/api/api_test.go

@@ -88,27 +88,25 @@ func newDeadmanTS(t *testing.T, deadman api.DeadmanConfig, notify ...api.NotifyC
 	return s
 }
 
-// setTeamDeadman configures the default team's switches over the API, rendering
+// setTeamDeadman gives the default team one switch per configured matcher, over
-// the matchers back into the string form the endpoint takes.
+// the API, the way an owner would add them.
 func setTeamDeadman(t *testing.T, s *ts, cfg api.DeadmanConfig) {
 	t.Helper()
-	matchers := make([]string, 0, len(cfg.Matchers))
 	for _, m := range cfg.Matchers {
 		parts := []string{"alertname=" + m.Name}
 		for k, v := range m.Labels {
 			parts = append(parts, k+"="+v)
 		}
 		sort.Strings(parts[1:])
-		matchers = append(matchers, strings.Join(parts, ","))
+		resp := s.req(t, http.MethodPost, "/api/teams/"+defaultTeam+"/deadman/switches", map[string]any{
-	}
+			"matcher":         strings.Join(parts, ","),
-	resp := s.req(t, http.MethodPut, "/api/teams/"+defaultTeam+"/deadman", map[string]any{
+			"timeout_seconds": int64(cfg.Timeout.Seconds()),
-		"matchers":        strings.Join(matchers, "; "),
+			"severity":        cfg.Severity,
-		"timeout_seconds": int64(cfg.Timeout.Seconds()),
+		})
-		"severity":        cfg.Severity,
+		resp.Body.Close()
-	})
+		if resp.StatusCode != http.StatusCreated {
-	defer resp.Body.Close()
+			t.Fatalf("add a dead man's switch: %d", resp.StatusCode)
-	if resp.StatusCode != http.StatusOK {
+		}
-		t.Fatalf("configure the team's dead man's switches: %d", resp.StatusCode)
 	}
 }
 

internal/api/deadman.go

@@ -4,6 +4,8 @@ import (
 	"context"
 	"database/sql"
 	"encoding/json"
+	"errors"
+	"fmt"
 	"log"
 	"sort"
 	"strings"
@@ -39,6 +41,17 @@ func (m DeadmanMatcher) String() string {
 	return m.Name + " (" + strings.Join(parts, ", ") + ")"
 }
 
+// config renders the matcher in the form parseDeadmanMatcher reads, which is
+// what a switch row stores: `alertname=Watchdog,cluster=prod`.
+func (m DeadmanMatcher) config() string {
+	parts := make([]string, 0, len(m.Labels))
+	for k, v := range m.Labels {
+		parts = append(parts, k+"="+v)
+	}
+	sort.Strings(parts)
+	return strings.Join(append([]string{"alertname=" + m.Name}, parts...), ",")
+}
+
 // matches reports whether an alert's labels satisfy every condition.
 func (m DeadmanMatcher) matches(labels map[string]string) bool {
 	if labels["alertname"] != m.Name {
@@ -52,12 +65,10 @@ func (m DeadmanMatcher) matches(labels map[string]string) bool {
 	return true
 }
 
-// DeadmanConfig inverts the handling of the alerts it matches: receiving one
+// DeadmanConfig is the server-wide default a team's switches are seeded from:
-// opens nothing, and the absence of one opens an incident.
+// the environment's matchers, timeout and severity. Switches themselves are rows
-//
+// of a team's own — see DeadmanSwitch — and this is only how a fresh install
-// The unit of monitoring is the fingerprint, not the matcher — two clusters
+// starts out.
-// sending the same heartbeat alertname are two independent switches, so one
-// healthy cluster cannot mask a dead one.
 type DeadmanConfig struct {
 	Matchers []DeadmanMatcher
 
@@ -76,41 +87,81 @@ type DeadmanConfig struct {
 // enabled reports whether there is anything to watch.
 func (c DeadmanConfig) enabled() bool { return c.Timeout > 0 && len(c.Matchers) > 0 }
 
-// match returns the first matcher an alert satisfies.
+// DeadmanSwitch inverts the handling of the alerts it matches: receiving one
-func (c DeadmanConfig) match(labels map[string]string) (DeadmanMatcher, bool) {
+// opens nothing, and the absence of one opens an incident.
-	if !c.enabled() {
+//
-		return DeadmanMatcher{}, false
+// The unit of monitoring is the fingerprint, not the switch — two clusters
-	}
+// sending the same heartbeat alertname are two independent heartbeats under one
-	for _, m := range c.Matchers {
+// switch, so one healthy cluster cannot mask a dead one.
-		if m.matches(labels) {
+type DeadmanSwitch struct {
-			return m, true
+	ID      int64
-		}
+	Name    string
-	}
+	Matcher DeadmanMatcher
-	return DeadmanMatcher{}, false
+
+	// Timeout is how long a heartbeat may go unheard before it is declared dead.
+	Timeout time.Duration
+
+	// Severity is what the incident opens at.
+	Severity string
 }
 
-// isDeadman is match without the matcher, for the ingest path.
+// deadmanSet is one team's switches.
-func (c DeadmanConfig) isDeadman(labels map[string]string) bool {
+type deadmanSet []DeadmanSwitch
-	_, ok := c.match(labels)
+
+// match returns the first switch an alert satisfies.
+func (d deadmanSet) match(labels map[string]string) (DeadmanSwitch, bool) {
+	for _, sw := range d {
+		if sw.Matcher.matches(labels) {
+			return sw, true
+		}
+	}
+	return DeadmanSwitch{}, false
+}
+
+// isDeadman is match without the switch, for the ingest path.
+func (d deadmanSet) isDeadman(labels map[string]string) bool {
+	_, ok := d.match(labels)
 	return ok
 }
 
 // names lists the distinct alertnames worth loading from the database.
-func (c DeadmanConfig) names() []string {
+func (d deadmanSet) names() []string {
 	seen := map[string]bool{}
-	out := make([]string, 0, len(c.Matchers))
+	out := make([]string, 0, len(d))
-	for _, m := range c.Matchers {
+	for _, sw := range d {
-		if !seen[m.Name] {
+		if !seen[sw.Matcher.Name] {
-			seen[m.Name] = true
+			seen[sw.Matcher.Name] = true
-			out = append(out, m.Name)
+			out = append(out, sw.Matcher.Name)
 		}
 	}
 	return out
 }
 
+// parseDeadmanMatcher reads one matcher from its configured form: "," separates
+// the conditions and "=" is exact label equality — `alertname=Watchdog,cluster=prod`.
+// The error says what is wrong with it, in words a form can show.
+func parseDeadmanMatcher(entry string) (DeadmanMatcher, error) {
+	m := DeadmanMatcher{Labels: map[string]string{}}
+	for _, cond := range strings.Split(strings.TrimSpace(entry), ",") {
+		k, v, ok := strings.Cut(cond, "=")
+		k, v = strings.TrimSpace(k), strings.TrimSpace(v)
+		if !ok || k == "" || v == "" {
+			return DeadmanMatcher{}, fmt.Errorf("%q is not label=value", strings.TrimSpace(cond))
+		}
+		if k == "alertname" {
+			m.Name = v
+			continue
+		}
+		m.Labels[k] = v
+	}
+	if m.Name == "" {
+		return DeadmanMatcher{}, errors.New("no alertname condition")
+	}
+	return m, nil
+}
+
 // ParseDeadmanConfig reads the matcher list from its configured form:
-// ";" separates matchers, "," separates the conditions within one, and "=" is
+// ";" separates matchers, and each is parsed as parseDeadmanMatcher does.
-// exact label equality — `alertname=Watchdog,cluster=prod; alertname=Heartbeat`.
 //
 // A malformed or alertname-less entry is dropped rather than fatal, following
 // config.duration's rule that one bad tuning knob should not take the server
@@ -125,28 +176,9 @@ func ParseDeadmanConfig(matchers string, timeout time.Duration, severity string)
 		if entry == "" {
 			continue
 		}
-
+		m, err := parseDeadmanMatcher(entry)
-		m := DeadmanMatcher{Labels: map[string]string{}}
+		if err != nil {
-		malformed := false
+			log.Printf("deadman: ignoring matcher %q: %v", entry, err)
-		for _, cond := range strings.Split(entry, ",") {
-			k, v, ok := strings.Cut(cond, "=")
-			k, v = strings.TrimSpace(k), strings.TrimSpace(v)
-			if !ok || k == "" || v == "" {
-				log.Printf("deadman: ignoring matcher %q: %q is not label=value", entry, strings.TrimSpace(cond))
-				malformed = true
-				break
-			}
-			if k == "alertname" {
-				m.Name = v
-				continue
-			}
-			m.Labels[k] = v
-		}
-		if malformed {
-			continue
-		}
-		if m.Name == "" {
-			log.Printf("deadman: ignoring matcher %q: no alertname condition", entry)
 			continue
 		}
 		cfg.Matchers = append(cfg.Matchers, m)
@@ -162,23 +194,35 @@ func ParseDeadmanConfig(matchers string, timeout time.Duration, severity string)
 		for _, m := range cfg.Matchers {
 			rendered = append(rendered, m.String())
 		}
-		log.Printf("deadman: watching %s, timeout %s, severity %s",
+		log.Printf("deadman: default for new teams: %s, timeout %s, severity %s",
 			strings.Join(rendered, "; "), timeout, severity)
 	}
 	return cfg
 }
 
-// deadmanAlert is one switch: the alert row carrying its last heartbeat.
+// deadmanAlert is one heartbeat: the alert row carrying its last sighting, and
+// the switch that claimed it.
 type deadmanAlert struct {
 	id          int64
 	teamID      int64
 	fingerprint string
 	labels      map[string]string
-	matcher     DeadmanMatcher
+	sw          DeadmanSwitch
 	resolved    bool
 	receivedAt  int64
 }
 
+// dead is the one rule for a silent heartbeat, shared by the sweeper that pages
+// on it and the status the Switches page shows, so the page cannot disagree
+// with the pager.
+//
+// An explicit resolved from Alertmanager is a stronger death signal than mere
+// absence: the sender is telling us the heartbeat stopped, so there is nothing
+// left to wait out.
+func (a deadmanAlert) dead(now time.Time) bool {
+	return a.resolved || a.receivedAt < now.Add(-a.sw.Timeout).Unix()
+}
+
 // groupKey is the switch's identity as an incident. Per fingerprint, so each
 // source is tracked on its own.
 func (a deadmanAlert) groupKey() string { return deadmanGroupPrefix + a.fingerprint }
@@ -189,13 +233,13 @@ func (a deadmanAlert) groupKey() string { return deadmanGroupPrefix + a.fingerpr
 // It returns the ids of the alerts it owns, because the generic staleness
 // expiry must leave them alone — staleAfter and ends_at would otherwise resolve
 // a heartbeat long before its own, much tighter, timeout ever fired.
-// Each team is swept against its own configuration: its own matchers, its own
+// Each team is swept against its own switches, each with its own matcher,
-// timeout, its own severity. A team watching nothing is skipped entirely, which
+// timeout and severity. A team watching nothing is skipped entirely, which is
-// is most of them.
+// most of them.
 func sweepDeadman(ctx context.Context, db *sql.DB, notify NotifyConfig) map[int64]bool {
 	owned := map[int64]bool{}
 
-	configs, err := deadmanConfigs(ctx, db)
+	configs, err := deadmanSets(ctx, db)
 	if err != nil {
 		log.Printf("deadman: load configs: %v", err)
 		return owned
@@ -203,39 +247,35 @@ func sweepDeadman(ctx context.Context, db *sql.DB, notify NotifyConfig) map[int6
 
 	now := time.Now()
 	for teamID, cfg := range configs {
-		switches, err := deadmanAlerts(ctx, db, teamID, cfg)
+		heartbeats, err := deadmanAlerts(ctx, db, teamID, cfg)
 		if err != nil {
-			log.Printf("deadman: load switches for team %d: %v", teamID, err)
+			log.Printf("deadman: load heartbeats for team %d: %v", teamID, err)
 			continue
 		}
-		cutoff := now.Add(-cfg.Timeout).Unix()
 
-		for _, sw := range switches {
+		for _, hb := range heartbeats {
-			owned[sw.id] = true
+			owned[hb.id] = true
 
-			// An explicit resolved from Alertmanager is a stronger death signal
+			if hb.dead(now) {
-			// than mere absence: the sender is telling us the heartbeat
+				if err := deadmanDied(ctx, db, notify, hb, now); err != nil {
-			// stopped, so there is nothing left to wait out.
+					log.Printf("deadman: open incident for %s: %v", hb.sw.Matcher.Name, err)
-			if sw.resolved || sw.receivedAt < cutoff {
-				if err := deadmanDied(ctx, db, cfg, notify, sw, now); err != nil {
-					log.Printf("deadman: open incident for %s: %v", sw.matcher.Name, err)
 				}
 				continue
 			}
-			if err := deadmanRecovered(ctx, db, sw); err != nil {
+			if err := deadmanRecovered(ctx, db, hb); err != nil {
-				log.Printf("deadman: resolve incident for %s: %v", sw.matcher.Name, err)
+				log.Printf("deadman: resolve incident for %s: %v", hb.sw.Matcher.Name, err)
 			}
 		}
 	}
 	return owned
 }
 
-// deadmanAlerts loads every alert row that a matcher claims. The candidate query
+// deadmanAlerts loads every alert row that one of a team's switches claims. The candidate query
 // is narrowed by alertname so it rides alerts_name_idx; the rest of the matching
 // happens in Go, which keeps one implementation of the rules. The rows are read
 // in full before the caller writes, so the writes do not run against an open
 // cursor over the same table.
-func deadmanAlerts(ctx context.Context, db *sql.DB, teamID int64, cfg DeadmanConfig) ([]deadmanAlert, error) {
+func deadmanAlerts(ctx context.Context, db *sql.DB, teamID int64, cfg deadmanSet) ([]deadmanAlert, error) {
 	names := cfg.names()
 	args := &sqlArgs{}
 	nameList := make([]any, len(names))
@@ -263,11 +303,11 @@ func deadmanAlerts(ctx context.Context, db *sql.DB, teamID int64, cfg DeadmanCon
 		}
 		json.Unmarshal([]byte(labelsJSON), &a.labels) //nolint:errcheck
 
-		m, ok := cfg.match(a.labels)
+		sw, ok := cfg.match(a.labels)
 		if !ok {
 			continue
 		}
-		a.matcher = m
+		a.sw = sw
 		a.resolved = status == "resolved"
 		out = append(out, a)
 	}
@@ -284,16 +324,16 @@ func deadmanAlerts(ctx context.Context, db *sql.DB, teamID int64, cfg DeadmanCon
 // incidentForGroup), and a source that is gone for good is a one-time page
 // rather than a nag. Only a heartbeat that comes back and dies again earns a new
 // incident.
-func deadmanDied(ctx context.Context, db *sql.DB, cfg DeadmanConfig, notify NotifyConfig, sw deadmanAlert, now time.Time) error {
+func deadmanDied(ctx context.Context, db *sql.DB, notify NotifyConfig, hb deadmanAlert, now time.Time) error {
 	var lastTriggered, open int64
 	if err := db.QueryRowContext(ctx, `
 		SELECT COALESCE(MAX(triggered_at), 0),
 		       COUNT(*) FILTER (WHERE resolved_at IS NULL)
 		FROM   incidents WHERE team_id = $1 AND group_key = $2`,
-		sw.teamID, sw.groupKey()).Scan(&lastTriggered, &open); err != nil {
+		hb.teamID, hb.groupKey()).Scan(&lastTriggered, &open); err != nil {
 		return err
 	}
-	if open > 0 || sw.receivedAt <= lastTriggered {
+	if open > 0 || hb.receivedAt <= lastTriggered {
 		return nil
 	}
 
@@ -306,18 +346,18 @@ func deadmanDied(ctx context.Context, db *sql.DB, cfg DeadmanConfig, notify Noti
 	// A heartbeat nobody has heard from is not firing, and saying otherwise in
 	// the alert list would be a lie. An Alertmanager-sourced resolution keeps its
 	// own source: it told us the truth first.
-	if !sw.resolved {
+	if !hb.resolved {
 		if _, err := tx.ExecContext(ctx, `
 			UPDATE alerts
 			   SET status            = 'resolved',
 			       resolution_source = $1,
 			       ends_at           = COALESCE(ends_at, `+nowEpoch+`)
-			 WHERE id = $2 AND status = 'firing'`, resolutionDeadman, sw.id); err != nil {
+			 WHERE id = $2 AND status = 'firing'`, resolutionDeadman, hb.id); err != nil {
 			return err
 		}
 	}
 
-	severity := cfg.Severity
+	severity := hb.sw.Severity
 	var sev *string
 	if severity != "" {
 		sev = &severity
@@ -325,14 +365,14 @@ func deadmanDied(ctx context.Context, db *sql.DB, cfg DeadmanConfig, notify Noti
 
 	// The incident opens in the team whose integration received the heartbeat:
 	// the switch belongs to whoever is watching that source, not to the install.
-	incidentID, err := openIncident(ctx, tx, notify, sw.teamID, sw.groupKey(),
+	incidentID, err := openIncident(ctx, tx, notify, hb.teamID, hb.groupKey(),
-		"No heartbeat from "+sw.matcher.String(), sw.labels, sev)
+		"No heartbeat from "+hb.sw.Matcher.String(), hb.labels, sev)
 	if err != nil {
 		return err
 	}
 
-	alertID := sw.id
+	alertID := hb.id
-	detail := "last heartbeat " + humanDuration(now.Sub(time.Unix(sw.receivedAt, 0))) + " ago"
+	detail := "last heartbeat " + humanDuration(now.Sub(time.Unix(hb.receivedAt, 0))) + " ago"
 	if err := logEvent(ctx, tx, incidentID, evDeadmanSilent, nil, &alertID, &detail); err != nil {
 		return err
 	}
@@ -340,7 +380,7 @@ func deadmanDied(ctx context.Context, db *sql.DB, cfg DeadmanConfig, notify Noti
 	if err := tx.Commit(); err != nil {
 		return err
 	}
-	log.Printf("deadman: %s went silent, opened incident %d", sw.matcher.String(), incidentID)
+	log.Printf("deadman: %s went silent, opened incident %d", hb.sw.Matcher.String(), incidentID)
 	return nil
 }
 
@@ -350,12 +390,12 @@ func deadmanDied(ctx context.Context, db *sql.DB, cfg DeadmanConfig, notify Noti
 // member alerts (linking the heartbeat would have the settled-incident cascade
 // close it on the very same sweep that opened it), so the alert-driven cascade
 // ignores it entirely and recovery is the only automatic way out.
-func deadmanRecovered(ctx context.Context, db *sql.DB, sw deadmanAlert) error {
+func deadmanRecovered(ctx context.Context, db *sql.DB, hb deadmanAlert) error {
 	var incidentID int64
 	switch err := db.QueryRowContext(ctx, `
 		SELECT id FROM incidents
 		 WHERE team_id = $1 AND group_key = $2 AND resolved_at IS NULL`,
-		sw.teamID, sw.groupKey()).Scan(&incidentID); {
+		hb.teamID, hb.groupKey()).Scan(&incidentID); {
 	case err == sql.ErrNoRows:
 		return nil
 	case err != nil:
@@ -387,116 +427,256 @@ func deadmanRecovered(ctx context.Context, db *sql.DB, sw deadmanAlert) error {
 	if err := tx.Commit(); err != nil {
 		return err
 	}
-	log.Printf("deadman: %s is back, resolved incident %d", sw.matcher.String(), incidentID)
+	log.Printf("deadman: %s is back, resolved incident %d", hb.sw.Matcher.String(), incidentID)
 	return nil
 }
 
 // ---------------------------------------------------------------------------
-// Per-team configuration
+// A team's switches
 // ---------------------------------------------------------------------------
 
-// deadmanConfigForTeam reads one team's switches. A team with no row, or with
+const deadmanSwitchColumns = "id, team_id, name, matcher, timeout_seconds, severity"
-// nothing configured, gets a disabled config — which is the right answer rather
-// than an error: most teams watch no heartbeat at all.
-func deadmanConfigForTeam(ctx context.Context, q querier, teamID int64) (DeadmanConfig, error) {
-	var matchers, severity string
-	var timeout int64
-	err := q.QueryRowContext(ctx,
-		"SELECT matchers, timeout_seconds, severity FROM deadman_configs WHERE team_id = $1",
-		teamID).Scan(&matchers, &timeout, &severity)
-	if err == sql.ErrNoRows {
-		return DeadmanConfig{}, nil
-	}
-	if err != nil {
-		return DeadmanConfig{}, err
-	}
-	return parseDeadmanQuietly(matchers, time.Duration(timeout)*time.Second, severity), nil
-}
 
-// deadmanConfigs reads every team's switches in one query, for the sweeper.
+// scanDeadmanSwitches reads switch rows into per-team sets. A row whose matcher
-func deadmanConfigs(ctx context.Context, db *sql.DB) (map[int64]DeadmanConfig, error) {
+// no longer parses is skipped rather than fatal: the API refuses to store one,
-	rows, err := db.QueryContext(ctx,
+// so it can only mean a hand edit, and one bad row must not stop the others
-		"SELECT team_id, matchers, timeout_seconds, severity FROM deadman_configs")
+// from being watched.
-	if err != nil {
+func scanDeadmanSwitches(rows *sql.Rows) (map[int64]deadmanSet, error) {
-		return nil, err
-	}
 	defer rows.Close()
-
+	out := map[int64]deadmanSet{}
-	out := map[int64]DeadmanConfig{}
 	for rows.Next() {
+		var sw DeadmanSwitch
 		var teamID, timeout int64
-		var matchers, severity string
+		var matcher string
-		if err := rows.Scan(&teamID, &matchers, &timeout, &severity); err != nil {
+		if err := rows.Scan(&sw.ID, &teamID, &sw.Name, &matcher, &timeout, &sw.Severity); err != nil {
 			return nil, err
 		}
-		cfg := parseDeadmanQuietly(matchers, time.Duration(timeout)*time.Second, severity)
+		m, err := parseDeadmanMatcher(matcher)
-		if cfg.enabled() {
+		if err != nil {
-			out[teamID] = cfg
+			log.Printf("deadman: switch %d has an unusable matcher %q: %v", sw.ID, matcher, err)
+			continue
 		}
+		sw.Matcher = m
+		sw.Timeout = time.Duration(timeout) * time.Second
+		out[teamID] = append(out[teamID], sw)
 	}
 	return out, rows.Err()
 }
 
-// SeedDeadmanConfigs gives every team without a row the server's environment
+// deadmanSetForTeam reads one team's switches. A team with none gets an empty
-// configuration, so the install that upgrades into per-team switches keeps
+// set — which is the right answer rather than an error: most teams watch no
-// watching exactly what it was watching before.
+// heartbeat at all.
+func deadmanSetForTeam(ctx context.Context, q querier, teamID int64) (deadmanSet, error) {
+	rows, err := q.QueryContext(ctx,
+		"SELECT "+deadmanSwitchColumns+" FROM deadman_switches WHERE team_id = $1 ORDER BY id", teamID)
+	if err != nil {
+		return nil, err
+	}
+	sets, err := scanDeadmanSwitches(rows)
+	return sets[teamID], err
+}
+
+// deadmanSets reads every team's switches in one query, for the sweeper.
+func deadmanSets(ctx context.Context, db *sql.DB) (map[int64]deadmanSet, error) {
+	rows, err := db.QueryContext(ctx,
+		"SELECT "+deadmanSwitchColumns+" FROM deadman_switches ORDER BY id")
+	if err != nil {
+		return nil, err
+	}
+	return scanDeadmanSwitches(rows)
+}
+
+// deadmanSeededKey is the settings row that records the environment defaults
+// were handed out. Without it, a team that deleted its last switch would get
+// the default back on the next restart.
+const deadmanSeededKey = "deadman_seeded"
+
+// SeedDeadmanConfigs gives every team the server's environment defaults as
+// switches, exactly once per install, so a fresh install watches Watchdog
+// without anybody setting it up.
 //
-// Idempotent, and never overwrites: once a team has a row it owns its own
+// Once seeded it never runs again: a team's switches are its own, and a redeploy
-// configuration, and a redeploy must not quietly put the environment's value
+// must not quietly put the environment's value back over an owner's edit or
-// back over an owner's edit.
+// deletion. Installs that upgraded from per-team configuration were already
+// seeded, which migration 009 records.
 //
-// A team created after startup gets no row and therefore watches nothing until
+// A team created after that gets none and watches nothing until its owner says
-// its owner says otherwise. That is deliberate: inheriting an install-wide
+// otherwise. That is deliberate: inheriting an install-wide heartbeat would page
-// heartbeat would page a new team about a source it has never heard of, and a
+// a new team about a source it has never heard of, and a switch nobody chose is
-// switch nobody chose is the kind that gets muted rather than fixed.
+// the kind that gets muted rather than fixed.
 func SeedDeadmanConfigs(ctx context.Context, db *sql.DB, cfg DeadmanConfig) error {
-	matchers := make([]string, 0, len(cfg.Matchers))
+	if !cfg.enabled() {
+		return nil
+	}
+
+	tx, err := db.BeginTx(ctx, nil)
+	if err != nil {
+		return err
+	}
+	defer tx.Rollback() //nolint:errcheck
+
+	res, err := tx.ExecContext(ctx,
+		"INSERT INTO settings (key, value) VALUES ($1, '1') ON CONFLICT (key) DO NOTHING",
+		deadmanSeededKey)
+	if err != nil {
+		return err
+	}
+	if n, _ := res.RowsAffected(); n == 0 {
+		return nil
+	}
+
 	for _, m := range cfg.Matchers {
-		parts := []string{"alertname=" + m.Name}
+		if _, err := tx.ExecContext(ctx, `
-		for k, v := range m.Labels {
+			INSERT INTO deadman_switches (team_id, name, matcher, timeout_seconds, severity)
-			parts = append(parts, k+"="+v)
+			SELECT id, $1, $1, $2, $3 FROM teams`,
+			m.config(), int64(cfg.Timeout.Seconds()), cfg.Severity); err != nil {
+			return err
 		}
-		sort.Strings(parts[1:])
-		matchers = append(matchers, strings.Join(parts, ","))
 	}
-
+	return tx.Commit()
-	_, err := db.ExecContext(ctx, `
-		INSERT INTO deadman_configs (team_id, matchers, timeout_seconds, severity)
-		SELECT id, $1, $2, $3 FROM teams
-		ON CONFLICT (team_id) DO NOTHING`,
-		strings.Join(matchers, "; "), int64(cfg.Timeout.Seconds()), cfg.Severity)
-	return err
 }
 
-// parseDeadmanQuietly is ParseDeadmanConfig without the startup logging: a
+// ---------------------------------------------------------------------------
-// team's configuration is read on every sweep and every webhook, and logging it
+// Status
-// each time would bury everything else.
+// ---------------------------------------------------------------------------
-func parseDeadmanQuietly(matchers string, timeout time.Duration, severity string) DeadmanConfig {
+
-	cfg := DeadmanConfig{Timeout: timeout, Severity: severity}
+const (
-	for _, entry := range strings.Split(matchers, ";") {
+	switchHealthy = "healthy"
-		entry = strings.TrimSpace(entry)
+	switchDead    = "dead"
-		if entry == "" {
+	switchDormant = "dormant"
-			continue
+)
-		}
+
-		m := DeadmanMatcher{Labels: map[string]string{}}
+// deadmanSource is one heartbeat under a switch: a fingerprint that matched.
-		malformed := false
+type deadmanSource struct {
-		for _, cond := range strings.Split(entry, ",") {
+	Fingerprint     string            `json:"fingerprint"`
-			k, v, ok := strings.Cut(cond, "=")
+	Labels          map[string]string `json:"labels"`
-			k, v = strings.TrimSpace(k), strings.TrimSpace(v)
+	Status          string            `json:"status"`
-			if !ok || k == "" || v == "" {
+	LastHeartbeatAt time.Time         `json:"last_heartbeat_at"`
-				malformed = true
+	LastTriggeredAt *time.Time        `json:"last_triggered_at"`
-				break
+	IncidentID      *int64            `json:"incident_id"`
-			}
+}
-			if k == "alertname" {
+
-				m.Name = v
+// deadmanSwitchStatus is a switch as the Switches page shows it.
-				continue
+type deadmanSwitchStatus struct {
-			}
+	ID             int64  `json:"id"`
-			m.Labels[k] = v
+	Name           string `json:"name"`
-		}
+	Matcher        string `json:"matcher"`
-		if malformed || m.Name == "" {
+	TimeoutSeconds int64  `json:"timeout_seconds"`
-			continue
+	Severity       string `json:"severity"`
-		}
+
-		cfg.Matchers = append(cfg.Matchers, m)
+	// Status is dead when any source is, dormant when none has ever been heard
-	}
+	// from, healthy otherwise — a live cluster must not hide a dead one.
-	return cfg
+	Status          string          `json:"status"`
+	LastHeartbeatAt *time.Time      `json:"last_heartbeat_at"`
+	LastTriggeredAt *time.Time      `json:"last_triggered_at"`
+	OpenIncidentID  *int64          `json:"open_incident_id"`
+	Sources         []deadmanSource `json:"sources"`
+}
+
+// deadmanStatuses reports every switch of a team with what its heartbeats are
+// doing. The liveness verdict is deadmanAlert.dead, the sweeper's own.
+func deadmanStatuses(ctx context.Context, db *sql.DB, teamID int64, set deadmanSet, now time.Time) ([]deadmanSwitchStatus, error) {
+	out := make([]deadmanSwitchStatus, 0, len(set))
+	if len(set) == 0 {
+		return out, nil
+	}
+
+	heartbeats, err := deadmanAlerts(ctx, db, teamID, set)
+	if err != nil {
+		return nil, err
+	}
+
+	// One query for every switch's incident history, keyed the way the sweeper
+	// keys it.
+	type history struct {
+		triggeredAt int64
+		openID      int64
+	}
+	incidents := map[string]history{}
+	rows, err := db.QueryContext(ctx, `
+		SELECT group_key, MAX(triggered_at), COALESCE(MAX(id) FILTER (WHERE resolved_at IS NULL), 0)
+		FROM   incidents
+		WHERE  team_id = $1 AND group_key LIKE $2
+		GROUP  BY group_key`, teamID, deadmanGroupPrefix+"%")
+	if err != nil {
+		return nil, err
+	}
+	defer rows.Close()
+	for rows.Next() {
+		var key string
+		var h history
+		if err := rows.Scan(&key, &h.triggeredAt, &h.openID); err != nil {
+			return nil, err
+		}
+		incidents[key] = h
+	}
+	if err := rows.Err(); err != nil {
+		return nil, err
+	}
+
+	bySwitch := map[int64][]deadmanAlert{}
+	for _, hb := range heartbeats {
+		bySwitch[hb.sw.ID] = append(bySwitch[hb.sw.ID], hb)
+	}
+
+	later := func(cur *time.Time, unix int64) *time.Time {
+		t := time.Unix(unix, 0).UTC()
+		if cur == nil || t.After(*cur) {
+			return &t
+		}
+		return cur
+	}
+
+	for _, sw := range set {
+		st := deadmanSwitchStatus{
+			ID: sw.ID, Name: sw.Name, Matcher: sw.Matcher.config(),
+			TimeoutSeconds: int64(sw.Timeout.Seconds()), Severity: sw.Severity,
+			Status: switchDormant, Sources: []deadmanSource{},
+		}
+
+		for _, hb := range bySwitch[sw.ID] {
+			src := deadmanSource{
+				Fingerprint:     hb.fingerprint,
+				Labels:          hb.labels,
+				Status:          switchHealthy,
+				LastHeartbeatAt: time.Unix(hb.receivedAt, 0).UTC(),
+			}
+			if hb.dead(now) {
+				src.Status = switchDead
+			}
+			if h, ok := incidents[hb.groupKey()]; ok {
+				t := time.Unix(h.triggeredAt, 0).UTC()
+				src.LastTriggeredAt = &t
+				st.LastTriggeredAt = later(st.LastTriggeredAt, h.triggeredAt)
+				if h.openID != 0 {
+					id := h.openID
+					src.IncidentID = &id
+					if st.OpenIncidentID == nil || id > *st.OpenIncidentID {
+						st.OpenIncidentID = &id
+					}
+				}
+			}
+			st.LastHeartbeatAt = later(st.LastHeartbeatAt, hb.receivedAt)
+			st.Sources = append(st.Sources, src)
+
+			switch {
+			case src.Status == switchDead:
+				st.Status = switchDead
+			case st.Status == switchDormant:
+				st.Status = switchHealthy
+			}
+		}
+
+		// Dead ones first, then by fingerprint: what needs attention leads, and
+		// the order does not shuffle between refreshes.
+		sort.Slice(st.Sources, func(i, j int) bool {
+			a, b := st.Sources[i], st.Sources[j]
+			if (a.Status == switchDead) != (b.Status == switchDead) {
+				return a.Status == switchDead
+			}
+			return a.Fingerprint < b.Fingerprint
+		})
+		out = append(out, st)
+	}
+	return out, nil
 }

internal/api/deadman_test.go

@@ -1,6 +1,7 @@
 package api_test
 
 import (
+	"context"
 	"net/http"
 	"strings"
 	"testing"
@@ -483,13 +484,13 @@ func TestDeadman_ConfigurationIsPerTeam(t *testing.T) {
 	unwatched := newTeam(t, s, "unwatched")
 
 	// Only the first team calls Watchdog a heartbeat.
-	resp := s.req(t, http.MethodPut, "/api/teams/"+id64(watched.id)+"/deadman", map[string]any{
+	resp := s.req(t, http.MethodPost, "/api/teams/"+id64(watched.id)+"/deadman/switches", map[string]any{
-		"matchers":        "alertname=Watchdog",
+		"matcher":         "alertname=Watchdog",
 		"timeout_seconds": 3600,
 		"severity":        "critical",
 	})
 	resp.Body.Close()
-	if resp.StatusCode != http.StatusOK {
+	if resp.StatusCode != http.StatusCreated {
 		t.Fatalf("configure the watched team: %d", resp.StatusCode)
 	}
 
@@ -550,9 +551,9 @@ func TestDeadman_ConfigurationIsOwnerOnly(t *testing.T) {
 	decode(t, s.req(t, http.MethodPost, "/api/users/"+id64(user.ID)+"/api-keys",
 		map[string]string{"name": "test"}), &key)
 
-	req, _ := http.NewRequest(http.MethodPut,
+	req, _ := http.NewRequest(http.MethodPost,
-		s.URL+"/api/teams/"+id64(team.id)+"/deadman",
+		s.URL+"/api/teams/"+id64(team.id)+"/deadman/switches",
-		strings.NewReader(`{"matchers":"alertname=Watchdog","timeout_seconds":60}`))
+		strings.NewReader(`{"matcher":"alertname=Watchdog","timeout_seconds":60}`))
 	req.Header.Set("Authorization", "Bearer "+key.Key)
 	req.Header.Set("Content-Type", "application/json")
 	resp, err := http.DefaultClient.Do(req)
@@ -564,7 +565,7 @@ func TestDeadman_ConfigurationIsOwnerOnly(t *testing.T) {
 		t.Errorf("a member editing the switches: expected 403, got %d", resp.StatusCode)
 	}
 
-	read, _ := http.NewRequest(http.MethodGet, s.URL+"/api/teams/"+id64(team.id)+"/deadman", nil)
+	read, _ := http.NewRequest(http.MethodGet, s.URL+"/api/teams/"+id64(team.id)+"/deadman/switches", nil)
 	read.Header.Set("Authorization", "Bearer "+key.Key)
 	got, err := http.DefaultClient.Do(read)
 	if err != nil {
@@ -577,16 +578,165 @@ func TestDeadman_ConfigurationIsOwnerOnly(t *testing.T) {
 }
 
 // A matcher with no alertname watches nothing, silently, which is the failure
-// this feature exists to prevent — so it is refused at the door.
+// this feature exists to prevent — so it is refused at the door, along with the
-func TestDeadman_UnusableMatchersAreRejected(t *testing.T) {
+// other things that would make a switch unable to fire.
+func TestDeadman_UnusableSwitchesAreRejected(t *testing.T) {
 	s, _ := deadmanTS(t, deadmanCfg())
 
-	resp := s.req(t, http.MethodPut, "/api/teams/"+defaultTeam+"/deadman", map[string]any{
+	for name, body := range map[string]map[string]any{
-		"matchers":        "cluster=prod",
+		"no alertname":  {"matcher": "cluster=prod", "timeout_seconds": 900},
-		"timeout_seconds": 900,
+		"malformed":     {"matcher": "alertname=Watchdog,garbage", "timeout_seconds": 900},
-	})
+		"several":       {"matcher": "alertname=A; alertname=B", "timeout_seconds": 900},
-	resp.Body.Close()
+		"zero timeout":  {"matcher": "alertname=Watchdog", "timeout_seconds": 0},
-	if resp.StatusCode != http.StatusBadRequest {
+		"bad severity":  {"matcher": "alertname=Watchdog", "timeout_seconds": 900, "severity": "loud"},
-		t.Errorf("expected 400 for a matcher with no alertname, got %d", resp.StatusCode)
+		"empty matcher": {"matcher": "", "timeout_seconds": 900},
+	} {
+		resp := s.req(t, http.MethodPost, "/api/teams/"+defaultTeam+"/deadman/switches", body)
+		resp.Body.Close()
+		if resp.StatusCode != http.StatusBadRequest {
+			t.Errorf("%s: expected 400, got %d", name, resp.StatusCode)
+		}
+	}
+}
+
+// ---------------------------------------------------------------------------
+// The switch list
+// ---------------------------------------------------------------------------
+
+// listSwitches reads the default team's switches as the Switches page does.
+func listSwitches(t *testing.T, s *ts) []map[string]any {
+	t.Helper()
+	return list(t, s.req(t, http.MethodGet, "/api/teams/"+defaultTeam+"/deadman/switches", nil))
+}
+
+// A switch is healthy while its heartbeat is fresh, dead once it is silent, and
+// dormant until the first one arrives.
+func TestDeadman_ListReportsStatus(t *testing.T) {
+	s, _ := deadmanTS(t, api.ParseDeadmanConfig("alertname=Watchdog; alertname=NeverSent", time.Hour, "critical"))
+
+	got := listSwitches(t, s)
+	if len(got) != 2 {
+		t.Fatalf("expected 2 switches, got %d", len(got))
+	}
+	for _, sw := range got {
+		if sw["status"] != "dormant" || sw["last_heartbeat_at"] != nil || sw["last_triggered_at"] != nil {
+			t.Errorf("a switch nobody has heard from should be dormant and blank, got %v", sw)
+		}
+	}
+
+	heartbeat(t, s, "fp-watchdog", nil)
+	got = listSwitches(t, s)
+	if got[0]["status"] != "healthy" || got[0]["last_heartbeat_at"] == nil {
+		t.Errorf("a fresh heartbeat should be healthy with a timestamp, got %v", got[0])
+	}
+	if got[1]["status"] != "dormant" {
+		t.Errorf("the other switch is still dormant, got %v", got[1]["status"])
+	}
+
+	silence(t, s, "fp-watchdog", 2*time.Hour)
+	sweep(t, s, noArchive)
+	got = listSwitches(t, s)
+	if got[0]["status"] != "dead" {
+		t.Fatalf("a silent heartbeat should be dead, got %v", got[0]["status"])
+	}
+	if got[0]["last_triggered_at"] == nil || got[0]["open_incident_id"] == nil {
+		t.Errorf("a dead switch should show when it triggered and its open incident, got %v", got[0])
+	}
+}
+
+// One matcher, several clusters: the switch is as bad as its worst heartbeat and
+// each heartbeat is listed on its own.
+func TestDeadman_ListBreaksDownByFingerprint(t *testing.T) {
+	s, _ := deadmanTS(t, deadmanCfg())
+
+	heartbeat(t, s, "fp-a", map[string]string{"cluster": "a"})
+	heartbeat(t, s, "fp-b", map[string]string{"cluster": "b"})
+	silence(t, s, "fp-b", 2*time.Hour)
+
+	sw := listSwitches(t, s)[0]
+	if sw["status"] != "dead" {
+		t.Errorf("one dead cluster makes the switch dead, got %v", sw["status"])
+	}
+	sources := sw["sources"].([]any)
+	if len(sources) != 2 {
+		t.Fatalf("expected 2 sources, got %d", len(sources))
+	}
+	first, second := sources[0].(map[string]any), sources[1].(map[string]any)
+	if first["fingerprint"] != "fp-b" || first["status"] != "dead" || second["status"] != "healthy" {
+		t.Errorf("the dead source should lead, got %v then %v", first, second)
+	}
+}
+
+// Every switch keeps its own deadline.
+func TestDeadman_TimeoutsArePerSwitch(t *testing.T) {
+	s, _ := deadmanTS(t, deadmanCfg())
+	resp := s.req(t, http.MethodPost, "/api/teams/"+defaultTeam+"/deadman/switches", map[string]any{
+		"matcher": "alertname=Edge", "timeout_seconds": 300,
+	})
+	resp.Body.Close()
+
+	heartbeat(t, s, "fp-watchdog", nil)
+	postWebhook(t, s, []map[string]any{
+		amAlert("fp-edge", "Edge", "firing", "2026-05-20T10:00:00Z", zeroTime, nil),
+	}, `{}:{alertname="Edge"}`)
+
+	// Ten minutes of silence: past the Edge switch's five, inside Watchdog's hour.
+	silence(t, s, "fp-watchdog", 10*time.Minute)
+	silence(t, s, "fp-edge", 10*time.Minute)
+
+	got := listSwitches(t, s)
+	if got[0]["status"] != "healthy" || got[1]["status"] != "dead" {
+		t.Errorf("want Watchdog healthy and Edge dead, got %v and %v", got[0]["status"], got[1]["status"])
+	}
+}
+
+// Deleting is an owner's, is scoped to the team, and leaves what the switch
+// already opened alone.
+func TestDeadman_DeleteIsScopedToTheTeam(t *testing.T) {
+	s, _ := deadmanTS(t, deadmanCfg())
+	other := newTeam(t, s, "other")
+
+	id := int64(listSwitches(t, s)[0]["id"].(float64))
+
+	// Another team's owner cannot reach it.
+	resp := other.call(http.MethodDelete, "/api/teams/"+id64(other.id)+"/deadman/switches/"+id64(id), nil)
+	resp.Body.Close()
+	if resp.StatusCode != http.StatusNotFound {
+		t.Errorf("deleting another team's switch: expected 404, got %d", resp.StatusCode)
+	}
+	if got := len(listSwitches(t, s)); got != 1 {
+		t.Fatalf("the switch should have survived, %d left", got)
+	}
+
+	resp = s.req(t, http.MethodDelete, "/api/teams/"+defaultTeam+"/deadman/switches/"+id64(id), nil)
+	resp.Body.Close()
+	if resp.StatusCode != http.StatusNoContent {
+		t.Fatalf("deleting: expected 204, got %d", resp.StatusCode)
+	}
+	if got := len(listSwitches(t, s)); got != 0 {
+		t.Errorf("expected no switches, got %d", got)
+	}
+}
+
+// The environment's defaults are handed out once and then belong to the teams.
+func TestDeadman_SeedRunsOnce(t *testing.T) {
+	s := newTS(t)
+	cfg := api.ParseDeadmanConfig("alertname=Watchdog", time.Hour, "critical")
+
+	if err := api.SeedDeadmanConfigs(context.Background(), s.db, cfg); err != nil {
+		t.Fatalf("seed: %v", err)
+	}
+	if got := len(listSwitches(t, s)); got != 1 {
+		t.Fatalf("the first seed should add the default, got %d switches", got)
+	}
+
+	// The owner deletes it; a restart must not put it back.
+	id := int64(listSwitches(t, s)[0]["id"].(float64))
+	s.req(t, http.MethodDelete, "/api/teams/"+defaultTeam+"/deadman/switches/"+id64(id), nil).Body.Close()
+	if err := api.SeedDeadmanConfigs(context.Background(), s.db, cfg); err != nil {
+		t.Fatalf("seed again: %v", err)
+	}
+	if got := len(listSwitches(t, s)); got != 0 {
+		t.Errorf("a second seed resurrected %d switch(es)", got)
 	}
 }

internal/api/router.go

@@ -144,12 +144,14 @@ func NewRouter(db *sql.DB, notify NotifyConfig, cfg config.Config) http.Handler
 
 		// A team's own dead man's switches: which of its alerts are heartbeats,
 		// and how long a silence has to last before somebody is paged.
-		r.Get("/api/teams/{teamID}/deadman", handleGetTeamDeadman(db))
+		r.Get("/api/teams/{teamID}/deadman/switches", handleListTeamDeadman(db))
-		r.Put("/api/teams/{teamID}/deadman", handleSetTeamDeadman(db))
+		r.Post("/api/teams/{teamID}/deadman/switches", handleCreateTeamDeadman(db))
+		r.Delete("/api/teams/{teamID}/deadman/switches/{switchID}", handleDeleteTeamDeadman(db))
 
 		// Integrations: where a team's alerts come in, and the key that says so.
 		r.Get("/api/teams/{teamID}/integrations", handleListIntegrations(db))
 		r.Post("/api/teams/{teamID}/integrations", handleCreateIntegration(db, notify.PublicURL))
+		r.Patch("/api/teams/{teamID}/integrations/{integrationID}", handleRenameIntegration(db))
 		r.Delete("/api/teams/{teamID}/integrations/{integrationID}", handleDeleteIntegration(db))
 
 		// The rota is per team. /api/schedule/current is the exception: it

internal/api/sources_test.go

@@ -0,0 +1,171 @@
+package api_test
+
+import (
+	"bytes"
+	"net/http"
+	"strings"
+	"testing"
+	"time"
+)
+
+// listSources reads a team's alert sources as the Sources page does.
+func listSources(t *testing.T, tm teamFixture) []map[string]any {
+	t.Helper()
+	return list(t, tm.call(http.MethodGet, "/api/teams/"+id64(tm.id)+"/integrations", nil))
+}
+
+// addSource mints a second source in a team and returns its key.
+func addSource(t *testing.T, tm teamFixture, name string) string {
+	t.Helper()
+	var out struct {
+		Key string `json:"key"`
+	}
+	decode(t, tm.call(http.MethodPost, "/api/teams/"+id64(tm.id)+"/integrations",
+		map[string]string{"name": name}), &out)
+	return out.Key
+}
+
+// A source that has never posted is "never", with nothing to say about alerts.
+func TestSources_NeverUsedIsBlank(t *testing.T) {
+	s := newTS(t)
+	tm := newTeam(t, s, "red")
+
+	got := listSources(t, tm)
+	if len(got) != 1 {
+		t.Fatalf("expected 1 source, got %d", len(got))
+	}
+	src := got[0]
+	if src["status"] != "never" || src["last_used_at"] != nil || src["last_alert_at"] != nil {
+		t.Errorf("a source nobody has posted on should be blank, got %v", src)
+	}
+	if src["alerts_24h"].(float64) != 0 {
+		t.Errorf("alerts_24h = %v, want 0", src["alerts_24h"])
+	}
+}
+
+// Each source is credited with what arrived on its own key, and only that.
+func TestSources_AlertsAreAttributedToTheirSource(t *testing.T) {
+	s := newTS(t)
+	tm := newTeam(t, s, "red")
+	second := addSource(t, tm, "staging")
+
+	postToIntegration(t, s, tm.key, "fp-1", "DiskFull")
+	postToIntegration(t, s, tm.key, "fp-2", "CPUHot")
+
+	got := listSources(t, tm)
+	first, other := got[0], got[1]
+	if first["status"] != "active" || first["last_used_at"] == nil || first["last_alert_at"] == nil {
+		t.Errorf("the source that posted should be active with timestamps, got %v", first)
+	}
+	if first["alerts_24h"].(float64) != 2 {
+		t.Errorf("alerts_24h = %v, want 2", first["alerts_24h"])
+	}
+	if other["status"] != "never" || other["alerts_24h"].(float64) != 0 {
+		t.Errorf("the other source should be untouched, got %v", other)
+	}
+
+	// Re-sending the same alert on the other key moves it: last sender wins.
+	postToIntegration(t, s, second, "fp-1", "DiskFull")
+	got = listSources(t, tm)
+	if got[0]["alerts_24h"].(float64) != 1 || got[1]["alerts_24h"].(float64) != 1 {
+		t.Errorf("fp-1 should have moved to the second source, got %v and %v",
+			got[0]["alerts_24h"], got[1]["alerts_24h"])
+	}
+}
+
+// A payload with no alerts in it is a webhook, not an alert: the source was
+// heard from, and nothing arrived.
+func TestSources_EmptyPayloadStampsUseButNotAlert(t *testing.T) {
+	s := newTS(t)
+	tm := newTeam(t, s, "red")
+
+	resp, err := http.Post(s.URL+"/api/integrations/"+tm.key+"/alertmanager",
+		"application/json", bytes.NewReader([]byte(`{"version":"4","status":"firing","alerts":[]}`)))
+	if err != nil {
+		t.Fatalf("post: %v", err)
+	}
+	resp.Body.Close()
+
+	src := listSources(t, tm)[0]
+	if src["status"] != "active" || src["last_alert_at"] != nil {
+		t.Errorf("want active with no alert yet, got %v", src)
+	}
+}
+
+// Quiet is "has posted, not lately"; the alert counter forgets after a day but
+// the last alert's timestamp is kept.
+func TestSources_QuietAfterADay(t *testing.T) {
+	s := newTS(t)
+	tm := newTeam(t, s, "red")
+	postToIntegration(t, s, tm.key, "fp-1", "DiskFull")
+
+	old := time.Now().Add(-48 * time.Hour).Unix()
+	s.exec(t, "UPDATE integrations SET last_used_at = $1", old)
+	s.exec(t, "UPDATE alerts SET received_at = $1 WHERE fingerprint = 'fp-1'", old)
+
+	src := listSources(t, tm)[0]
+	if src["status"] != "quiet" {
+		t.Errorf("status = %v, want quiet", src["status"])
+	}
+	if src["alerts_24h"].(float64) != 0 {
+		t.Errorf("alerts_24h = %v, want 0", src["alerts_24h"])
+	}
+	if src["last_alert_at"] == nil {
+		t.Error("last_alert_at should survive the day")
+	}
+}
+
+// Revoking a source does not take its alerts with it.
+func TestSources_RevokeKeepsTheAlerts(t *testing.T) {
+	s := newTS(t)
+	tm := newTeam(t, s, "red")
+	postToIntegration(t, s, tm.key, "fp-1", "DiskFull")
+
+	id := int64(listSources(t, tm)[0]["id"].(float64))
+	resp := tm.call(http.MethodDelete, "/api/teams/"+id64(tm.id)+"/integrations/"+id64(id), nil)
+	resp.Body.Close()
+	if resp.StatusCode != http.StatusNoContent {
+		t.Fatalf("revoke: %d", resp.StatusCode)
+	}
+	if got := len(list(t, tm.call(http.MethodGet, "/api/alerts", nil))); got != 1 {
+		t.Errorf("the alert should outlive its source, got %d alerts", got)
+	}
+}
+
+// Renaming is an owner's, scoped to the team, and does not touch the key.
+func TestSources_Rename(t *testing.T) {
+	s := newTS(t)
+	tm := newTeam(t, s, "red")
+	other := newTeam(t, s, "blue")
+	id := int64(listSources(t, tm)[0]["id"].(float64))
+	path := "/api/teams/" + id64(tm.id) + "/integrations/" + id64(id)
+
+	resp := tm.call(http.MethodPatch, path, map[string]string{"name": "  prod  "})
+	resp.Body.Close()
+	if resp.StatusCode != http.StatusNoContent {
+		t.Fatalf("rename: %d", resp.StatusCode)
+	}
+	if name := listSources(t, tm)[0]["name"]; name != "prod" {
+		t.Errorf("name = %q, want it trimmed to prod", name)
+	}
+	postToIntegration(t, s, tm.key, "fp-1", "DiskFull") // the old key still works
+
+	for name, body := range map[string]map[string]string{
+		"empty":    {"name": "  "},
+		"too long": {"name": strings.Repeat("x", 101)},
+	} {
+		resp := tm.call(http.MethodPatch, path, body)
+		resp.Body.Close()
+		if resp.StatusCode != http.StatusBadRequest {
+			t.Errorf("%s name: expected 400, got %d", name, resp.StatusCode)
+		}
+	}
+
+	// Another team's owner cannot reach it.
+	resp = other.call(http.MethodPatch, "/api/teams/"+id64(other.id)+"/integrations/"+id64(id),
+		map[string]string{"name": "mine now"})
+	resp.Body.Close()
+	if resp.StatusCode != http.StatusNotFound {
+		t.Errorf("renaming another team's source: expected 404, got %d", resp.StatusCode)
+	}
+}

internal/api/teams.go

@@ -355,8 +355,42 @@ func isLastTeamOwner(ctx context.Context, db *sql.DB, teamID, userID int64) (boo
 // Integrations
 // ---------------------------------------------------------------------------
 
-// handleListIntegrations lists a team's integrations. Never the keys: those
+// sourceQuietAfter is how long a source may go without posting before the
-// exist in plaintext only in the response that created them.
+// Sources page calls it quiet rather than active. A day is longer than any
+// repeat_interval worth having, so an Alertmanager that is up and has anything
+// firing never crosses it; a source with nothing firing may, and that is a
+// reason to look, not proof of a fault — which is why this is a colour and not
+// an alarm. Dead man's switches are where silence pages.
+const sourceQuietAfter = 24 * time.Hour
+
+const (
+	sourceActive = "active"
+	sourceQuiet  = "quiet"
+	sourceNever  = "never"
+)
+
+// integrationStatus is an integration as the Sources page shows it.
+type integrationStatus struct {
+	models.Integration
+
+	// Status is active when the key posted within sourceQuietAfter, quiet when
+	// it has posted but not lately, never when it has not posted at all.
+	Status string `json:"status"`
+
+	// LastAlertAt is when an alert last arrived on this source, which is not the
+	// same as when it last posted: a payload with nothing usable in it stamps
+	// last_used_at and not this. Absent until an alert has arrived since
+	// migration 010 started recording it.
+	LastAlertAt *time.Time `json:"last_alert_at,omitempty"`
+
+	// Alerts24h counts the distinct alerts this source refreshed in the last
+	// day. An alert re-sent every few hours counts once, not once per re-send.
+	Alerts24h int64 `json:"alerts_24h"`
+}
+
+// handleListIntegrations lists a team's integrations with what each has been
+// delivering. Never the keys: those exist in plaintext only in the response that
+// created them.
 func handleListIntegrations(db *sql.DB) http.HandlerFunc {
 	return func(w http.ResponseWriter, r *http.Request) {
 		teamID, ok := teamParam(w, r)
@@ -367,28 +401,45 @@ func handleListIntegrations(db *sql.DB) http.HandlerFunc {
 			return
 		}
 
+		now := time.Now()
 		rows, err := db.QueryContext(r.Context(), `
-			SELECT id, team_id, kind, name, created_at, last_used_at
+			SELECT i.id, i.team_id, i.kind, i.name, i.created_at, i.last_used_at,
-			FROM   integrations
+			       -- Scalar subqueries, not a join and GROUP BY: each is a
-			WHERE  team_id = $1
+			       -- single range over alerts_integration_idx, where the join
-			ORDER  BY id`, teamID)
+			       -- would read every alert a source ever delivered.
+			       (SELECT MAX(received_at) FROM alerts WHERE integration_id = i.id),
+			       (SELECT COUNT(*) FROM alerts
+			         WHERE integration_id = i.id AND received_at >= $2)
+			FROM   integrations i
+			WHERE  i.team_id = $1
+			ORDER  BY i.id`, teamID, now.Add(-sourceQuietAfter).Unix())
 		if err != nil {
 			respond(w, http.StatusInternalServerError, errResp("internal error"))
 			return
 		}
 		defer rows.Close()
 
-		integrations := []models.Integration{}
+		integrations := []integrationStatus{}
 		for rows.Next() {
-			var i models.Integration
+			var i integrationStatus
 			var created int64
-			var lastUsed *int64
+			var lastUsed, lastAlert *int64
-			if err := rows.Scan(&i.ID, &i.TeamID, &i.Kind, &i.Name, &created, &lastUsed); err != nil {
+			if err := rows.Scan(&i.ID, &i.TeamID, &i.Kind, &i.Name, &created, &lastUsed,
+				&lastAlert, &i.Alerts24h); err != nil {
 				respond(w, http.StatusInternalServerError, errResp("internal error"))
 				return
 			}
 			i.CreatedAt = time.Unix(created, 0).UTC()
 			i.LastUsedAt = unixPtr(lastUsed)
+			i.LastAlertAt = unixPtr(lastAlert)
+			switch {
+			case i.LastUsedAt == nil:
+				i.Status = sourceNever
+			case now.Sub(*i.LastUsedAt) > sourceQuietAfter:
+				i.Status = sourceQuiet
+			default:
+				i.Status = sourceActive
+			}
 			integrations = append(integrations, i)
 		}
 		if err := rows.Err(); err != nil {
@@ -457,6 +508,54 @@ func handleCreateIntegration(db *sql.DB, publicURL string) http.HandlerFunc {
 	}
 }
 
+// handleRenameIntegration renames a source. The key is untouched, so nothing
+// posting with it notices.
+func handleRenameIntegration(db *sql.DB) http.HandlerFunc {
+	return func(w http.ResponseWriter, r *http.Request) {
+		teamID, ok := teamParam(w, r)
+		if !ok {
+			return
+		}
+		if !requireTeamOwner(w, r, teamID) {
+			return
+		}
+		id, err := strconv.ParseInt(chi.URLParam(r, "integrationID"), 10, 64)
+		if err != nil {
+			respond(w, http.StatusBadRequest, errResp("invalid integration id"))
+			return
+		}
+
+		var req struct {
+			Name string `json:"name"`
+		}
+		if err := decodeJSON(r, &req); err != nil {
+			respond(w, http.StatusBadRequest, errResp("invalid request body"))
+			return
+		}
+		req.Name = strings.TrimSpace(req.Name)
+		if req.Name == "" {
+			respond(w, http.StatusBadRequest, errResp("name is required"))
+			return
+		}
+		if len(req.Name) > 100 {
+			respond(w, http.StatusBadRequest, errResp("name is too long"))
+			return
+		}
+
+		res, err := db.ExecContext(r.Context(),
+			"UPDATE integrations SET name = $1 WHERE id = $2 AND team_id = $3", req.Name, id, teamID)
+		if err != nil {
+			respond(w, http.StatusInternalServerError, errResp("internal error"))
+			return
+		}
+		if n, _ := res.RowsAffected(); n == 0 {
+			respond(w, http.StatusNotFound, errResp("not found"))
+			return
+		}
+		w.WriteHeader(http.StatusNoContent)
+	}
+}
+
 func handleDeleteIntegration(db *sql.DB) http.HandlerFunc {
 	return func(w http.ResponseWriter, r *http.Request) {
 		teamID, ok := teamParam(w, r)
@@ -493,24 +592,32 @@ func integrationPath(key, kind string) string {
 	return "/api/integrations/" + key + "/" + kind
 }
 
-// teamIDForKey resolves an integration key to its team, and stamps the key's
+// alertSource is who an arriving webhook is from: the integration whose key it
+// used, and the team that integration puts its alerts in.
+type alertSource struct {
+	integrationID int64
+	teamID        int64
+}
+
+// sourceForKey resolves an integration key to its source, and stamps the key's
 // last use. An unknown key is not an error worth distinguishing: the caller is
 // told nothing beyond "no".
-func teamIDForKey(ctx context.Context, db *sql.DB, key string) (int64, error) {
+func sourceForKey(ctx context.Context, db *sql.DB, key string) (alertSource, error) {
-	var teamID int64
+	var src alertSource
 	err := db.QueryRowContext(ctx,
-		"SELECT team_id FROM integrations WHERE key_hash = $1", hashToken(key)).Scan(&teamID)
+		"SELECT id, team_id FROM integrations WHERE key_hash = $1", hashToken(key)).
+		Scan(&src.integrationID, &src.teamID)
 	if errors.Is(err, sql.ErrNoRows) {
-		return 0, errUnknownIntegration
+		return alertSource{}, errUnknownIntegration
 	}
 	if err != nil {
-		return 0, err
+		return alertSource{}, err
 	}
 	// Best effort, like an API key's: a failed stamp must not reject an alert.
 	db.ExecContext(ctx, //nolint:errcheck
-		"UPDATE integrations SET last_used_at = $1 WHERE key_hash = $2",
+		"UPDATE integrations SET last_used_at = $1 WHERE id = $2",
-		time.Now().Unix(), hashToken(key))
+		time.Now().Unix(), src.integrationID)
-	return teamID, nil
+	return src, nil
 }
 
 var errUnknownIntegration = errors.New("unknown integration key")
@@ -539,17 +646,24 @@ func defaultTeamID(ctx context.Context, db *sql.DB) (int64, error) {
 // A team's dead man's switches
 // ---------------------------------------------------------------------------
 
-// deadmanResponse is the wire shape of a team's switch configuration. The
+// deadmanSwitchRequest is what creating a switch takes. The timeout is seconds,
-// timeout is seconds rather than a duration string, because that is what the
+// because that is what the column holds and what arithmetic is done on; a client
-// column holds and what arithmetic is done on; a client renders it.
+// renders it.
-type deadmanResponse struct {
+type deadmanSwitchRequest struct {
-	TeamID         int64  `json:"team_id"`
+	Name           string `json:"name"`
-	Matchers       string `json:"matchers"`
+	Matcher        string `json:"matcher"`
 	TimeoutSeconds int64  `json:"timeout_seconds"`
 	Severity       string `json:"severity"`
 }
 
-func handleGetTeamDeadman(db *sql.DB) http.HandlerFunc {
+// deadmanSeverities are the severities an incident can open at.
+var deadmanSeverities = map[string]bool{"critical": true, "error": true, "warning": true, "info": true}
+
+// handleListTeamDeadman lists a team's switches with what each one's heartbeats
+// are doing. A team with none gets an empty list, which is a configuration and
+// not an absence: answering 404 would make "off" indistinguishable from "this
+// server does not do this".
+func handleListTeamDeadman(db *sql.DB) http.HandlerFunc {
 	return func(w http.ResponseWriter, r *http.Request) {
 		teamID, ok := teamParam(w, r)
 		if !ok {
@@ -559,27 +673,26 @@ func handleGetTeamDeadman(db *sql.DB) http.HandlerFunc {
 			return
 		}
 
-		out := deadmanResponse{TeamID: teamID, Severity: "critical"}
+		set, err := deadmanSetForTeam(r.Context(), db, teamID)
-		err := db.QueryRowContext(r.Context(),
+		if err != nil {
-			"SELECT matchers, timeout_seconds, severity FROM deadman_configs WHERE team_id = $1",
+			respond(w, http.StatusInternalServerError, errResp("internal error"))
-			teamID).Scan(&out.Matchers, &out.TimeoutSeconds, &out.Severity)
+			return
-		if err != nil && !errors.Is(err, sql.ErrNoRows) {
+		}
+		out, err := deadmanStatuses(r.Context(), db, teamID, set, time.Now())
+		if err != nil {
 			respond(w, http.StatusInternalServerError, errResp("internal error"))
 			return
 		}
-		// A team with no row watches nothing, which is a configuration and not
-		// an absence: answering 404 would make "off" indistinguishable from
-		// "this server does not do this".
 		respond(w, http.StatusOK, out)
 	}
 }
 
-// handleSetTeamDeadman replaces a team's switch configuration.
+// handleCreateTeamDeadman adds one switch.
 //
-// Validated by parsing: a matcher string that survives ParseDeadmanConfig with
+// Validated by parsing: a matcher with no alertname is rejected rather than
-// nothing usable in it is rejected rather than stored, because a switch that
+// stored, because a switch that silently watches nothing is the failure this
-// silently watches nothing is the failure this feature exists to prevent.
+// feature exists to prevent.
-func handleSetTeamDeadman(db *sql.DB) http.HandlerFunc {
+func handleCreateTeamDeadman(db *sql.DB) http.HandlerFunc {
 	return func(w http.ResponseWriter, r *http.Request) {
 		teamID, ok := teamParam(w, r)
 		if !ok {
@@ -589,50 +702,87 @@ func handleSetTeamDeadman(db *sql.DB) http.HandlerFunc {
 			return
 		}
 
-		var req struct {
+		var req deadmanSwitchRequest
-			Matchers       string `json:"matchers"`
-			TimeoutSeconds int64  `json:"timeout_seconds"`
-			Severity       string `json:"severity"`
-		}
 		if err := decodeJSON(r, &req); err != nil {
 			respond(w, http.StatusBadRequest, errResp("invalid request body"))
 			return
 		}
-		req.Matchers = strings.TrimSpace(req.Matchers)
+		req.Matcher = strings.TrimSpace(req.Matcher)
+		req.Name = strings.TrimSpace(req.Name)
 		if req.Severity == "" {
 			req.Severity = "critical"
 		}
-		if req.TimeoutSeconds < 0 {
+		if !deadmanSeverities[req.Severity] {
-			respond(w, http.StatusBadRequest, errResp("timeout_seconds must not be negative"))
+			respond(w, http.StatusBadRequest, errResp("severity must be critical, error, warning or info"))
 			return
 		}
-		if req.Matchers != "" {
+		if req.TimeoutSeconds <= 0 {
-			parsed := parseDeadmanQuietly(req.Matchers, time.Duration(req.TimeoutSeconds)*time.Second, req.Severity)
+			respond(w, http.StatusBadRequest, errResp("timeout_seconds must be positive"))
-			if len(parsed.Matchers) == 0 {
+			return
-				respond(w, http.StatusBadRequest, errResp(
+		}
-					"no usable matchers: each must name an alertname, as in alertname=Watchdog,cluster=prod"))
+		if strings.Contains(req.Matcher, ";") {
-				return
+			respond(w, http.StatusBadRequest, errResp("one matcher per switch: add another switch instead of separating with ;"))
-			}
+			return
+		}
+		m, err := parseDeadmanMatcher(req.Matcher)
+		if err != nil {
+			respond(w, http.StatusBadRequest, errResp(
+				"unusable matcher ("+err.Error()+"): each must name an alertname, as in alertname=Watchdog,cluster=prod"))
+			return
+		}
+		if req.Name == "" {
+			req.Name = m.config()
+		}
+		if len(req.Name) > 100 {
+			respond(w, http.StatusBadRequest, errResp("name is too long"))
+			return
 		}
 
-		if _, err := db.ExecContext(r.Context(), `
+		var id int64
-			INSERT INTO deadman_configs (team_id, matchers, timeout_seconds, severity, updated_at)
+		if err := db.QueryRowContext(r.Context(), `
-			VALUES ($1, $2, $3, $4, `+nowEpoch+`)
+			INSERT INTO deadman_switches (team_id, name, matcher, timeout_seconds, severity)
-			ON CONFLICT (team_id) DO UPDATE SET
+			VALUES ($1, $2, $3, $4, $5) RETURNING id`,
-				matchers        = excluded.matchers,
+			teamID, req.Name, m.config(), req.TimeoutSeconds, req.Severity).Scan(&id); err != nil {
-				timeout_seconds = excluded.timeout_seconds,
-				severity        = excluded.severity,
-				updated_at      = excluded.updated_at`,
-			teamID, req.Matchers, req.TimeoutSeconds, req.Severity); err != nil {
 			respond(w, http.StatusInternalServerError, errResp("internal error"))
 			return
 		}
 
-		respond(w, http.StatusOK, deadmanResponse{
+		respond(w, http.StatusCreated, deadmanSwitchStatus{
-			TeamID:         teamID,
+			ID: id, Name: req.Name, Matcher: m.config(),
-			Matchers:       req.Matchers,
+			TimeoutSeconds: req.TimeoutSeconds, Severity: req.Severity,
-			TimeoutSeconds: req.TimeoutSeconds,
+			Status: switchDormant, Sources: []deadmanSource{},
-			Severity:       req.Severity,
 		})
 	}
 }
+
+// handleDeleteTeamDeadman removes a switch. An incident it already opened stays
+// open until somebody resolves it: deleting the switch says "stop watching", not
+// "the problem is gone".
+func handleDeleteTeamDeadman(db *sql.DB) http.HandlerFunc {
+	return func(w http.ResponseWriter, r *http.Request) {
+		teamID, ok := teamParam(w, r)
+		if !ok {
+			return
+		}
+		if !requireTeamOwner(w, r, teamID) {
+			return
+		}
+		switchID, err := strconv.ParseInt(chi.URLParam(r, "switchID"), 10, 64)
+		if err != nil {
+			respond(w, http.StatusBadRequest, errResp("invalid switch id"))
+			return
+		}
+
+		res, err := db.ExecContext(r.Context(),
+			"DELETE FROM deadman_switches WHERE id = $1 AND team_id = $2", switchID, teamID)
+		if err != nil {
+			respond(w, http.StatusInternalServerError, errResp("internal error"))
+			return
+		}
+		if n, _ := res.RowsAffected(); n == 0 {
+			respond(w, http.StatusNotFound, errResp("switch not found"))
+			return
+		}
+		w.WriteHeader(http.StatusNoContent)
+	}
+}

internal/db/migrations/009_deadman_switches.sql

@@ -0,0 +1,54 @@
+-- Dead man's switches become rows of their own.
+--
+-- 004 kept a team's switches in one string with one timeout and one severity,
+-- which was enough to configure them and not enough to show them: there was no
+-- thing to list, nothing to hang a status on, and every switch in a team had to
+-- share a deadline. A row per switch gives each its own name, matcher, timeout
+-- and severity, and gives the Team → Switches page something to be a list of.
+--
+-- The matcher keeps the syntax the string used, one matcher per row:
+-- `alertname=Watchdog,cluster=prod`. The unit of monitoring is still the
+-- fingerprint, so a matcher that many clusters satisfy is still one switch row
+-- watching several independent heartbeats.
+CREATE TABLE deadman_switches (
+    id      BIGSERIAL PRIMARY KEY,
+    team_id BIGINT NOT NULL REFERENCES teams(id) ON DELETE CASCADE,
+
+    -- What the owner calls it. Defaults to the matcher when they do not say.
+    name TEXT NOT NULL,
+
+    -- "," separates the label conditions, "=" is exact equality, and alertname is
+    -- mandatory: it is what keeps the sweeper's candidate query on an index.
+    matcher TEXT NOT NULL,
+
+    -- Seconds of silence before the switch is declared dead. Never zero: a switch
+    -- that cannot fire is deleted, not disabled.
+    timeout_seconds BIGINT NOT NULL CHECK (timeout_seconds > 0),
+
+    -- The severity its incidents open at. See 004 for why they carry their own.
+    severity TEXT NOT NULL DEFAULT 'critical',
+
+    created_at BIGINT NOT NULL DEFAULT FLOOR(EXTRACT(EPOCH FROM now()))::bigint
+);
+
+CREATE INDEX deadman_switches_team_idx ON deadman_switches (team_id);
+
+-- Carry every team's configuration over, one row per matcher. A team whose
+-- timeout was zero had switches turned off, which is now "no rows".
+INSERT INTO deadman_switches (team_id, name, matcher, timeout_seconds, severity)
+SELECT c.team_id, btrim(m), btrim(m), c.timeout_seconds, c.severity
+FROM   deadman_configs c,
+       LATERAL regexp_split_to_table(c.matchers, ';') AS m
+WHERE  c.timeout_seconds > 0
+  AND  btrim(m) <> ''
+ORDER  BY c.team_id;
+
+-- The server seeds environment defaults into teams once, and remembers that it
+-- did. An install that had a row per team was already seeded; without this
+-- marker the first start after upgrading would seed teams that had switched
+-- theirs off.
+INSERT INTO settings (key, value)
+SELECT 'deadman_seeded', '1'
+WHERE  EXISTS (SELECT 1 FROM deadman_configs);
+
+DROP TABLE deadman_configs;

internal/db/migrations/010_alert_source.sql

@@ -0,0 +1,21 @@
+-- Which alert source an alert last arrived on.
+--
+-- Team -> Sources shows when each source last posted, which integrations
+-- already knew (last_used_at, stamped on every webhook). What it could not say
+-- was what a source delivered: an alert never recorded the key it came in on, so
+-- "prod alertmanager" and "staging alertmanager" were indistinguishable once
+-- inside. This column is that link, and lets the page show each source's last
+-- alert and how many alerts it has kept fresh over the past day.
+--
+-- Last sender wins: every accepted payload restamps it, the way it advances
+-- received_at. Two sources posting the same fingerprint into one team is
+-- already one alert, and it is attributed to whichever spoke last.
+--
+-- Nullable, and not backfilled. Alerts that arrived before this migration have
+-- no source, and NULL says so honestly rather than guessing. It heals by itself:
+-- Alertmanager re-sends every alert each repeat_interval, and each re-send is an
+-- accepted payload. Deleting a source keeps its alerts, unattributed.
+ALTER TABLE alerts ADD COLUMN integration_id BIGINT REFERENCES integrations(id) ON DELETE SET NULL;
+
+CREATE INDEX alerts_integration_idx ON alerts (integration_id, received_at)
+    WHERE integration_id IS NOT NULL;

internal/web/incident_test.go

@@ -0,0 +1,26 @@
+package web
+
+import (
+	"io/fs"
+	"strings"
+	"testing"
+)
+
+func TestCopyIncidentIsEmbedded(t *testing.T) {
+	sub, err := fs.Sub(files, "static")
+	if err != nil {
+		t.Fatal(err)
+	}
+	for file, want := range map[string]string{
+		"js/incident.js": "copyIncident",
+		"js/ui.js":       "copy:",
+	} {
+		b, err := fs.ReadFile(sub, file)
+		if err != nil {
+			t.Fatal(err)
+		}
+		if !strings.Contains(string(b), want) {
+			t.Errorf("%s lacks %s", file, want)
+		}
+	}
+}

internal/web/static/app.css

@@ -359,7 +359,13 @@ input:focus, textarea:focus { outline: none; border-color: var(--accent); box-sh
 .badge.st-triggered, .badge.st-firing { background: var(--crit-soft); color: var(--crit); }
 .badge.st-acknowledged { background: var(--warn-soft); color: var(--warn); }
 .badge.st-snoozed { background: var(--snooze-soft); color: var(--snooze); }
-.badge.st-resolved { background: var(--ok-soft); color: var(--ok); }
+.badge.st-resolved, .badge.st-healthy { background: var(--ok-soft); color: var(--ok); }
+.badge.st-dead { background: var(--crit-soft); color: var(--crit); }
+/* Dormant is the plain badge on purpose: nothing has gone wrong and nothing has
+   gone right, which is what the muted default already says. */
+.badge.st-dormant, .badge.st-never { background: var(--surface-2); color: var(--muted); }
+.badge.st-active { background: var(--ok-soft); color: var(--ok); }
+.badge.st-quiet { background: var(--warn-soft); color: var(--warn); }
 .badge.sev-critical { background: var(--crit-soft); color: var(--crit); }
 .badge.sev-warning { background: var(--warn-soft); color: var(--warn); }
 .badge.sev-info { background: var(--info-soft); color: var(--info); }
@@ -377,6 +383,8 @@ input:focus, textarea:focus { outline: none; border-color: var(--accent); box-sh
   -webkit-backdrop-filter: saturate(1.4) blur(12px);
   border-bottom: 1px solid var(--border);
 }
+.detail-head .copy { margin-left: auto; }
+.clip-buffer { position: fixed; top: 0; left: 0; opacity: 0; pointer-events: none; }
 .detail-head .crumb { font-weight: 600; color: var(--muted); font-size: 14px; }
 .detail-title { font-size: 21px; font-weight: 750; letter-spacing: -0.01em; margin: 16px 0 8px; overflow-wrap: anywhere; }
 .detail-badges { display: flex; flex-wrap: wrap; gap: 6px; margin-bottom: 14px; }
@@ -689,6 +697,18 @@ kbd {
 .disabled-row td { opacity: 0.55; }
 .btn-sm.danger { color: var(--crit); border-color: var(--crit-soft); }
 
+/* --- status lists: alert sources and dead man's switches -----------------
+   Six columns do not fit a phone, so the table scrolls inside its card rather
+   than the page. A heartbeat under a switch with several is indented, the way
+   the escalation ladder indents its levels. */
+.card-head { display: flex; align-items: center; justify-content: space-between; gap: 12px; flex-wrap: wrap; }
+.table-scroll { overflow-x: auto; margin-top: 12px; }
+.status-table th, .status-table td { white-space: nowrap; }
+.status-table td:nth-child(2) { white-space: normal; min-width: 12em; }
+.status-table .source-row td { border-bottom-style: dashed; }
+.status-table .source-row td:first-child { padding-left: 16px; }
+.source-labels { display: flex; flex-wrap: wrap; gap: 4px; align-items: center; }
+
 .inline-form { display: flex; gap: 8px; margin-top: 12px; }
 .inline-form input { flex: 1; min-width: 0; }
 

internal/web/static/js/api.js

@@ -133,11 +133,16 @@ export const removeTeamMember = (id, userID) => call('DELETE', `/teams/${id}/mem
 export const integrations = (id) => call('GET', `/teams/${id}/integrations`);
 export const createIntegration = (id, name) =>
   call('POST', `/teams/${id}/integrations`, { body: { name } });
+export const renameIntegration = (id, integrationID, name) =>
+  call('PATCH', `/teams/${id}/integrations/${integrationID}`, { body: { name } });
 export const deleteIntegration = (id, integrationID) =>
   call('DELETE', `/teams/${id}/integrations/${integrationID}`);
 
-export const deadman = (id) => call('GET', `/teams/${id}/deadman`);
+export const deadmanSwitches = (id) => call('GET', `/teams/${id}/deadman/switches`);
-export const setDeadman = (id, body) => call('PUT', `/teams/${id}/deadman`, { body });
+export const createDeadmanSwitch = (id, body) =>
+  call('POST', `/teams/${id}/deadman/switches`, { body });
+export const deleteDeadmanSwitch = (id, switchID) =>
+  call('DELETE', `/teams/${id}/deadman/switches/${switchID}`);
 
 export const escalation = (id) => call('GET', `/teams/${id}/escalation`);
 export const setEscalation = (id, body) => call('PUT', `/teams/${id}/escalation`, { body });

internal/web/static/js/incident.js

@@ -66,6 +66,8 @@ function render() {
     h('button', { class: 'btn btn-ghost btn-icon back', type: 'button', 'aria-label': 'Back to queue', onclick: back },
       icon('back')),
     h('span', { class: 'crumb', text: currentID != null ? `Incident #${currentID}` : '' }),
+    inc && h('button', { class: 'btn btn-ghost btn-icon copy', type: 'button', 'aria-label': 'Copy incident', title: 'Copy incident (y)', onclick: copyIncident },
+      icon('copy')),
   );
 
   if (!inc) {
@@ -186,8 +188,9 @@ function alertItem(a) {
 
 // ---------- timeline ----------
 
-function eventText(ev) {
+// named spells users out instead of "you", for text that leaves this page.
-  const person = ev.user_id != null ? who(ev.user_id, ev.username) : null;
+function eventText(ev, named = false) {
+  const person = ev.user_id != null ? (named ? ev.username || 'someone' : who(ev.user_id, ev.username)) : null;
   const strong = (t) => h('span', { class: 'who', text: t || 'someone' });
   const alertName = () => {
     const a = (inc.alerts || []).find((x) => x.id === ev.alert_id);
@@ -263,6 +266,100 @@ function timelineItem(ev) {
   );
 }
 
+// ---------- copy ----------
+
+const fence = (rows) => ['```', ...rows, '```'];
+const pairs = (obj) => Object.entries(obj || {}).sort(([a], [b]) => a.localeCompare(b)).map(([k, v]) => `${k}=${v}`);
+
+// incidentMarkdown is everything on this page as text that reads well in a chat
+// or an agent prompt. Times are ISO 8601, since "3 min ago" means nothing once
+// it has been pasted somewhere else.
+function incidentMarkdown() {
+  const out = [`# Incident #${inc.id}: ${inc.title}`, ''];
+  const add = (k, v) => { if (v != null && v !== '') out.push(`- ${k}: ${v}`); };
+  add('Status', inc.status);
+  add('Severity', inc.severity);
+  add('Team', inc.team_name);
+  add('Assigned to', inc.assigned_to_id != null ? inc.assigned_to || 'someone' : 'unassigned');
+  add('Triggered', inc.triggered_at);
+  if (inc.acknowledged_at) add('Acknowledged', `${inc.acknowledged_at} by ${inc.acknowledged_by || 'someone'}`);
+  if (isOpen() && isFuture(inc.snoozed_until)) add('Snoozed until', inc.snoozed_until);
+  if (inc.escalation_level > 0) add('Escalation level', inc.escalation_level);
+  if (inc.resolved_at) add('Resolved', `${inc.resolved_at} (${inc.resolution_source === 'manual' ? 'manually' : 'all alerts stopped firing'})`);
+  if (inc.archived_at) add('Archived', inc.archived_at);
+  const group = pairs(inc.group_labels);
+  if (group.length) out.push('- Grouped by:', ...group.map((g) => `  - ${g}`));
+
+  const alerts = inc.alerts || [];
+  out.push('', `## Alerts (${alerts.length})`);
+  for (const a of alerts) {
+    out.push('', `### ${a.name} (${a.status})`);
+    out.push(`- Started: ${a.starts_at}`);
+    if (a.status === 'resolved' && a.ends_at) out.push(`- Ended: ${a.ends_at}`);
+    if (a.generator_url) out.push(`- Source: ${a.generator_url}`);
+    const labels = pairs(a.labels);
+    if (labels.length) out.push('', 'Labels:', ...fence(labels));
+    const annotations = Object.entries(a.annotations || {}).sort(([x], [y]) => x.localeCompare(y));
+    if (annotations.length) out.push('', 'Annotations:', ...fence(annotations.map(([k, v]) => `${k}: ${v}`)));
+  }
+
+  const sorted = [...events].sort((a, b) => Date.parse(a.created_at) - Date.parse(b.created_at) || a.id - b.id);
+  if (sorted.length) {
+    out.push('', '## Timeline', '');
+    for (const ev of sorted) {
+      const text = eventText(ev, true).map((f) => (f instanceof Node ? f.textContent : f)).join('');
+      out.push(`- ${ev.created_at} ${text}`);
+      if (isNote(ev) && ev.detail) {
+        const label = ev.type === 'resolution_note' ? '  (what fixed it)' : '';
+        out.push(...(label ? [label] : []), ...ev.detail.split('\n').map((l) => `  > ${l}`));
+      }
+    }
+  }
+
+  if (similarList.length) {
+    out.push('', '## Seen before', '', 'Earlier incidents with the same signature:');
+    for (const s of similarList) {
+      out.push(`- #${s.id} ${s.title} (resolved ${s.resolved_at})`);
+      for (const n of s.resolution_notes || []) {
+        out.push('  - What fixed it:', ...(n.detail || '').split('\n').map((l) => `    > ${l}`));
+      }
+    }
+  }
+
+  out.push('', `_Copied from Terminal Duty at ${new Date().toISOString()}_`, '');
+  return out.join('\n');
+}
+
+// writeClipboard falls back to execCommand: the async API needs a secure
+// context, and this server is often reached over plain HTTP.
+async function writeClipboard(text) {
+  try {
+    await navigator.clipboard.writeText(text);
+    return;
+  } catch {
+    // fall through
+  }
+  const ta = h('textarea', { readonly: true, 'aria-hidden': 'true', class: 'clip-buffer' });
+  ta.value = text;
+  document.body.append(ta);
+  ta.select();
+  try {
+    if (!document.execCommand('copy')) throw new Error('copy refused');
+  } finally {
+    ta.remove();
+  }
+}
+
+async function copyIncident() {
+  if (!inc) return;
+  try {
+    await writeClipboard(incidentMarkdown());
+    toast('Copied incident');
+  } catch {
+    toast('Could not copy', 'error');
+  }
+}
+
 // ---------- actions ----------
 
 const isOpen = () => inc.status !== 'resolved';
@@ -470,10 +567,12 @@ async function moreMenu() {
     items.push(item('user', 'Assign…', assign));
     items.push(isSnoozed() ? item('bell', 'End snooze', unsnooze) : item('clock', 'Snooze…', snooze));
     items.push(item('note', 'Add note…', addNote));
+    items.push(item('copy', 'Copy incident', copyIncident));
     items.push(h('li', { class: 'menu-sep', role: 'separator' }));
     items.push(item('checkCircle', 'Resolve…', resolve, 'danger'));
   } else {
     items.push(item('note', 'Add note…', addNote));
+    items.push(item('copy', 'Copy incident', copyIncident));
     items.push(inc.archived_at ? item('undo', 'Unarchive', unarchive) : item('archive', 'Archive', archive));
   }
 
@@ -502,6 +601,7 @@ export function key(e) {
     case 'z': if (isOpen() && !isSnoozed()) snooze(); return true;
     case 'Z': if (isSnoozed()) unsnooze(); return true;
     case 'c': addNote(); return true;
+    case 'y': copyIncident(); return true;
     case 'x': if (!isOpen()) (inc.archived_at ? unarchive() : archive()); return true;
     default: return false;
   }

internal/web/static/js/team.js

@@ -18,9 +18,9 @@
 // than no form, but it is not the thing enforcing anything.
 
 import * as api from './api.js';
-import { h, clear, spinner, confirm, icon, openSheet, closeSheet, menuCard } from './ui.js';
+import { h, clear, spinner, confirm, icon, openSheet, closeSheet, menuCard, badge, labelChip } from './ui.js';
 import { state, currentTeam, users as allUsers, myID } from './state.js';
-import { isoDate, addDays, mondayOf, initial } from './format.js';
+import { isoDate, addDays, mondayOf, initial, ago, when, duration } from './format.js';
 
 const view = () => document.getElementById('view-team');
 
@@ -111,14 +111,14 @@ async function load(id) {
     return { members, escalation };
   }
   if (tab === 'sources') return { integrations: await api.integrations(id) };
-  if (tab === 'deadman') return { deadman: await api.deadman(id) };
+  if (tab === 'deadman') return { deadman: await api.deadmanSwitches(id) };
 
   const grid = gridDays();
   const [members, integrations, escalation, deadman, schedule] = await Promise.all([
     api.teamMembers(id),
     api.integrations(id),
     api.escalation(id),
-    api.deadman(id),
+    api.deadmanSwitches(id),
     api.schedule(id, isoDate(grid.start), isoDate(addDays(grid.start, grid.count - 1))),
   ]);
   return { members, integrations, escalation, deadman, schedule };
@@ -203,8 +203,8 @@ function overview() {
   const levels = (data.escalation?.levels || []).length;
   const keys = (data.integrations || []).length;
   const unused = (data.integrations || []).filter((i) => !i.last_used_at).length;
-  const switches = (data.deadman?.matchers || '')
+  const switches = (data.deadman || []).length;
-    .split(';').map((x) => x.trim()).filter(Boolean).length;
+  const dead = (data.deadman || []).filter((s) => s.status === 'dead').length;
 
   return h('div', { class: 'overview-menu' },
     menuCard('/team/rota', 'Rota', null,
@@ -220,7 +220,9 @@ function overview() {
         ? (unused ? `${unused} of them never used.` : 'All in use.')
         : 'No key yet, so nothing can reach this team.'),
     menuCard('/team/deadman', 'Dead man’s switches', switches || null,
-      switches ? 'Alerts whose absence opens an incident.' : 'Nothing watched.'),
+      switches
+        ? (dead ? `${dead} of them silent.` : 'All quiet, as they should be.')
+        : 'Nothing watched.'),
   );
 }
 
@@ -574,33 +576,54 @@ function minutesInput(seconds, onChange) {
 function integrationsCard() {
   const rows = (data.integrations || []).map((i) =>
     h('tr', {},
-      h('td', {}, h('strong', { text: i.name })),
+      h('td', {}, sourceBadge(i.status)),
-      h('td', { class: 'muted small', text: i.kind }),
+      h('td', {},
-      h('td', { class: 'muted small', text: i.last_used_at ? 'in use' : 'never used' }),
+        h('strong', { text: i.name }),
-      h('td', {}, isOwner() && h('button', {
+        h('div', { class: 'muted small', text: i.kind })),
-        class: 'btn-sm danger', type: 'button', text: 'Revoke',
+      // When the key last posted, and when an alert last arrived on it. They
-        onclick: async () => {
+      // differ: a payload with nothing usable in it stamps only the first.
-          if (!(await confirm({
+      h('td', { class: 'muted small' }, timeCell(i.last_used_at)),
-            title: `Revoke ${i.name}?`,
+      h('td', { class: 'muted small' }, timeCell(i.last_alert_at)),
-            text: 'Anything posting with this key stops delivering immediately.',
+      h('td', { class: 'muted small num', title: 'Distinct alerts refreshed in the last 24 hours',
-            confirmLabel: 'Revoke',
+        text: String(i.alerts_24h ?? 0) }),
-            danger: true,
+      h('td', { class: 'muted small' }, h('span', { title: when(i.created_at), text: ago(i.created_at) })),
-          }))) return;
+      h('td', {}, isOwner() && h('div', { class: 'row-actions' },
-          act(() => api.deleteIntegration(teamID, i.id));
+        h('button', {
-        },
+          class: 'btn-sm', type: 'button', text: 'Rename', onclick: () => openRenameSource(i),
-      })),
+        }),
+        h('button', {
+          class: 'btn-sm danger', type: 'button', text: 'Revoke',
+          onclick: async () => {
+            if (!(await confirm({
+              title: `Revoke ${i.name}?`,
+              text: 'Anything posting with this key stops delivering immediately. Alerts it already delivered stay.',
+              confirmLabel: 'Revoke',
+              danger: true,
+            }))) return;
+            act(() => api.deleteIntegration(teamID, i.id));
+          },
+        }))),
     ));
 
   return h('div', { class: 'card' },
-    h('h2', { text: 'Alert sources' }),
+    h('div', { class: 'card-head' },
+      h('h2', { text: 'Alert sources' }),
+      isOwner() && h('button', {
+        class: 'btn', type: 'button', text: 'New source', onclick: openNewSource,
+      })),
     h('p', { class: 'muted small' },
       'Alerts arrive on an integration key, which says both that the sender may ',
       'post and which team the alerts belong to.'),
-    rows.length
-      ? h('table', { class: 'admin-table' }, h('tbody', {}, rows))
-      : h('p', { class: 'muted', text: 'No alert source yet, so nothing can reach this team.' }),
     freshKey && newKeyPanel(),
-    isOwner() && !freshKey && newIntegrationForm(),
+    rows.length
+      ? h('div', { class: 'table-scroll' },
+        h('table', { class: 'admin-table status-table' },
+          h('thead', {}, h('tr', {},
+            h('th', { text: 'Status' }), h('th', { text: 'Source' }),
+            h('th', { text: 'Last webhook' }), h('th', { text: 'Last alert' }),
+            h('th', { class: 'num', text: 'Alerts 24h' }), h('th', { text: 'Created' }), h('th'))),
+          h('tbody', {}, rows)))
+      : h('p', { class: 'muted', text: 'No alert source yet, so nothing can reach this team.' }),
   );
 }
 
@@ -631,66 +654,204 @@ function newKeyPanel() {
   );
 }
 
-function newIntegrationForm() {
+// A sheet with one name field, for adding a source and for renaming one: the two
-  const name = h('input', { type: 'text', placeholder: 'prod alertmanager', required: true });
+// differ only in what they call and what they put in the box.
-  const form = h('form', { class: 'inline-form' }, name,
+function openNameSheet({ title, submit, value, run }) {
-    h('button', { class: 'btn', type: 'submit', text: 'Add' }));
+  const name = h('input', {
+    type: 'text', placeholder: 'prod alertmanager', required: true, value, autofocus: true,
+  });
+  const problem = h('p', { class: 'load-error', hidden: true });
+  const form = h('form', { class: 'stacked-form' },
+    h('label', {}, 'Name ', name),
+    problem,
+    h('div', { class: 'sheet-actions' },
+      h('button', { class: 'btn', type: 'button', text: 'Cancel', onclick: () => closeSheet(false) }),
+      h('button', { class: 'btn btn-primary', type: 'submit', text: submit })));
+
   form.addEventListener('submit', async (e) => {
     e.preventDefault();
     try {
-      freshKey = await api.createIntegration(teamID, name.value.trim());
+      await run(name.value.trim());
-      await refresh();
     } catch (err) {
-      error = err.message;
+      problem.textContent = err.message;
-      render();
+      problem.hidden = false;
+      return;
     }
+    closeSheet(true);
+    refresh();
+  });
+  openSheet(() => [h('h2', { class: 'sheet-title', text: title }), form]);
+}
+
+function openNewSource() {
+  openNameSheet({
+    title: 'New source', submit: 'Add source', value: '',
+    // The key comes back once, and the card shows it until dismissed.
+    run: async (name) => { freshKey = await api.createIntegration(teamID, name); },
+  });
+}
+
+function openRenameSource(i) {
+  openNameSheet({
+    title: `Rename ${i.name}`, submit: 'Rename', value: i.name,
+    run: (name) => api.renameIntegration(teamID, i.id, name),
   });
-  return form;
 }
 
 // --- dead man's switches ---------------------------------------------------
 
+// Status badges, shared by the Sources and Switches lists: a table of label and
+// hint per status, and one function to draw it. Module-level, so the two cards
+// can be defined in either order.
+const SWITCH_STATUS = {
+  healthy: { label: 'Healthy', hint: 'Heard from within its timeout.' },
+  dead: { label: 'Dead', hint: 'Silent for longer than its timeout.' },
+  dormant: { label: 'Dormant', hint: 'Nothing has matched yet, so there is nothing to lose.' },
+};
+
+const SOURCE_STATUS = {
+  active: { label: 'Active', hint: 'Posted within the last day.' },
+  quiet: { label: 'Quiet', hint: 'Has posted, but not in the last day. Nothing firing is a fine reason.' },
+  never: { label: 'Never used', hint: 'Nothing has been posted with this key yet.' },
+};
+
+function statusBadge(table, status, fallback) {
+  const s = table[status] || table[fallback];
+  const el = badge(s.label, `st-${status}`);
+  el.title = s.hint;
+  return el;
+}
+
+const switchBadge = (status) => statusBadge(SWITCH_STATUS, status, 'dormant');
+const sourceBadge = (status) => statusBadge(SOURCE_STATUS, status, 'never');
+
+const timeCell = (iso) => iso
+  ? h('span', { title: when(iso), text: ago(iso) })
+  : h('span', { class: 'muted', text: 'never' });
+
+// When it last opened an incident. An incident that is still open is a link,
+// because that is the thing somebody looking at a red row wants next.
+const triggeredCell = (iso, incidentID) => {
+  if (!iso) return h('span', { class: 'muted', text: 'never' });
+  return incidentID
+    ? h('a', { href: `/incidents/${incidentID}`, title: when(iso) }, `#${incidentID} · ${ago(iso)}`)
+    : h('span', { title: when(iso), text: ago(iso) });
+};
+
+function switchRows(sw) {
+  const main = h('tr', {},
+    h('td', {}, switchBadge(sw.status)),
+    h('td', {},
+      h('strong', { text: sw.name }),
+      sw.name !== sw.matcher && h('div', { class: 'muted small' }, h('code', { text: sw.matcher }))),
+    h('td', { class: 'muted small' }, timeCell(sw.last_heartbeat_at)),
+    h('td', { class: 'muted small' }, triggeredCell(sw.last_triggered_at, sw.open_incident_id)),
+    h('td', { class: 'muted small', text: duration(sw.timeout_seconds * 1000) }),
+    h('td', {}, isOwner() && h('button', {
+      class: 'btn-sm danger', type: 'button', text: 'Remove',
+      onclick: async () => {
+        if (!(await confirm({
+          title: `Remove ${sw.name}?`,
+          text: 'It stops being watched. An incident it already opened stays open until it is resolved.',
+          confirmLabel: 'Remove',
+          danger: true,
+        }))) return;
+        act(() => api.deleteDeadmanSwitch(teamID, sw.id));
+      },
+    })),
+  );
+
+  // One heartbeat is the switch's own times; several are worth telling apart,
+  // since a live cluster must not hide a dead one.
+  const sources = sw.sources.length > 1
+    ? sw.sources.map((src) => h('tr', { class: 'source-row' },
+      h('td', {}, switchBadge(src.status)),
+      h('td', { class: 'source-labels' },
+        ...Object.entries(src.labels || {})
+          .filter(([k]) => k !== 'alertname')
+          .map(([k, v]) => labelChip(k, v)),
+        !Object.keys(src.labels || {}).some((k) => k !== 'alertname')
+          && h('code', { class: 'small', text: src.fingerprint })),
+      h('td', { class: 'muted small' }, timeCell(src.last_heartbeat_at)),
+      h('td', { class: 'muted small' }, triggeredCell(src.last_triggered_at, src.incident_id)),
+      h('td'), h('td')))
+    : [];
+  return [main, ...sources];
+}
+
 function deadmanCard() {
-  const d = data.deadman || {};
+  const switches = data.deadman || [];
-  const matchers = h('input', {
-    type: 'text', value: d.matchers || '', placeholder: 'alertname=Watchdog',
-    class: 'wide',
-  });
-  const timeout = h('input', {
-    type: 'number', min: '0', class: 'setting-value',
-    value: String(Math.round((d.timeout_seconds || 0) / 60)),
-  });
-  const severity = h('select', {},
-    ...['critical', 'error', 'warning', 'info'].map((s) =>
-      h('option', { value: s, text: s, selected: (d.severity || 'critical') === s })));
-
-  const form = h('form', { class: 'stacked-form' },
-    h('label', {}, 'Heartbeat alerts ', matchers),
-    h('label', {}, 'Declare dead after ', timeout, ' minutes of silence'),
-    h('label', {}, 'Open the incident at severity ', severity),
-    h('button', { class: 'btn', type: 'submit', text: 'Save switches' }));
-
-  form.addEventListener('submit', (e) => {
-    e.preventDefault();
-    act(() => api.setDeadman(teamID, {
-      matchers: matchers.value.trim(),
-      timeout_seconds: Number(timeout.value) * 60,
-      severity: severity.value,
-    }));
-  });
 
   return h('div', { class: 'card' },
-    h('h2', { text: 'Dead man’s switches' }),
+    h('div', { class: 'card-head' },
+      h('h2', { text: 'Dead man’s switches' }),
+      isOwner() && h('button', {
+        class: 'btn', type: 'button', text: 'New switch', onclick: openNewSwitch,
+      })),
     h('p', { class: 'muted small' },
       'Alerts whose ABSENCE is the signal. Receiving one opens nothing; going ',
-      'quiet for longer than the timeout opens an incident. ',
+      'quiet for longer than the switch’s timeout opens an incident.'),
-      h('code', { text: 'alertname=Watchdog,cluster=prod; alertname=EdgeHeartbeat' }),
+    switches.length
-      ' — semicolons separate switches, commas separate conditions, and every ',
+      ? h('div', { class: 'table-scroll' },
-      'switch must name an alertname. Leave empty to watch nothing.'),
+        h('table', { class: 'admin-table status-table' },
-    isOwner() ? form : h('p', { class: 'muted', text: d.matchers || 'Nothing watched.' }),
+          h('thead', {}, h('tr', {},
+            h('th', { text: 'Status' }), h('th', { text: 'Switch' }),
+            h('th', { text: 'Last heartbeat' }), h('th', { text: 'Last triggered' }),
+            h('th', { text: 'Silent after' }), h('th'))),
+          h('tbody', {}, switches.flatMap(switchRows))))
+      : h('p', { class: 'muted', text: 'Nothing watched.' }),
   );
 }
 
+// The form lives in the sheet, not on the page: most visits are to look at the
+// list, and a form that is always open is the page this replaced.
+function openNewSwitch() {
+  const name = h('input', { type: 'text', placeholder: 'Prod Watchdog', autofocus: true });
+  const matcher = h('input', {
+    type: 'text', placeholder: 'alertname=Watchdog,cluster=prod', class: 'wide', required: true,
+  });
+  const timeout = h('input', {
+    type: 'number', min: '1', value: '15', class: 'setting-value', required: true,
+  });
+  const severity = h('select', {},
+    ...['critical', 'error', 'warning', 'info'].map((s) => h('option', { value: s, text: s })));
+  const problem = h('p', { class: 'load-error', hidden: true });
+
+  const form = h('form', { class: 'stacked-form' },
+    h('label', {}, 'Name (optional) ', name),
+    h('label', {}, 'Heartbeat alert ', matcher),
+    h('p', { class: 'muted small' },
+      'Conditions are ', h('code', { text: 'label=value' }), ' separated by commas, and one ',
+      'must be ', h('code', { text: 'alertname' }), '. Every distinct label set that ',
+      'matches is watched on its own.'),
+    h('label', {}, 'Declare dead after ', timeout, ' minutes of silence'),
+    h('label', {}, 'Open the incident at severity ', severity),
+    problem,
+    h('div', { class: 'sheet-actions' },
+      h('button', { class: 'btn', type: 'button', text: 'Cancel', onclick: () => closeSheet(false) }),
+      h('button', { class: 'btn btn-primary', type: 'submit', text: 'Add switch' })));
+
+  form.addEventListener('submit', async (e) => {
+    e.preventDefault();
+    try {
+      await api.createDeadmanSwitch(teamID, {
+        name: name.value.trim(),
+        matcher: matcher.value.trim(),
+        timeout_seconds: Math.round(Number(timeout.value) * 60),
+        severity: severity.value,
+      });
+    } catch (err) {
+      problem.textContent = err.message;
+      problem.hidden = false;
+      return;
+    }
+    closeSheet(true);
+    refresh();
+  });
+
+  openSheet(() => [h('h2', { class: 'sheet-title', text: 'New switch' }), form]);
+}
+
 // --- members ---------------------------------------------------------------
 
 function membersCard() {

internal/web/static/js/ui.js

@@ -48,6 +48,7 @@ const ICONS = {
   note: ['M5 4h14v12l-4 4H5z', 'M15 20v-4h4', 'M9 9h6M9 13h4'],
   archive: ['M3.5 5h17v4h-17z', 'M5 9v10h14V9', 'M10 13h4'],
   flag: ['M5 21V4', 'M5 4h11l-2 4 2 4H5'],
+  copy: ['rect:9,9,11,11,2', 'M5 15V6a2 2 0 0 1 2-2h9'],
   trash: ['M4 7h16', 'M9 7V4h6v3', 'M6 7l1 13h10l1-13'],
   chevronLeft: ['M15 18l-6-6 6-6'],
   chevronRight: ['M9 6l6 6-6 6'],