Note that a release does not bump the demos' terdut-server pin

terdut-demo in Ryuvia/charts, a TerdutServer CR reconciled by terdut-operator,
and the kind demo in terdut-operator both pin this image, and chart-bump only
moves the terdut-server wrapper. terdut-demo therefore sat at v0.37.0 through
v0.41.0-v0.43.0 before it was synced on 2026-10-08, and nothing said it was
behind. CLAUDE.md now lists the step after the release one: bump the demo's
tag to the same digest and its chart version, as its own PR, and read the
range's migrations first, since the demo's database migrates forward at start.

Documentation only: nothing is built from this file, so it needs no release.

CLAUDE.md

@@ -14,7 +14,7 @@ Preconditions and the plan, without side effects:
 Config is `.release.conf` here plus `make release-vars`. The process itself lives in
 `~/.claude/skills/release/`; why it is shaped this way is in README.md §Releasing.
 
-Two things about this repo specifically:
+Three things about this repo specifically:
 
 - **The image is scanned after it is published, not before.** `scan-image` runs trivy
   against the pushed image, because trivy cannot read a locally built one on this runner.
@@ -25,6 +25,16 @@ Two things about this repo specifically:
   so `chart-bump` needs `--image "$IMAGE"` to know which one moves. That sidecar backs up
   SQLite; the Postgres move (#2) retires it in favour of a `postgresql` CR with a k8up
   `pg_dump` annotation, after which only the app image's tag is left.
+- **Two demos pin this image, and `chart-bump` moves neither.** `terdut-demo` in
+  `Ryuvia/charts` is a `TerdutServer` CR that terdut-operator reconciles, and its
+  `values.yaml` `image.tag` is meant to match production's pin (same digest). The kind demo
+  in terdut-operator (`examples/demo/01-server.yaml`) pins a tag too. A release only bumps
+  the `terdut-server` wrapper, so both drift silently: `terdut-demo` sat at v0.37.0 through
+  v0.41.0-v0.43.0 until it was synced on 2026-10-08. After a release, bump `terdut-demo`'s
+  tag to the same `image-digest` and its `Chart.yaml` `version:` (Flux reconciles on
+  ChartVersion), as its own PR, and say in the release report whether you did. Neither
+  demo has anything but the pin to change, but read the version range's migrations first:
+  the demo's Postgres migrates forward at startup.
 
 ## Checks
 

README.md

@@ -444,6 +444,29 @@ high-water mark — the highest `severity` label any of its alerts has carried
 an incident that hit `critical` still reads as critical after the critical alert
 clears.
 
+### Several clusters, one team
+
+A team with one Alertmanager per Kubernetes cluster, each posting to its own
+source, needs two settings or the clusters run together.
+
+1. Give every alert a `cluster` label at the source. In Prometheus that is
+   `externalLabels: {cluster: prod-eu}` (kube-prometheus-stack:
+   `prometheus.prometheusSpec.externalLabels`).
+2. Add `cluster` to `group_by` in `alertmanager.yml`.
+
+The second one is the one that matters. Incidents are matched on the team and
+Alertmanager's `groupKey`, and the `groupKey` does not include external labels:
+without `cluster` in `group_by`, the same alert in two clusters has the same
+key and joins one incident. With it, each cluster gets its own, `cluster` is in
+the incident's `group_labels`, and the web UI shows it as a coloured chip on the
+queue, the incident and the alert list, instead of leaving it in the title.
+An alert that is not grouped by `cluster` still shows the chip on the alert
+list, which reads the label from the alert itself.
+
+The queue has a cluster dropdown once there are two or more values to choose
+between. It filters on the incident's `cluster` group label
+(`GET /api/incidents?cluster=...`), so it only sees incidents grouped by it.
+
 ### An incident opens only on a new occurrence
 
 An incident opens when an alert **transitions into firing**: a fingerprint that
@@ -926,7 +949,8 @@ the team gets the same `404` as anybody else.
 
 | Method | Path | Description |
 |---|---|---|
-| `GET` | `/api/incidents` | List incidents. Filters: `?status=triggered\|acknowledged\|resolved`, `?severity=`, `?assigned_to=<user id>`, `?archived=true`, `?snoozed=true`, `?from=YYYY-MM-DD`, `?to=YYYY-MM-DD`, `?sort=severity`, `?limit=` (default 50, max 500) |
+| `GET` | `/api/incidents` | List incidents. Filters: `?status=triggered\|acknowledged\|resolved`, `?severity=`, `?assigned_to=<user id>`, `?archived=true`, `?snoozed=true`, `?from=YYYY-MM-DD`, `?to=YYYY-MM-DD`, `?sort=severity`, `?cluster=<value of the cluster group label>`, `?limit=` (default 50, max 500) |
+| `GET` | `/api/incidents/clusters` | The distinct `cluster` values on the caller's incidents from the last 90 days, sorted (`?team_id=` narrows it). An empty array when nothing carries the label |
 | `GET` | `/api/incidents/{id}` | Get single incident, with its alerts inline |
 | `GET` | `/api/incidents/{id}/alerts` | Alerts under this incident |
 | `GET` | `/api/incidents/{id}/timeline` | Full event history, chronological |

charts/terdut-server/Chart.yaml

@@ -15,5 +15,5 @@ type: application
 # appVersion and image.tag in values.yaml no longer agree, and that is not an oversight:
 # image.tag stays "latest", which is what a local install actually pulls. appVersion is
 # metadata and drives nothing.
-version: 0.41.2
+version: 0.43.0
-appVersion: "v0.41.2"
+appVersion: "v0.43.0"

internal/api/incidents.go

@@ -13,6 +13,52 @@ import (
 	"github.com/go-chi/chi/v5"
 )
 
+// handleListClusters answers GET /api/incidents/clusters: the distinct values of
+// the origin label across the caller's incidents from the last 90 days, sorted,
+// so the queue can offer them as a filter. Optional team_id narrows it to one
+// team. Empty when nothing carries the label, which is how the UI knows to show
+// no filter at all.
+func handleListClusters(db *sql.DB) http.HandlerFunc {
+	return func(w http.ResponseWriter, r *http.Request) {
+		args := &sqlArgs{}
+		where := []string{
+			"team_id = ANY(" + args.add(callerTeamIDs(r.Context())) + ")",
+			"triggered_at >= " + args.add(time.Now().AddDate(0, 0, -90).Unix()),
+		}
+		if team := r.URL.Query().Get("team_id"); team != "" {
+			if n, err := strconv.ParseInt(team, 10, 64); err == nil {
+				where = append(where, "team_id = "+args.add(n))
+			}
+		}
+		label := args.add(originLabel)
+
+		rows, err := db.QueryContext(r.Context(),
+			fmt.Sprintf("SELECT DISTINCT group_labels->>%[1]s AS v FROM incidents WHERE %[2]s AND group_labels->>%[1]s <> '' ORDER BY v LIMIT 200",
+				label, strings.Join(where, " AND ")),
+			args.all()...)
+		if err != nil {
+			respond(w, http.StatusInternalServerError, errResp("internal error"))
+			return
+		}
+		defer rows.Close()
+
+		clusters := []string{}
+		for rows.Next() {
+			var v string
+			if err := rows.Scan(&v); err != nil {
+				respond(w, http.StatusInternalServerError, errResp("internal error"))
+				return
+			}
+			clusters = append(clusters, v)
+		}
+		if err := rows.Err(); err != nil {
+			respond(w, http.StatusInternalServerError, errResp("internal error"))
+			return
+		}
+		respond(w, http.StatusOK, clusters)
+	}
+}
+
 func handleListIncidents(db *sql.DB) http.HandlerFunc {
 	return func(w http.ResponseWriter, r *http.Request) {
 		q := r.URL.Query()
@@ -53,6 +99,12 @@ func handleListIncidents(db *sql.DB) http.HandlerFunc {
 		if severity := q.Get("severity"); severity != "" {
 			where = append(where, "i.severity = "+args.add(severity))
 		}
+		// Where it came from: the value of the origin label (originLabel, by
+		// convention "cluster") among the incident's group labels. An incident has
+		// it only when the label is in Alertmanager's group_by.
+		if cluster := q.Get("cluster"); cluster != "" {
+			where = append(where, "i.group_labels->>"+args.add(originLabel)+" = "+args.add(cluster))
+		}
 		if assignee := q.Get("assigned_to"); assignee != "" {
 			if n, err := strconv.ParseInt(assignee, 10, 64); err == nil {
 				where = append(where, "i.assigned_to = "+args.add(n))

internal/api/incidents_test.go

@@ -4,6 +4,7 @@ import (
 	"context"
 	"fmt"
 	"net/http"
+	"strings"
 	"testing"
 	"time"
 
@@ -972,3 +973,37 @@ func TestServiceAccount_AssignArchiveUnarchiveRecordActor(t *testing.T) {
 		}
 	}
 }
+
+// The queue can be narrowed to one cluster, and the distinct clusters are
+// offered so the filter has something to list.
+func TestIncidents_FilterByCluster(t *testing.T) {
+	s := newTS(t)
+
+	for _, c := range []string{"prod-eu", "prod-us"} {
+		fireGroupedAs(t, s, "fp-"+c,
+			map[string]string{"alertname": "PodRestarting", "cluster": c, "namespace": "n"},
+			`{}:{alertname="PodRestarting",cluster="`+c+`",namespace="n"}`)
+	}
+	// A cluster-less incident exists too, and must never match a cluster filter.
+	fireGroupedAs(t, s, "fp-none", map[string]string{"alertname": "DiskFull"}, `{}:{alertname="DiskFull"}`)
+
+	if got := len(listIncidents(t, s, "")); got != 3 {
+		t.Fatalf("expected 3 open incidents, got %d", got)
+	}
+	eu := listIncidents(t, s, "?cluster=prod-eu")
+	if len(eu) != 1 {
+		t.Fatalf("expected 1 incident for prod-eu, got %d", len(eu))
+	}
+	if labels, _ := eu[0]["group_labels"].(map[string]any); labels["cluster"] != "prod-eu" {
+		t.Errorf("filtered to the wrong cluster: %v", eu[0]["group_labels"])
+	}
+	if got := len(listIncidents(t, s, "?cluster=nowhere")); got != 0 {
+		t.Errorf("an unknown cluster should match nothing, got %d", got)
+	}
+
+	var clusters []string
+	decode(t, s.req(t, http.MethodGet, "/api/incidents/clusters", nil), &clusters)
+	if want := "prod-eu,prod-us"; strings.Join(clusters, ",") != want {
+		t.Errorf("clusters = %v, want %s", clusters, want)
+	}
+}

internal/api/notifier.go

@@ -8,6 +8,7 @@ import (
 	"fmt"
 	"log"
 	"net/http"
+	"regexp"
 	"strings"
 	"time"
 
@@ -409,9 +410,11 @@ func renderNotification(inc models.Incident, n outboxRow, firing int, cfg Notify
 			strings.TrimSuffix(cfg.PublicURL, "/"), inc.ID)
 	}
 
+	title := pageTitle(inc)
+
 	switch n.kind {
 	case notifyResolved:
-		msg.Title = "Resolved: " + inc.Title
+		msg.Title = "Resolved: " + title
 		msg.Message = "All alerts stopped firing after " +
 			humanDuration(time.Since(inc.TriggeredAt))
 		msg.Priority = ntfyPriorityLow
@@ -419,9 +422,9 @@ func renderNotification(inc models.Incident, n outboxRow, firing int, cfg Notify
 		return msg
 
 	case notifyReminder:
-		msg.Title = "Still unacknowledged: " + inc.Title
+		msg.Title = "Still unacknowledged: " + title
 	default:
-		msg.Title = inc.Title
+		msg.Title = title
 	}
 
 	severity := derefString(inc.Severity)
@@ -443,6 +446,48 @@ func renderNotification(inc models.Incident, n outboxRow, firing int, cfg Notify
 	return msg
 }
 
+// originLabel is the label that says where an alert came from, for a team with
+// several Kubernetes clusters behind it. It comes from Prometheus's
+// externalLabels and reaches an incident through Alertmanager's group_by; the
+// web UI reads the same label, and the README ("Several clusters, one team")
+// explains how to set it up.
+const originLabel = "cluster"
+
+// pageTitle is the incident's title for a notification. A phone's lock screen
+// cuts a long title off at the end, and the incident title puts the grouping
+// labels there, so the cluster would be the first thing lost. When the incident
+// has an origin it leads instead, "[prod-eu] PodRestarting (namespace=foo)", and
+// is dropped from the parenthesis so it is not said twice. A title that is not
+// in incidentTitle's "name (k=v, k=v)" shape keeps its text and gains the prefix.
+func pageTitle(inc models.Incident) string {
+	origin := inc.GroupLabels[originLabel]
+	if origin == "" {
+		return inc.Title
+	}
+	return "[" + origin + "] " + titleWithoutLabel(inc.Title, originLabel, origin)
+}
+
+var titleShape = regexp.MustCompile(`(?s)^(.*?) \((.*)\)$`)
+
+// titleWithoutLabel removes "key=value" from the parenthesised tail of a title
+// built by incidentTitle, and the parentheses with it if nothing else is left.
+func titleWithoutLabel(title, key, value string) string {
+	m := titleShape.FindStringSubmatch(title)
+	if m == nil {
+		return title
+	}
+	var rest []string
+	for _, part := range strings.Split(m[2], ", ") {
+		if part != key+"="+value {
+			rest = append(rest, part)
+		}
+	}
+	if len(rest) == 0 {
+		return m[1]
+	}
+	return m[1] + " (" + strings.Join(rest, ", ") + ")"
+}
+
 // ntfy's priority scale. Max is the one that overrides the phone's quiet
 // settings, which is the whole point of paging on critical.
 const (

internal/api/notify_test.go

@@ -1,6 +1,7 @@
 package api_test
 
 import (
+	"bytes"
 	"context"
 	"encoding/json"
 	"fmt"
@@ -164,10 +165,91 @@ func fireCritical(t *testing.T, s *ts) {
 	}, "{}:{alertname=\"DiskFull\"}")
 }
 
+// fireGrouped posts one critical alert whose Alertmanager group carries the
+// given labels, the way group_by puts them on the webhook.
+func fireGrouped(t *testing.T, s *ts, groupLabels map[string]string, groupKey string) {
+	t.Helper()
+	fireGroupedAs(t, s, "fp-grouped", groupLabels, groupKey)
+}
+
+// fireGroupedAs is fireGrouped with its own alert fingerprint, for a test that
+// needs several alerts open at once.
+func fireGroupedAs(t *testing.T, s *ts, fingerprint string, groupLabels map[string]string, groupKey string) {
+	t.Helper()
+	payload := map[string]any{
+		"version": "4", "status": "firing", "groupKey": groupKey, "groupLabels": groupLabels,
+		"alerts": []map[string]any{amAlert(fingerprint, "PodRestarting", "firing",
+			"2026-05-20T10:00:00Z", zeroTime, map[string]string{"severity": "critical"})},
+	}
+	data, _ := json.Marshal(payload)
+	resp, err := http.Post(s.URL+"/api/integrations/"+s.ingestKey+"/alertmanager",
+		"application/json", bytes.NewReader(data))
+	if err != nil {
+		t.Fatalf("post webhook: %v", err)
+	}
+	resp.Body.Close()
+	if resp.StatusCode != http.StatusOK {
+		t.Fatalf("webhook returned %d", resp.StatusCode)
+	}
+}
+
 // ---------------------------------------------------------------------------
 // Delivery
 // ---------------------------------------------------------------------------
 
+// A phone cuts a long title off at the end, and the incident title keeps the
+// grouping labels there, so the cluster leads the page instead.
+func TestNotify_ClusterLeadsTheTitle(t *testing.T) {
+	s, f := notifyTS(t, api.NotifyConfig{PublicURL: "https://terdut.example.com"})
+
+	fireGrouped(t, s, map[string]string{
+		"alertname": "PodRestarting", "cluster": "prod-eu", "namespace": "shop",
+	}, `{}:{alertname="PodRestarting",cluster="prod-eu",namespace="shop"}`)
+	s.sweepNotify(t)
+
+	msgs := f.messages()
+	if len(msgs) != 1 {
+		t.Fatalf("expected 1 push, got %d", len(msgs))
+	}
+	if want := "[prod-eu] PodRestarting (namespace=shop)"; msgs[0].Title != want {
+		t.Errorf("title = %q, want %q", msgs[0].Title, want)
+	}
+}
+
+// Only the cluster is the whole grouping: no parenthesis is left behind.
+func TestNotify_ClusterAloneLeavesNoParenthesis(t *testing.T) {
+	s, f := notifyTS(t, api.NotifyConfig{})
+
+	fireGrouped(t, s, map[string]string{"alertname": "PodRestarting", "cluster": "prod-eu"},
+		`{}:{alertname="PodRestarting",cluster="prod-eu"}`)
+	s.sweepNotify(t)
+
+	msgs := f.messages()
+	if len(msgs) != 1 {
+		t.Fatalf("expected 1 push, got %d", len(msgs))
+	}
+	if want := "[prod-eu] PodRestarting"; msgs[0].Title != want {
+		t.Errorf("title = %q, want %q", msgs[0].Title, want)
+	}
+}
+
+// Nothing changes for a team whose alerts have no cluster label.
+func TestNotify_NoClusterKeepsTheTitle(t *testing.T) {
+	s, f := notifyTS(t, api.NotifyConfig{})
+
+	fireGrouped(t, s, map[string]string{"alertname": "PodRestarting", "namespace": "shop"},
+		`{}:{alertname="PodRestarting",namespace="shop"}`)
+	s.sweepNotify(t)
+
+	msgs := f.messages()
+	if len(msgs) != 1 {
+		t.Fatalf("expected 1 push, got %d", len(msgs))
+	}
+	if want := "PodRestarting (namespace=shop)"; msgs[0].Title != want {
+		t.Errorf("title = %q, want %q", msgs[0].Title, want)
+	}
+}
+
 func TestNotify_TriggeredIncidentPagesOnCall(t *testing.T) {
 	s, f := notifyTS(t, api.NotifyConfig{PublicURL: "https://terdut.example.com"})
 

internal/api/router.go

@@ -145,6 +145,7 @@ func NewRouter(db *sql.DB, notify NotifyConfig, cfg config.Config, version strin
 		r.Get("/api/alerts/{id}", handleGetAlert(db))
 
 		r.Get("/api/incidents", handleListIncidents(db))
+		r.Get("/api/incidents/clusters", handleListClusters(db))
 		r.Get("/api/incidents/{id}", handleGetIncident(db))
 		r.Get("/api/incidents/{id}/alerts", handleIncidentAlerts(db))
 		r.Get("/api/incidents/{id}/timeline", handleIncidentTimeline(db))

internal/web/static/app.css

@@ -395,16 +395,23 @@ input:focus, textarea:focus { outline: none; border-color: var(--accent); box-sh
 .chip .count { margin-left: 4px; opacity: 0.7; }
 /* Nothing in it: step back so the chips that have something stand out. */
 .chip .count.zero { opacity: 0.4; }
-/* An overlay, not a flex item: absolute against .chips' own (non-scrolling)
+/* A scrolling strip fades on the right edge while there is more to scroll to
-   box stays flush with its real right edge regardless of scroll position,
+   (fadeOnOverflow in ui.js sets data-more). A mask on the strip itself, not an
-   which turned out not to be true of position:sticky here — as a flex
+   element inside it: anything inside a scroller scrolls away with the content,
-   item, its sticky offset interacted with the row's gap and its own
+   which is what the old overlay did. */
-   negative margin, landing short of the edge by about one gap's width. */
+[data-more] {
-.chips-fade {
+  -webkit-mask-image: linear-gradient(90deg, #000 calc(100% - 28px), transparent);
-  position: absolute; top: 0; right: 0; bottom: 0;
+  mask-image: linear-gradient(90deg, #000 calc(100% - 28px), transparent);
-  width: 24px;
+}
-  background: linear-gradient(to right, transparent, var(--bg));
+
-  pointer-events: none;
+/* The queue's cluster filter: a pill-shaped select under the status chips. */
+.queue-origin { position: relative; display: flex; align-items: center; padding: 0 16px 8px; }
+.origin-select-icon { position: absolute; left: 28px; width: 16px; height: 16px; color: var(--muted); pointer-events: none; }
+.origin-select {
+  min-height: 34px; max-width: 100%; padding: 0 12px 0 34px;
+  border: 1px solid var(--border-strong); border-radius: 999px;
+  background: var(--surface); color: var(--text);
+  font: inherit; font-size: 13px; font-weight: 600; cursor: pointer;
 }
 
 /* ---------- lists ---------- */
@@ -491,7 +498,15 @@ input:focus, textarea:focus { outline: none; border-color: var(--accent); box-sh
 }
 .badge::before { content: ""; width: 7px; height: 7px; border-radius: 50%; background: currentColor; }
 .badge.plain::before { display: none; }
-.badge .badge-icon { width: 12px; height: 12px; stroke-width: 2.4; }
+.badge .badge-icon, .origin-chip .badge-icon { width: 12px; height: 12px; stroke-width: 2.4; }
+/* The cluster an incident or alert came from. Same shape as a badge, coloured
+   from the rcN palette (see originClass in format.js), never the severity one. */
+.origin-chip {
+  display: inline-flex; align-items: center; gap: 5px; max-width: 100%;
+  padding: 1px 8px; border-radius: 999px;
+  font-size: 12px; font-weight: 700; letter-spacing: 0.01em; white-space: nowrap;
+  overflow: hidden; text-overflow: ellipsis;
+}
 .badge.st-triggered, .badge.st-firing { background: var(--crit-soft); color: var(--crit); }
 .badge.st-acknowledged { background: var(--warn-soft); color: var(--warn); }
 .badge.st-snoozed { background: var(--snooze-soft); color: var(--snooze); }
@@ -886,7 +901,6 @@ kbd {
   /* The pane is 340-420px wide and a mouse cannot scroll a row whose scrollbar
      is hidden, so the chips wrap here instead: Archived stays reachable. */
   .pane-list .chips { flex-wrap: wrap; overflow-x: visible; }
-  .chips-fade { display: none; }
   /* With nothing selected there is no detail to show next to, so the list
      takes the whole row instead of leaving the second column as dead space
      around the placeholder text. Selecting an incident (.has-detail) drops
@@ -1236,11 +1250,6 @@ button.rota-week:hover { background: var(--surface-2); color: var(--text); }
   overflow-x: auto; scrollbar-width: none;
 }
 .subnav::-webkit-scrollbar { display: none; }
-/* A fade on the right edge says "there is more" where the strip overflows;
-   the desktop width fits every entry, so it is phone-only. */
-@media (max-width: 899px) {
-  .subnav { -webkit-mask-image: linear-gradient(90deg, #000 calc(100% - 28px), transparent); mask-image: linear-gradient(90deg, #000 calc(100% - 28px), transparent); }
-}
 .subnav-link {
   flex: none;
   padding: 8px 12px; margin-bottom: -1px;

internal/web/static/index.html

@@ -154,6 +154,9 @@
     <section id="view-queue" class="view view-queue" data-view="queue">
       <div class="pane pane-list">
         <div class="chips" id="queue-filters" role="tablist" aria-label="Filter"></div>
+        <!-- The cluster filter. Hidden until the queue has seen two or more
+             clusters; queue.js fills it in. -->
+        <div class="queue-origin" id="queue-origin" hidden></div>
         <div id="queue-list" class="list"></div>
       </div>
       <div class="pane pane-detail" id="detail" aria-live="polite"></div>

internal/web/static/js/admin.js

@@ -13,7 +13,7 @@
 // gate.
 
 import * as api from './api.js';
-import { h, clear, spinner, confirm, menuCard, ssoBadge, SSO_MANAGED } from './ui.js';
+import { h, clear, spinner, confirm, menuCard, ssoBadge, SSO_MANAGED, fadeOnOverflow } from './ui.js';
 import { state, myID } from './state.js';
 
 const view = () => document.getElementById('view-admin');
@@ -111,6 +111,7 @@ function subnav() {
     })));
   // On a phone the strip overflows; bring the open section into view so a
   // tab past the edge (Sources, Switches) is never the one that is hidden.
+  fadeOnOverflow(nav);
   requestAnimationFrame(() => nav.querySelector('[aria-current]')
     ?.scrollIntoView({ inline: 'center', block: 'nearest' }));
   return nav;

internal/web/static/js/alerts.js

@@ -2,8 +2,8 @@
 // incident it belongs to, which is where anything can be done about it.
 
 import * as api from './api.js';
-import { h, clear, badge, severityBadge, emptyState, spinner } from './ui.js';
+import { h, clear, badge, severityBadge, originChip, emptyState, spinner } from './ui.js';
-import { age, labelSummary } from './format.js';
+import { age, labelSummary, originOf, ORIGIN_LABEL } from './format.js';
 
 const FILTERS = [
   { id: 'firing', label: 'Firing', query: { status: 'firing' } },
@@ -77,7 +77,8 @@ function row(a) {
   const summary = (a.annotations && a.annotations.summary) || '';
   const sev = a.labels && a.labels.severity;
   const labels = labelSummary(Object.fromEntries(
-    Object.entries(a.labels || {}).filter(([k]) => k !== 'severity')));
+    Object.entries(a.labels || {}).filter(([k]) => k !== 'severity' && k !== ORIGIN_LABEL)));
+  const origin = originOf(a.labels);
   const linked = a.incident_id != null;
   return h(linked ? 'a' : 'div', {
     class: `row st-${a.status} ${linked ? '' : 'no-link'}`,
@@ -86,6 +87,7 @@ function row(a) {
     h('div', { class: 'row-title', text: a.name }),
     h('div', { class: 'row-age', title: a.starts_at, text: age(a.status === 'firing' ? a.starts_at : a.received_at) }),
     h('div', { class: 'row-meta' },
+      origin && originChip(origin),
       badge(a.status === 'firing' ? 'Firing' : 'Resolved', `st-${a.status}`),
       sev && severityBadge(sev),
       summary && h('span', { text: summary }),

internal/web/static/js/api.js

@@ -83,6 +83,7 @@ export const setNotifyTarget = (id, ntfyTopic) =>
 
 // incidents
 export const incidents = (query, opts) => call('GET', '/incidents', { query, ...opts });
+export const clusters = (query) => call('GET', '/incidents/clusters', { query });
 export const incident = (id) => call('GET', `/incidents/${id}`);
 export const timeline = (id) => call('GET', `/incidents/${id}/timeline`);
 

internal/web/static/js/format.js

@@ -117,3 +117,36 @@ export function labelSummary(labels, skip = 'alertname') {
 export function initial(name) {
   return (name || '?').trim().charAt(0) || '?';
 }
+
+// Where an incident or alert came from, for a team with several Kubernetes
+// clusters (or other origins) behind it: the value of one label, `cluster` by
+// convention. It comes from Prometheus's externalLabels, so it is on every
+// alert; an incident carries it only when it is in Alertmanager's group_by,
+// which is also what keeps two clusters' identical alerts from merging into one
+// incident (see the README, "Several clusters, one team").
+export const ORIGIN_LABEL = 'cluster';
+
+export function originOf(labels) {
+  const v = labels && labels[ORIGIN_LABEL];
+  return v ? String(v) : null;
+}
+
+// A stable colour for an origin, from the six-colour rcN palette app.css has,
+// so "prod-eu" is the same colour on every row without anything being stored.
+export function originClass(value) {
+  let n = 0;
+  for (const ch of value) n = (n * 31 + ch.charCodeAt(0)) >>> 0;
+  return `rc${(n % 6) + 1}`;
+}
+
+// The title without its `cluster=...` part, for a row that shows the origin as
+// a chip: the server puts every grouping label in the title, so the cluster
+// would otherwise appear twice. Anything that does not look like the server's
+// "name (k=v, k=v)" shape is returned unchanged.
+export function titleWithoutOrigin(title, value) {
+  if (!value) return title;
+  const m = title.match(/^(.*?) \((.*)\)$/s);
+  if (!m) return title;
+  const rest = m[2].split(', ').filter((p) => p !== `${ORIGIN_LABEL}=${value}`);
+  return rest.length ? `${m[1]} (${rest.join(', ')})` : m[1];
+}

internal/web/static/js/incident.js

@@ -4,10 +4,10 @@
 import * as api from './api.js';
 import * as poll from './poll.js';
 import {
-  h, clear, icon, badge, severityBadge, labelChip, openSheet, closeSheet, confirm, toast, spinner, emptyState,
+  h, clear, icon, badge, severityBadge, originChip, labelChip, openSheet, closeSheet, confirm, toast, spinner, emptyState,
 } from './ui.js';
 import {
-  ago, when, until, duration, isFuture, STATUS_LABEL,
+  ago, when, until, duration, isFuture, originOf, STATUS_LABEL,
 } from './format.js';
 import { myID, users } from './state.js';
 import { back } from './app.js';
@@ -99,6 +99,8 @@ function render() {
 
 function statusBadges() {
   const out = [];
+  const origin = originOf(inc.group_labels);
+  if (origin) out.push(originChip(origin));
   if (inc.severity) out.push(severityBadge(inc.severity));
   out.push(badge(STATUS_LABEL[inc.status] || inc.status, `st-${inc.status}`));
   if (inc.status !== 'resolved' && isFuture(inc.snoozed_until)) {

internal/web/static/js/queue.js

@@ -1,8 +1,8 @@
 // The incident queue: filter chips and a list of incident rows.
 
 import * as api from './api.js';
-import { h, clear, badge, severityBadge, emptyState, spinner } from './ui.js';
+import { h, clear, icon, badge, severityBadge, originChip, emptyState, spinner, fadeOnOverflow } from './ui.js';
-import { ago, until, isFuture, severityClass, labelSummary } from './format.js';
+import { ago, until, isFuture, severityClass, labelSummary, originOf, titleWithoutOrigin } from './format.js';
 import { state, myID, onTeamChange } from './state.js';
 import * as onboarding from './onboarding.js';
 import { navigate } from './app.js';
@@ -30,8 +30,10 @@ onboarding.onRerender(() => renderList());
 // The queue used to keep its own team filter (a per-tab sessionStorage value,
 // out of step with team.js's own picker); both now defer to the global
 // selector's shared state, so re-render whenever it changes.
-onTeamChange(() => {
+onTeamChange(async () => {
   renderChips();
+  await loadClusters(true);
+  renderOrigin();
   refresh({ fresh: true });
 });
 
@@ -52,6 +54,63 @@ function loadFilter() {
   return 'open';
 }
 
+// Which cluster the queue is narrowed to, or null for all of them. Kept in
+// localStorage like the selected team: it is a standing choice, not a
+// per-tab one.
+const ORIGIN_KEY = 'terdut.queue.cluster';
+let origin = loadOrigin();
+let clusters = []; // the clusters the server has seen, for the dropdown
+let clustersAt = 0;
+
+function loadOrigin() {
+  try {
+    return localStorage.getItem(ORIGIN_KEY) || null;
+  } catch {
+    return null; // storage unavailable
+  }
+}
+
+function saveOrigin() {
+  try {
+    if (origin) localStorage.setItem(ORIGIN_KEY, origin);
+    else localStorage.removeItem(ORIGIN_KEY);
+  } catch {
+    /* storage unavailable */
+  }
+}
+
+// The clusters to offer, from the server because the list on screen is only
+// the fifty most recent. Asked again at most every five minutes; a cluster
+// first met in an incident on screen is added at once (see noteClusters). A
+// remembered choice the server no longer knows would filter to nothing with no
+// hint why, so it is dropped.
+async function loadClusters(force = false) {
+  if (!force && Date.now() - clustersAt < 5 * 60 * 1000) return;
+  try {
+    const query = state.selectedTeamID != null ? { team_id: state.selectedTeamID } : undefined;
+    clusters = await api.clusters(query);
+    clustersAt = Date.now();
+    if (origin && !clusters.includes(origin)) {
+      origin = null;
+      saveOrigin();
+    }
+  } catch {
+    /* the filter is a convenience: without the list it stays hidden */
+  }
+}
+
+function noteClusters(list) {
+  let added = false;
+  for (const inc of list || []) {
+    const c = originOf(inc.group_labels);
+    if (c && !clusters.includes(c)) {
+      clusters = [...clusters, c].sort();
+      added = true;
+    }
+  }
+  return added;
+}
+
 function saveFilter() {
   try {
     sessionStorage.setItem('terdut.queue.filter', filter);
@@ -64,6 +123,7 @@ export function show(incidentID) {
   selected = incidentID;
   if (!built) {
     renderChips();
+    renderOrigin();
     built = true;
   }
   renderList();
@@ -73,15 +133,22 @@ export async function refresh({ fresh = false } = {}) {
   const f = FILTERS.find((x) => x.id === filter);
   const requested = filter;
   try {
+    // First, so a remembered cluster the server no longer knows is dropped
+    // before it is sent as a filter.
+    await loadClusters();
     // The open list is already fetched for the badges; no need to ask twice.
     // The cached open queue covers every team, so it can only be reused when
     // no team filter is applied.
-    const query = state.selectedTeamID != null ? { ...f.query, team_id: state.selectedTeamID } : f.query;
+    const query = { ...f.query };
-    const cached = filter === 'open' && !fresh && state.selectedTeamID == null;
+    if (state.selectedTeamID != null) query.team_id = state.selectedTeamID;
+    if (origin) query.cluster = origin;
+    const cached = filter === 'open' && !fresh && state.selectedTeamID == null && !origin;
     const result = cached ? state.open : await api.incidents(query);
     await onboarding.load();
     if (requested !== filter) return;
     items = result;
+    noteClusters(state.open);
+    noteClusters(items);
     error = null;
   } catch (err) {
     if (requested !== filter) return;
@@ -91,6 +158,7 @@ export async function refresh({ fresh = false } = {}) {
   // whichever caller updated it before calling here — app.js's poll, or the
   // `cached` branch above.
   renderChips();
+  renderOrigin();
   renderList();
 }
 
@@ -109,15 +177,19 @@ function setFilter(id) {
 // for the badges — Snoozed/Resolved/Archived would need a request of their
 // own, so those chips stay count-less for now.
 function chipCount(id) {
-  const open = state.selectedTeamID == null
+  const open = state.open.filter((i) =>
-    ? state.open
+    (state.selectedTeamID == null || i.team_id === state.selectedTeamID)
-    : state.open.filter((i) => i.team_id === state.selectedTeamID);
+    && (!origin || originOf(i.group_labels) === origin));
   if (id === 'open') return open.length;
   if (id === 'triggered') return open.filter((i) => i.status === 'triggered').length;
   if (id === 'acknowledged') return open.filter((i) => i.status === 'acknowledged').length;
   return null;
 }
 
+// Set the first time the chips are drawn: the strip element is the same one
+// for the life of the page, so one scroll listener and one observer will do.
+let chipsFade;
+
 function renderChips() {
   const el = document.getElementById('queue-filters');
   const chips = FILTERS.map((f) => {
@@ -135,9 +207,33 @@ function renderChips() {
   // There are no team chips here: the team selector in the sidebar (and in the
   // phone's top bar) is the one place the team is chosen, and chips for it
   // would be the same choice offered twice.
-  chips.push(h('span', { class: 'chips-fade', 'aria-hidden': 'true' }));
-
   clear(el, chips);
+  chipsFade ??= fadeOnOverflow(el);
+  chipsFade();
+}
+
+// The cluster dropdown, under the status chips. Not shown at all until there
+// are two clusters to choose between, the same rule the team selector follows.
+function renderOrigin() {
+  const el = document.getElementById('queue-origin');
+  if (clusters.length < 2) {
+    el.hidden = true;
+    return;
+  }
+  const select = h('select', { class: 'origin-select', 'aria-label': 'Cluster' },
+    h('option', { value: '', text: 'All clusters', selected: !origin }),
+    clusters.map((c) => h('option', { value: c, text: c, selected: c === origin })));
+  select.addEventListener('change', () => {
+    origin = select.value || null;
+    saveOrigin();
+    items = null;
+    cursor = -1;
+    renderChips();
+    renderList();
+    refresh({ fresh: true });
+  });
+  clear(el, icon('server', 'icon origin-select-icon'), select);
+  el.hidden = false;
 }
 
 function renderList() {
@@ -197,15 +293,18 @@ function row(inc, index) {
     ? h('span', { class: 'row-team', text: inc.team_name })
     : null;
 
+  const rowOrigin = originOf(inc.group_labels);
+
   return h('a', {
     class: `row ${severityClass(inc.severity)} ${resolved ? 'resolved' : ''} ${index === cursor ? 'kbd-focus' : ''}`,
     href: `/incidents/${inc.id}`,
     'aria-current': inc.id === selected ? 'true' : null,
     dataset: { index: String(index) },
   },
-    h('div', { class: 'row-title', text: inc.title }),
+    h('div', { class: 'row-title', text: titleWithoutOrigin(inc.title, rowOrigin) }),
     h('div', { class: 'row-age', title: inc.triggered_at, text: `Triggered ${ago(inc.triggered_at)}` }),
     h('div', { class: 'row-meta' },
+      rowOrigin && !origin && originChip(rowOrigin),
       !impliedStatus && status,
       // The left-border colour alone doesn't say what it means; spell it out
       // too, same badge the incident detail page uses for severity.

internal/web/static/js/team.js

@@ -18,7 +18,7 @@
 // than no form, but it is not the thing enforcing anything.
 
 import * as api from './api.js';
-import { h, clear, spinner, confirm, icon, openSheet, closeSheet, menuCard, badge, labelChip, ssoBadge, SSO_MANAGED } from './ui.js';
+import { h, clear, spinner, confirm, icon, openSheet, closeSheet, menuCard, badge, labelChip, ssoBadge, SSO_MANAGED, fadeOnOverflow } from './ui.js';
 import { state, currentTeam, onTeamChange, users as allUsers, myID } from './state.js';
 import { isoDate, addDays, mondayOf, isoWeek, initial, ago, when, duration } from './format.js';
 
@@ -177,6 +177,7 @@ function subnav() {
     })));
   // On a phone the strip overflows; bring the open section into view so a
   // tab past the edge (Sources, Switches) is never the one that is hidden.
+  fadeOnOverflow(nav);
   requestAnimationFrame(() => nav.querySelector('[aria-current]')
     ?.scrollIntoView({ inline: 'center', block: 'nearest' }));
   return nav;

internal/web/static/js/ui.js

@@ -1,6 +1,6 @@
 // DOM helpers, the bottom sheet, confirmation and toasts.
 
-import { severityClass } from './format.js';
+import { severityClass, originClass } from './format.js';
 
 // h builds an element. attrs: class, text, on<event>, dataset, aria/other
 // attributes; boolean true sets an empty attribute, false/null skips it.
@@ -32,8 +32,24 @@ export function clear(el, ...children) {
   return el;
 }
 
+// Marks a horizontally scrolling strip with data-more while there is more of it
+// to the right, which the CSS turns into a fade on that edge. The fade is a
+// mask on the strip itself, so it stays put at the edge of the screen: a fade
+// drawn as a child of the scroller scrolls away with the content. Returns the
+// update function, for a strip whose contents change under it.
+export function fadeOnOverflow(el) {
+  const update = () => {
+    el.toggleAttribute('data-more', el.scrollLeft + el.clientWidth < el.scrollWidth - 1);
+  };
+  el.addEventListener('scroll', update, { passive: true });
+  if (typeof ResizeObserver === 'function') new ResizeObserver(update).observe(el);
+  update();
+  return update;
+}
+
 // Stroke icons, 24×24. Built as SVG nodes so the CSP needs no inline anything.
 const ICONS = {
+  server: ['rect:3,4,18,6,2', 'rect:3,14,18,6,2', 'M7 7h.01', 'M7 17h.01'],
   alertTriangle: ['M12 4l9.5 16.5h-19z', 'M12 10v4.5', 'M12 17.5h.01'],
   alertCircle: ['circle:12,12,9', 'M12 7.5v5', 'M12 16h.01'],
   infoCircle: ['circle:12,12,9', 'M12 11v5.5', 'M12 7.5h.01'],
@@ -178,6 +194,13 @@ export function badge(text, cls = '') {
   return h('span', { class: `badge ${cls}`, text });
 }
 
+// Where it came from (the cluster): a chip in that origin's colour, with a
+// server icon so it reads as a place and not as a status.
+export function originChip(value) {
+  return h('span', { class: `origin-chip ${originClass(value)}`, title: `Origin: ${value}` },
+    icon('server', 'icon badge-icon'), value);
+}
+
 // The severity as a badge with a shape as well as a colour: a triangle for
 // critical, a circle with ! for warning, a circle with i for anything else, so
 // the coloured bar and badge are not the only thing that says it.