Show which cluster an incident came from, as a chip

A team with one Alertmanager per Kubernetes cluster could not tell at a
glance where an incident started: the cluster was only a word inside the
title. When an alert carries a `cluster` label, the queue rows, the incident
page and the alerts list now show it as a chip in that cluster's colour, a
stable pick from the existing six-colour palette. A queue row drops
`cluster=...` from its title, since the chip says it, and the incident page
keeps the full title.

An incident has the label only when it is in Alertmanager's group_by, which
is also what keeps two clusters' identical alerts apart: incidents are matched
on the team and the groupKey, and the groupKey does not include external
labels. The README has a section on the two settings (Prometheus
externalLabels and group_by). The alerts list reads the label from the alert
itself, so it shows the chip with only the external label.

Web UI and docs only: no endpoint or JSON shape changed, so nothing to mirror
in terdut-tui. Nothing changes for a team whose alerts have no cluster label.
This commit is contained in:
Niklas Ye
2026-10-08 17:24:52 +02:00
parent b848143471
commit fb86a18988
7 changed files with 85 additions and 10 deletions
+19
View File
@@ -444,6 +444,25 @@ high-water mark — the highest `severity` label any of its alerts has carried
an incident that hit `critical` still reads as critical after the critical alert
clears.
### Several clusters, one team
A team with one Alertmanager per Kubernetes cluster, each posting to its own
source, needs two settings or the clusters run together.
1. Give every alert a `cluster` label at the source. In Prometheus that is
`externalLabels: {cluster: prod-eu}` (kube-prometheus-stack:
`prometheus.prometheusSpec.externalLabels`).
2. Add `cluster` to `group_by` in `alertmanager.yml`.
The second one is the one that matters. Incidents are matched on the team and
Alertmanager's `groupKey`, and the `groupKey` does not include external labels:
without `cluster` in `group_by`, the same alert in two clusters has the same
key and joins one incident. With it, each cluster gets its own, `cluster` is in
the incident's `group_labels`, and the web UI shows it as a coloured chip on the
queue, the incident and the alert list, instead of leaving it in the title.
An alert that is not grouped by `cluster` still shows the chip on the alert
list, which reads the label from the alert itself.
### An incident opens only on a new occurrence
An incident opens when an alert **transitions into firing**: a fingerprint that