Show which cluster an incident came from, as a chip
A team with one Alertmanager per Kubernetes cluster could not tell at a glance where an incident started: the cluster was only a word inside the title. When an alert carries a `cluster` label, the queue rows, the incident page and the alerts list now show it as a chip in that cluster's colour, a stable pick from the existing six-colour palette. A queue row drops `cluster=...` from its title, since the chip says it, and the incident page keeps the full title. An incident has the label only when it is in Alertmanager's group_by, which is also what keeps two clusters' identical alerts apart: incidents are matched on the team and the groupKey, and the groupKey does not include external labels. The README has a section on the two settings (Prometheus externalLabels and group_by). The alerts list reads the label from the alert itself, so it shows the chip with only the external label. Web UI and docs only: no endpoint or JSON shape changed, so nothing to mirror in terdut-tui. Nothing changes for a team whose alerts have no cluster label.
This commit is contained in:
@@ -444,6 +444,25 @@ high-water mark — the highest `severity` label any of its alerts has carried
|
||||
an incident that hit `critical` still reads as critical after the critical alert
|
||||
clears.
|
||||
|
||||
### Several clusters, one team
|
||||
|
||||
A team with one Alertmanager per Kubernetes cluster, each posting to its own
|
||||
source, needs two settings or the clusters run together.
|
||||
|
||||
1. Give every alert a `cluster` label at the source. In Prometheus that is
|
||||
`externalLabels: {cluster: prod-eu}` (kube-prometheus-stack:
|
||||
`prometheus.prometheusSpec.externalLabels`).
|
||||
2. Add `cluster` to `group_by` in `alertmanager.yml`.
|
||||
|
||||
The second one is the one that matters. Incidents are matched on the team and
|
||||
Alertmanager's `groupKey`, and the `groupKey` does not include external labels:
|
||||
without `cluster` in `group_by`, the same alert in two clusters has the same
|
||||
key and joins one incident. With it, each cluster gets its own, `cluster` is in
|
||||
the incident's `group_labels`, and the web UI shows it as a coloured chip on the
|
||||
queue, the incident and the alert list, instead of leaving it in the title.
|
||||
An alert that is not grouped by `cluster` still shows the chip on the alert
|
||||
list, which reads the label from the alert itself.
|
||||
|
||||
### An incident opens only on a new occurrence
|
||||
|
||||
An incident opens when an alert **transitions into firing**: a fingerprint that
|
||||
|
||||
Reference in New Issue
Block a user