Compare commits
8 Commits
| Author | SHA1 | Date | |
|---|---|---|---|
| 1cb09525e3 | |||
| c4833067f0 | |||
| 2bc8f336a0 | |||
| a955356821 | |||
| db474ca909 | |||
| a23e88c16d | |||
| def0f68d00 | |||
| fb86a18988 |
@@ -14,7 +14,7 @@ Preconditions and the plan, without side effects:
|
||||
Config is `.release.conf` here plus `make release-vars`. The process itself lives in
|
||||
`~/.claude/skills/release/`; why it is shaped this way is in README.md §Releasing.
|
||||
|
||||
Two things about this repo specifically:
|
||||
Three things about this repo specifically:
|
||||
|
||||
- **The image is scanned after it is published, not before.** `scan-image` runs trivy
|
||||
against the pushed image, because trivy cannot read a locally built one on this runner.
|
||||
@@ -25,6 +25,16 @@ Two things about this repo specifically:
|
||||
so `chart-bump` needs `--image "$IMAGE"` to know which one moves. That sidecar backs up
|
||||
SQLite; the Postgres move (#2) retires it in favour of a `postgresql` CR with a k8up
|
||||
`pg_dump` annotation, after which only the app image's tag is left.
|
||||
- **Two demos pin this image, and `chart-bump` moves neither.** `terdut-demo` in
|
||||
`Ryuvia/charts` is a `TerdutServer` CR that terdut-operator reconciles, and its
|
||||
`values.yaml` `image.tag` is meant to match production's pin (same digest). The kind demo
|
||||
in terdut-operator (`examples/demo/01-server.yaml`) pins a tag too. A release only bumps
|
||||
the `terdut-server` wrapper, so both drift silently: `terdut-demo` sat at v0.37.0 through
|
||||
v0.41.0-v0.43.0 until it was synced on 2026-10-08. After a release, bump `terdut-demo`'s
|
||||
tag to the same `image-digest` and its `Chart.yaml` `version:` (Flux reconciles on
|
||||
ChartVersion), as its own PR, and say in the release report whether you did. Neither
|
||||
demo has anything but the pin to change, but read the version range's migrations first:
|
||||
the demo's Postgres migrates forward at startup.
|
||||
|
||||
## Checks
|
||||
|
||||
|
||||
@@ -444,6 +444,29 @@ high-water mark — the highest `severity` label any of its alerts has carried
|
||||
an incident that hit `critical` still reads as critical after the critical alert
|
||||
clears.
|
||||
|
||||
### Several clusters, one team
|
||||
|
||||
A team with one Alertmanager per Kubernetes cluster, each posting to its own
|
||||
source, needs two settings or the clusters run together.
|
||||
|
||||
1. Give every alert a `cluster` label at the source. In Prometheus that is
|
||||
`externalLabels: {cluster: prod-eu}` (kube-prometheus-stack:
|
||||
`prometheus.prometheusSpec.externalLabels`).
|
||||
2. Add `cluster` to `group_by` in `alertmanager.yml`.
|
||||
|
||||
The second one is the one that matters. Incidents are matched on the team and
|
||||
Alertmanager's `groupKey`, and the `groupKey` does not include external labels:
|
||||
without `cluster` in `group_by`, the same alert in two clusters has the same
|
||||
key and joins one incident. With it, each cluster gets its own, `cluster` is in
|
||||
the incident's `group_labels`, and the web UI shows it as a coloured chip on the
|
||||
queue, the incident and the alert list, instead of leaving it in the title.
|
||||
An alert that is not grouped by `cluster` still shows the chip on the alert
|
||||
list, which reads the label from the alert itself.
|
||||
|
||||
The queue has a cluster dropdown once there are two or more values to choose
|
||||
between. It filters on the incident's `cluster` group label
|
||||
(`GET /api/incidents?cluster=...`), so it only sees incidents grouped by it.
|
||||
|
||||
### An incident opens only on a new occurrence
|
||||
|
||||
An incident opens when an alert **transitions into firing**: a fingerprint that
|
||||
@@ -926,7 +949,8 @@ the team gets the same `404` as anybody else.
|
||||
|
||||
| Method | Path | Description |
|
||||
|---|---|---|
|
||||
| `GET` | `/api/incidents` | List incidents. Filters: `?status=triggered\|acknowledged\|resolved`, `?severity=`, `?assigned_to=<user id>`, `?archived=true`, `?snoozed=true`, `?from=YYYY-MM-DD`, `?to=YYYY-MM-DD`, `?sort=severity`, `?limit=` (default 50, max 500) |
|
||||
| `GET` | `/api/incidents` | List incidents. Filters: `?status=triggered\|acknowledged\|resolved`, `?severity=`, `?assigned_to=<user id>`, `?archived=true`, `?snoozed=true`, `?from=YYYY-MM-DD`, `?to=YYYY-MM-DD`, `?sort=severity`, `?cluster=<value of the cluster group label>`, `?limit=` (default 50, max 500) |
|
||||
| `GET` | `/api/incidents/clusters` | The distinct `cluster` values on the caller's incidents from the last 90 days, sorted (`?team_id=` narrows it). An empty array when nothing carries the label |
|
||||
| `GET` | `/api/incidents/{id}` | Get single incident, with its alerts inline |
|
||||
| `GET` | `/api/incidents/{id}/alerts` | Alerts under this incident |
|
||||
| `GET` | `/api/incidents/{id}/timeline` | Full event history, chronological |
|
||||
|
||||
@@ -15,5 +15,5 @@ type: application
|
||||
# appVersion and image.tag in values.yaml no longer agree, and that is not an oversight:
|
||||
# image.tag stays "latest", which is what a local install actually pulls. appVersion is
|
||||
# metadata and drives nothing.
|
||||
version: 0.41.3
|
||||
appVersion: "v0.41.3"
|
||||
version: 0.43.0
|
||||
appVersion: "v0.43.0"
|
||||
|
||||
@@ -13,6 +13,52 @@ import (
|
||||
"github.com/go-chi/chi/v5"
|
||||
)
|
||||
|
||||
// handleListClusters answers GET /api/incidents/clusters: the distinct values of
|
||||
// the origin label across the caller's incidents from the last 90 days, sorted,
|
||||
// so the queue can offer them as a filter. Optional team_id narrows it to one
|
||||
// team. Empty when nothing carries the label, which is how the UI knows to show
|
||||
// no filter at all.
|
||||
func handleListClusters(db *sql.DB) http.HandlerFunc {
|
||||
return func(w http.ResponseWriter, r *http.Request) {
|
||||
args := &sqlArgs{}
|
||||
where := []string{
|
||||
"team_id = ANY(" + args.add(callerTeamIDs(r.Context())) + ")",
|
||||
"triggered_at >= " + args.add(time.Now().AddDate(0, 0, -90).Unix()),
|
||||
}
|
||||
if team := r.URL.Query().Get("team_id"); team != "" {
|
||||
if n, err := strconv.ParseInt(team, 10, 64); err == nil {
|
||||
where = append(where, "team_id = "+args.add(n))
|
||||
}
|
||||
}
|
||||
label := args.add(originLabel)
|
||||
|
||||
rows, err := db.QueryContext(r.Context(),
|
||||
fmt.Sprintf("SELECT DISTINCT group_labels->>%[1]s AS v FROM incidents WHERE %[2]s AND group_labels->>%[1]s <> '' ORDER BY v LIMIT 200",
|
||||
label, strings.Join(where, " AND ")),
|
||||
args.all()...)
|
||||
if err != nil {
|
||||
respond(w, http.StatusInternalServerError, errResp("internal error"))
|
||||
return
|
||||
}
|
||||
defer rows.Close()
|
||||
|
||||
clusters := []string{}
|
||||
for rows.Next() {
|
||||
var v string
|
||||
if err := rows.Scan(&v); err != nil {
|
||||
respond(w, http.StatusInternalServerError, errResp("internal error"))
|
||||
return
|
||||
}
|
||||
clusters = append(clusters, v)
|
||||
}
|
||||
if err := rows.Err(); err != nil {
|
||||
respond(w, http.StatusInternalServerError, errResp("internal error"))
|
||||
return
|
||||
}
|
||||
respond(w, http.StatusOK, clusters)
|
||||
}
|
||||
}
|
||||
|
||||
func handleListIncidents(db *sql.DB) http.HandlerFunc {
|
||||
return func(w http.ResponseWriter, r *http.Request) {
|
||||
q := r.URL.Query()
|
||||
@@ -53,6 +99,12 @@ func handleListIncidents(db *sql.DB) http.HandlerFunc {
|
||||
if severity := q.Get("severity"); severity != "" {
|
||||
where = append(where, "i.severity = "+args.add(severity))
|
||||
}
|
||||
// Where it came from: the value of the origin label (originLabel, by
|
||||
// convention "cluster") among the incident's group labels. An incident has
|
||||
// it only when the label is in Alertmanager's group_by.
|
||||
if cluster := q.Get("cluster"); cluster != "" {
|
||||
where = append(where, "i.group_labels->>"+args.add(originLabel)+" = "+args.add(cluster))
|
||||
}
|
||||
if assignee := q.Get("assigned_to"); assignee != "" {
|
||||
if n, err := strconv.ParseInt(assignee, 10, 64); err == nil {
|
||||
where = append(where, "i.assigned_to = "+args.add(n))
|
||||
|
||||
@@ -4,6 +4,7 @@ import (
|
||||
"context"
|
||||
"fmt"
|
||||
"net/http"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
@@ -972,3 +973,37 @@ func TestServiceAccount_AssignArchiveUnarchiveRecordActor(t *testing.T) {
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// The queue can be narrowed to one cluster, and the distinct clusters are
|
||||
// offered so the filter has something to list.
|
||||
func TestIncidents_FilterByCluster(t *testing.T) {
|
||||
s := newTS(t)
|
||||
|
||||
for _, c := range []string{"prod-eu", "prod-us"} {
|
||||
fireGroupedAs(t, s, "fp-"+c,
|
||||
map[string]string{"alertname": "PodRestarting", "cluster": c, "namespace": "n"},
|
||||
`{}:{alertname="PodRestarting",cluster="`+c+`",namespace="n"}`)
|
||||
}
|
||||
// A cluster-less incident exists too, and must never match a cluster filter.
|
||||
fireGroupedAs(t, s, "fp-none", map[string]string{"alertname": "DiskFull"}, `{}:{alertname="DiskFull"}`)
|
||||
|
||||
if got := len(listIncidents(t, s, "")); got != 3 {
|
||||
t.Fatalf("expected 3 open incidents, got %d", got)
|
||||
}
|
||||
eu := listIncidents(t, s, "?cluster=prod-eu")
|
||||
if len(eu) != 1 {
|
||||
t.Fatalf("expected 1 incident for prod-eu, got %d", len(eu))
|
||||
}
|
||||
if labels, _ := eu[0]["group_labels"].(map[string]any); labels["cluster"] != "prod-eu" {
|
||||
t.Errorf("filtered to the wrong cluster: %v", eu[0]["group_labels"])
|
||||
}
|
||||
if got := len(listIncidents(t, s, "?cluster=nowhere")); got != 0 {
|
||||
t.Errorf("an unknown cluster should match nothing, got %d", got)
|
||||
}
|
||||
|
||||
var clusters []string
|
||||
decode(t, s.req(t, http.MethodGet, "/api/incidents/clusters", nil), &clusters)
|
||||
if want := "prod-eu,prod-us"; strings.Join(clusters, ",") != want {
|
||||
t.Errorf("clusters = %v, want %s", clusters, want)
|
||||
}
|
||||
}
|
||||
|
||||
@@ -8,6 +8,7 @@ import (
|
||||
"fmt"
|
||||
"log"
|
||||
"net/http"
|
||||
"regexp"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
@@ -409,9 +410,11 @@ func renderNotification(inc models.Incident, n outboxRow, firing int, cfg Notify
|
||||
strings.TrimSuffix(cfg.PublicURL, "/"), inc.ID)
|
||||
}
|
||||
|
||||
title := pageTitle(inc)
|
||||
|
||||
switch n.kind {
|
||||
case notifyResolved:
|
||||
msg.Title = "Resolved: " + inc.Title
|
||||
msg.Title = "Resolved: " + title
|
||||
msg.Message = "All alerts stopped firing after " +
|
||||
humanDuration(time.Since(inc.TriggeredAt))
|
||||
msg.Priority = ntfyPriorityLow
|
||||
@@ -419,9 +422,9 @@ func renderNotification(inc models.Incident, n outboxRow, firing int, cfg Notify
|
||||
return msg
|
||||
|
||||
case notifyReminder:
|
||||
msg.Title = "Still unacknowledged: " + inc.Title
|
||||
msg.Title = "Still unacknowledged: " + title
|
||||
default:
|
||||
msg.Title = inc.Title
|
||||
msg.Title = title
|
||||
}
|
||||
|
||||
severity := derefString(inc.Severity)
|
||||
@@ -443,6 +446,48 @@ func renderNotification(inc models.Incident, n outboxRow, firing int, cfg Notify
|
||||
return msg
|
||||
}
|
||||
|
||||
// originLabel is the label that says where an alert came from, for a team with
|
||||
// several Kubernetes clusters behind it. It comes from Prometheus's
|
||||
// externalLabels and reaches an incident through Alertmanager's group_by; the
|
||||
// web UI reads the same label, and the README ("Several clusters, one team")
|
||||
// explains how to set it up.
|
||||
const originLabel = "cluster"
|
||||
|
||||
// pageTitle is the incident's title for a notification. A phone's lock screen
|
||||
// cuts a long title off at the end, and the incident title puts the grouping
|
||||
// labels there, so the cluster would be the first thing lost. When the incident
|
||||
// has an origin it leads instead, "[prod-eu] PodRestarting (namespace=foo)", and
|
||||
// is dropped from the parenthesis so it is not said twice. A title that is not
|
||||
// in incidentTitle's "name (k=v, k=v)" shape keeps its text and gains the prefix.
|
||||
func pageTitle(inc models.Incident) string {
|
||||
origin := inc.GroupLabels[originLabel]
|
||||
if origin == "" {
|
||||
return inc.Title
|
||||
}
|
||||
return "[" + origin + "] " + titleWithoutLabel(inc.Title, originLabel, origin)
|
||||
}
|
||||
|
||||
var titleShape = regexp.MustCompile(`(?s)^(.*?) \((.*)\)$`)
|
||||
|
||||
// titleWithoutLabel removes "key=value" from the parenthesised tail of a title
|
||||
// built by incidentTitle, and the parentheses with it if nothing else is left.
|
||||
func titleWithoutLabel(title, key, value string) string {
|
||||
m := titleShape.FindStringSubmatch(title)
|
||||
if m == nil {
|
||||
return title
|
||||
}
|
||||
var rest []string
|
||||
for _, part := range strings.Split(m[2], ", ") {
|
||||
if part != key+"="+value {
|
||||
rest = append(rest, part)
|
||||
}
|
||||
}
|
||||
if len(rest) == 0 {
|
||||
return m[1]
|
||||
}
|
||||
return m[1] + " (" + strings.Join(rest, ", ") + ")"
|
||||
}
|
||||
|
||||
// ntfy's priority scale. Max is the one that overrides the phone's quiet
|
||||
// settings, which is the whole point of paging on critical.
|
||||
const (
|
||||
|
||||
@@ -1,6 +1,7 @@
|
||||
package api_test
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"context"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
@@ -164,10 +165,91 @@ func fireCritical(t *testing.T, s *ts) {
|
||||
}, "{}:{alertname=\"DiskFull\"}")
|
||||
}
|
||||
|
||||
// fireGrouped posts one critical alert whose Alertmanager group carries the
|
||||
// given labels, the way group_by puts them on the webhook.
|
||||
func fireGrouped(t *testing.T, s *ts, groupLabels map[string]string, groupKey string) {
|
||||
t.Helper()
|
||||
fireGroupedAs(t, s, "fp-grouped", groupLabels, groupKey)
|
||||
}
|
||||
|
||||
// fireGroupedAs is fireGrouped with its own alert fingerprint, for a test that
|
||||
// needs several alerts open at once.
|
||||
func fireGroupedAs(t *testing.T, s *ts, fingerprint string, groupLabels map[string]string, groupKey string) {
|
||||
t.Helper()
|
||||
payload := map[string]any{
|
||||
"version": "4", "status": "firing", "groupKey": groupKey, "groupLabels": groupLabels,
|
||||
"alerts": []map[string]any{amAlert(fingerprint, "PodRestarting", "firing",
|
||||
"2026-05-20T10:00:00Z", zeroTime, map[string]string{"severity": "critical"})},
|
||||
}
|
||||
data, _ := json.Marshal(payload)
|
||||
resp, err := http.Post(s.URL+"/api/integrations/"+s.ingestKey+"/alertmanager",
|
||||
"application/json", bytes.NewReader(data))
|
||||
if err != nil {
|
||||
t.Fatalf("post webhook: %v", err)
|
||||
}
|
||||
resp.Body.Close()
|
||||
if resp.StatusCode != http.StatusOK {
|
||||
t.Fatalf("webhook returned %d", resp.StatusCode)
|
||||
}
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// Delivery
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
// A phone cuts a long title off at the end, and the incident title keeps the
|
||||
// grouping labels there, so the cluster leads the page instead.
|
||||
func TestNotify_ClusterLeadsTheTitle(t *testing.T) {
|
||||
s, f := notifyTS(t, api.NotifyConfig{PublicURL: "https://terdut.example.com"})
|
||||
|
||||
fireGrouped(t, s, map[string]string{
|
||||
"alertname": "PodRestarting", "cluster": "prod-eu", "namespace": "shop",
|
||||
}, `{}:{alertname="PodRestarting",cluster="prod-eu",namespace="shop"}`)
|
||||
s.sweepNotify(t)
|
||||
|
||||
msgs := f.messages()
|
||||
if len(msgs) != 1 {
|
||||
t.Fatalf("expected 1 push, got %d", len(msgs))
|
||||
}
|
||||
if want := "[prod-eu] PodRestarting (namespace=shop)"; msgs[0].Title != want {
|
||||
t.Errorf("title = %q, want %q", msgs[0].Title, want)
|
||||
}
|
||||
}
|
||||
|
||||
// Only the cluster is the whole grouping: no parenthesis is left behind.
|
||||
func TestNotify_ClusterAloneLeavesNoParenthesis(t *testing.T) {
|
||||
s, f := notifyTS(t, api.NotifyConfig{})
|
||||
|
||||
fireGrouped(t, s, map[string]string{"alertname": "PodRestarting", "cluster": "prod-eu"},
|
||||
`{}:{alertname="PodRestarting",cluster="prod-eu"}`)
|
||||
s.sweepNotify(t)
|
||||
|
||||
msgs := f.messages()
|
||||
if len(msgs) != 1 {
|
||||
t.Fatalf("expected 1 push, got %d", len(msgs))
|
||||
}
|
||||
if want := "[prod-eu] PodRestarting"; msgs[0].Title != want {
|
||||
t.Errorf("title = %q, want %q", msgs[0].Title, want)
|
||||
}
|
||||
}
|
||||
|
||||
// Nothing changes for a team whose alerts have no cluster label.
|
||||
func TestNotify_NoClusterKeepsTheTitle(t *testing.T) {
|
||||
s, f := notifyTS(t, api.NotifyConfig{})
|
||||
|
||||
fireGrouped(t, s, map[string]string{"alertname": "PodRestarting", "namespace": "shop"},
|
||||
`{}:{alertname="PodRestarting",namespace="shop"}`)
|
||||
s.sweepNotify(t)
|
||||
|
||||
msgs := f.messages()
|
||||
if len(msgs) != 1 {
|
||||
t.Fatalf("expected 1 push, got %d", len(msgs))
|
||||
}
|
||||
if want := "PodRestarting (namespace=shop)"; msgs[0].Title != want {
|
||||
t.Errorf("title = %q, want %q", msgs[0].Title, want)
|
||||
}
|
||||
}
|
||||
|
||||
func TestNotify_TriggeredIncidentPagesOnCall(t *testing.T) {
|
||||
s, f := notifyTS(t, api.NotifyConfig{PublicURL: "https://terdut.example.com"})
|
||||
|
||||
|
||||
@@ -145,6 +145,7 @@ func NewRouter(db *sql.DB, notify NotifyConfig, cfg config.Config, version strin
|
||||
r.Get("/api/alerts/{id}", handleGetAlert(db))
|
||||
|
||||
r.Get("/api/incidents", handleListIncidents(db))
|
||||
r.Get("/api/incidents/clusters", handleListClusters(db))
|
||||
r.Get("/api/incidents/{id}", handleGetIncident(db))
|
||||
r.Get("/api/incidents/{id}/alerts", handleIncidentAlerts(db))
|
||||
r.Get("/api/incidents/{id}/timeline", handleIncidentTimeline(db))
|
||||
|
||||
@@ -404,6 +404,16 @@ input:focus, textarea:focus { outline: none; border-color: var(--accent); box-sh
|
||||
mask-image: linear-gradient(90deg, #000 calc(100% - 28px), transparent);
|
||||
}
|
||||
|
||||
/* The queue's cluster filter: a pill-shaped select under the status chips. */
|
||||
.queue-origin { position: relative; display: flex; align-items: center; padding: 0 16px 8px; }
|
||||
.origin-select-icon { position: absolute; left: 28px; width: 16px; height: 16px; color: var(--muted); pointer-events: none; }
|
||||
.origin-select {
|
||||
min-height: 34px; max-width: 100%; padding: 0 12px 0 34px;
|
||||
border: 1px solid var(--border-strong); border-radius: 999px;
|
||||
background: var(--surface); color: var(--text);
|
||||
font: inherit; font-size: 13px; font-weight: 600; cursor: pointer;
|
||||
}
|
||||
|
||||
/* ---------- lists ---------- */
|
||||
|
||||
.list { padding: 0 16px 16px; display: grid; gap: 8px; }
|
||||
@@ -488,7 +498,15 @@ input:focus, textarea:focus { outline: none; border-color: var(--accent); box-sh
|
||||
}
|
||||
.badge::before { content: ""; width: 7px; height: 7px; border-radius: 50%; background: currentColor; }
|
||||
.badge.plain::before { display: none; }
|
||||
.badge .badge-icon { width: 12px; height: 12px; stroke-width: 2.4; }
|
||||
.badge .badge-icon, .origin-chip .badge-icon { width: 12px; height: 12px; stroke-width: 2.4; }
|
||||
/* The cluster an incident or alert came from. Same shape as a badge, coloured
|
||||
from the rcN palette (see originClass in format.js), never the severity one. */
|
||||
.origin-chip {
|
||||
display: inline-flex; align-items: center; gap: 5px; max-width: 100%;
|
||||
padding: 1px 8px; border-radius: 999px;
|
||||
font-size: 12px; font-weight: 700; letter-spacing: 0.01em; white-space: nowrap;
|
||||
overflow: hidden; text-overflow: ellipsis;
|
||||
}
|
||||
.badge.st-triggered, .badge.st-firing { background: var(--crit-soft); color: var(--crit); }
|
||||
.badge.st-acknowledged { background: var(--warn-soft); color: var(--warn); }
|
||||
.badge.st-snoozed { background: var(--snooze-soft); color: var(--snooze); }
|
||||
|
||||
@@ -154,6 +154,9 @@
|
||||
<section id="view-queue" class="view view-queue" data-view="queue">
|
||||
<div class="pane pane-list">
|
||||
<div class="chips" id="queue-filters" role="tablist" aria-label="Filter"></div>
|
||||
<!-- The cluster filter. Hidden until the queue has seen two or more
|
||||
clusters; queue.js fills it in. -->
|
||||
<div class="queue-origin" id="queue-origin" hidden></div>
|
||||
<div id="queue-list" class="list"></div>
|
||||
</div>
|
||||
<div class="pane pane-detail" id="detail" aria-live="polite"></div>
|
||||
|
||||
@@ -2,8 +2,8 @@
|
||||
// incident it belongs to, which is where anything can be done about it.
|
||||
|
||||
import * as api from './api.js';
|
||||
import { h, clear, badge, severityBadge, emptyState, spinner } from './ui.js';
|
||||
import { age, labelSummary } from './format.js';
|
||||
import { h, clear, badge, severityBadge, originChip, emptyState, spinner } from './ui.js';
|
||||
import { age, labelSummary, originOf, ORIGIN_LABEL } from './format.js';
|
||||
|
||||
const FILTERS = [
|
||||
{ id: 'firing', label: 'Firing', query: { status: 'firing' } },
|
||||
@@ -77,7 +77,8 @@ function row(a) {
|
||||
const summary = (a.annotations && a.annotations.summary) || '';
|
||||
const sev = a.labels && a.labels.severity;
|
||||
const labels = labelSummary(Object.fromEntries(
|
||||
Object.entries(a.labels || {}).filter(([k]) => k !== 'severity')));
|
||||
Object.entries(a.labels || {}).filter(([k]) => k !== 'severity' && k !== ORIGIN_LABEL)));
|
||||
const origin = originOf(a.labels);
|
||||
const linked = a.incident_id != null;
|
||||
return h(linked ? 'a' : 'div', {
|
||||
class: `row st-${a.status} ${linked ? '' : 'no-link'}`,
|
||||
@@ -86,6 +87,7 @@ function row(a) {
|
||||
h('div', { class: 'row-title', text: a.name }),
|
||||
h('div', { class: 'row-age', title: a.starts_at, text: age(a.status === 'firing' ? a.starts_at : a.received_at) }),
|
||||
h('div', { class: 'row-meta' },
|
||||
origin && originChip(origin),
|
||||
badge(a.status === 'firing' ? 'Firing' : 'Resolved', `st-${a.status}`),
|
||||
sev && severityBadge(sev),
|
||||
summary && h('span', { text: summary }),
|
||||
|
||||
@@ -83,6 +83,7 @@ export const setNotifyTarget = (id, ntfyTopic) =>
|
||||
|
||||
// incidents
|
||||
export const incidents = (query, opts) => call('GET', '/incidents', { query, ...opts });
|
||||
export const clusters = (query) => call('GET', '/incidents/clusters', { query });
|
||||
export const incident = (id) => call('GET', `/incidents/${id}`);
|
||||
export const timeline = (id) => call('GET', `/incidents/${id}/timeline`);
|
||||
|
||||
|
||||
@@ -117,3 +117,36 @@ export function labelSummary(labels, skip = 'alertname') {
|
||||
export function initial(name) {
|
||||
return (name || '?').trim().charAt(0) || '?';
|
||||
}
|
||||
|
||||
// Where an incident or alert came from, for a team with several Kubernetes
|
||||
// clusters (or other origins) behind it: the value of one label, `cluster` by
|
||||
// convention. It comes from Prometheus's externalLabels, so it is on every
|
||||
// alert; an incident carries it only when it is in Alertmanager's group_by,
|
||||
// which is also what keeps two clusters' identical alerts from merging into one
|
||||
// incident (see the README, "Several clusters, one team").
|
||||
export const ORIGIN_LABEL = 'cluster';
|
||||
|
||||
export function originOf(labels) {
|
||||
const v = labels && labels[ORIGIN_LABEL];
|
||||
return v ? String(v) : null;
|
||||
}
|
||||
|
||||
// A stable colour for an origin, from the six-colour rcN palette app.css has,
|
||||
// so "prod-eu" is the same colour on every row without anything being stored.
|
||||
export function originClass(value) {
|
||||
let n = 0;
|
||||
for (const ch of value) n = (n * 31 + ch.charCodeAt(0)) >>> 0;
|
||||
return `rc${(n % 6) + 1}`;
|
||||
}
|
||||
|
||||
// The title without its `cluster=...` part, for a row that shows the origin as
|
||||
// a chip: the server puts every grouping label in the title, so the cluster
|
||||
// would otherwise appear twice. Anything that does not look like the server's
|
||||
// "name (k=v, k=v)" shape is returned unchanged.
|
||||
export function titleWithoutOrigin(title, value) {
|
||||
if (!value) return title;
|
||||
const m = title.match(/^(.*?) \((.*)\)$/s);
|
||||
if (!m) return title;
|
||||
const rest = m[2].split(', ').filter((p) => p !== `${ORIGIN_LABEL}=${value}`);
|
||||
return rest.length ? `${m[1]} (${rest.join(', ')})` : m[1];
|
||||
}
|
||||
|
||||
@@ -4,10 +4,10 @@
|
||||
import * as api from './api.js';
|
||||
import * as poll from './poll.js';
|
||||
import {
|
||||
h, clear, icon, badge, severityBadge, labelChip, openSheet, closeSheet, confirm, toast, spinner, emptyState,
|
||||
h, clear, icon, badge, severityBadge, originChip, labelChip, openSheet, closeSheet, confirm, toast, spinner, emptyState,
|
||||
} from './ui.js';
|
||||
import {
|
||||
ago, when, until, duration, isFuture, STATUS_LABEL,
|
||||
ago, when, until, duration, isFuture, originOf, STATUS_LABEL,
|
||||
} from './format.js';
|
||||
import { myID, users } from './state.js';
|
||||
import { back } from './app.js';
|
||||
@@ -99,6 +99,8 @@ function render() {
|
||||
|
||||
function statusBadges() {
|
||||
const out = [];
|
||||
const origin = originOf(inc.group_labels);
|
||||
if (origin) out.push(originChip(origin));
|
||||
if (inc.severity) out.push(severityBadge(inc.severity));
|
||||
out.push(badge(STATUS_LABEL[inc.status] || inc.status, `st-${inc.status}`));
|
||||
if (inc.status !== 'resolved' && isFuture(inc.snoozed_until)) {
|
||||
|
||||
@@ -1,8 +1,8 @@
|
||||
// The incident queue: filter chips and a list of incident rows.
|
||||
|
||||
import * as api from './api.js';
|
||||
import { h, clear, badge, severityBadge, emptyState, spinner, fadeOnOverflow } from './ui.js';
|
||||
import { ago, until, isFuture, severityClass, labelSummary } from './format.js';
|
||||
import { h, clear, icon, badge, severityBadge, originChip, emptyState, spinner, fadeOnOverflow } from './ui.js';
|
||||
import { ago, until, isFuture, severityClass, labelSummary, originOf, titleWithoutOrigin } from './format.js';
|
||||
import { state, myID, onTeamChange } from './state.js';
|
||||
import * as onboarding from './onboarding.js';
|
||||
import { navigate } from './app.js';
|
||||
@@ -30,8 +30,10 @@ onboarding.onRerender(() => renderList());
|
||||
// The queue used to keep its own team filter (a per-tab sessionStorage value,
|
||||
// out of step with team.js's own picker); both now defer to the global
|
||||
// selector's shared state, so re-render whenever it changes.
|
||||
onTeamChange(() => {
|
||||
onTeamChange(async () => {
|
||||
renderChips();
|
||||
await loadClusters(true);
|
||||
renderOrigin();
|
||||
refresh({ fresh: true });
|
||||
});
|
||||
|
||||
@@ -52,6 +54,63 @@ function loadFilter() {
|
||||
return 'open';
|
||||
}
|
||||
|
||||
// Which cluster the queue is narrowed to, or null for all of them. Kept in
|
||||
// localStorage like the selected team: it is a standing choice, not a
|
||||
// per-tab one.
|
||||
const ORIGIN_KEY = 'terdut.queue.cluster';
|
||||
let origin = loadOrigin();
|
||||
let clusters = []; // the clusters the server has seen, for the dropdown
|
||||
let clustersAt = 0;
|
||||
|
||||
function loadOrigin() {
|
||||
try {
|
||||
return localStorage.getItem(ORIGIN_KEY) || null;
|
||||
} catch {
|
||||
return null; // storage unavailable
|
||||
}
|
||||
}
|
||||
|
||||
function saveOrigin() {
|
||||
try {
|
||||
if (origin) localStorage.setItem(ORIGIN_KEY, origin);
|
||||
else localStorage.removeItem(ORIGIN_KEY);
|
||||
} catch {
|
||||
/* storage unavailable */
|
||||
}
|
||||
}
|
||||
|
||||
// The clusters to offer, from the server because the list on screen is only
|
||||
// the fifty most recent. Asked again at most every five minutes; a cluster
|
||||
// first met in an incident on screen is added at once (see noteClusters). A
|
||||
// remembered choice the server no longer knows would filter to nothing with no
|
||||
// hint why, so it is dropped.
|
||||
async function loadClusters(force = false) {
|
||||
if (!force && Date.now() - clustersAt < 5 * 60 * 1000) return;
|
||||
try {
|
||||
const query = state.selectedTeamID != null ? { team_id: state.selectedTeamID } : undefined;
|
||||
clusters = await api.clusters(query);
|
||||
clustersAt = Date.now();
|
||||
if (origin && !clusters.includes(origin)) {
|
||||
origin = null;
|
||||
saveOrigin();
|
||||
}
|
||||
} catch {
|
||||
/* the filter is a convenience: without the list it stays hidden */
|
||||
}
|
||||
}
|
||||
|
||||
function noteClusters(list) {
|
||||
let added = false;
|
||||
for (const inc of list || []) {
|
||||
const c = originOf(inc.group_labels);
|
||||
if (c && !clusters.includes(c)) {
|
||||
clusters = [...clusters, c].sort();
|
||||
added = true;
|
||||
}
|
||||
}
|
||||
return added;
|
||||
}
|
||||
|
||||
function saveFilter() {
|
||||
try {
|
||||
sessionStorage.setItem('terdut.queue.filter', filter);
|
||||
@@ -64,6 +123,7 @@ export function show(incidentID) {
|
||||
selected = incidentID;
|
||||
if (!built) {
|
||||
renderChips();
|
||||
renderOrigin();
|
||||
built = true;
|
||||
}
|
||||
renderList();
|
||||
@@ -73,15 +133,22 @@ export async function refresh({ fresh = false } = {}) {
|
||||
const f = FILTERS.find((x) => x.id === filter);
|
||||
const requested = filter;
|
||||
try {
|
||||
// First, so a remembered cluster the server no longer knows is dropped
|
||||
// before it is sent as a filter.
|
||||
await loadClusters();
|
||||
// The open list is already fetched for the badges; no need to ask twice.
|
||||
// The cached open queue covers every team, so it can only be reused when
|
||||
// no team filter is applied.
|
||||
const query = state.selectedTeamID != null ? { ...f.query, team_id: state.selectedTeamID } : f.query;
|
||||
const cached = filter === 'open' && !fresh && state.selectedTeamID == null;
|
||||
const query = { ...f.query };
|
||||
if (state.selectedTeamID != null) query.team_id = state.selectedTeamID;
|
||||
if (origin) query.cluster = origin;
|
||||
const cached = filter === 'open' && !fresh && state.selectedTeamID == null && !origin;
|
||||
const result = cached ? state.open : await api.incidents(query);
|
||||
await onboarding.load();
|
||||
if (requested !== filter) return;
|
||||
items = result;
|
||||
noteClusters(state.open);
|
||||
noteClusters(items);
|
||||
error = null;
|
||||
} catch (err) {
|
||||
if (requested !== filter) return;
|
||||
@@ -91,6 +158,7 @@ export async function refresh({ fresh = false } = {}) {
|
||||
// whichever caller updated it before calling here — app.js's poll, or the
|
||||
// `cached` branch above.
|
||||
renderChips();
|
||||
renderOrigin();
|
||||
renderList();
|
||||
}
|
||||
|
||||
@@ -109,9 +177,9 @@ function setFilter(id) {
|
||||
// for the badges — Snoozed/Resolved/Archived would need a request of their
|
||||
// own, so those chips stay count-less for now.
|
||||
function chipCount(id) {
|
||||
const open = state.selectedTeamID == null
|
||||
? state.open
|
||||
: state.open.filter((i) => i.team_id === state.selectedTeamID);
|
||||
const open = state.open.filter((i) =>
|
||||
(state.selectedTeamID == null || i.team_id === state.selectedTeamID)
|
||||
&& (!origin || originOf(i.group_labels) === origin));
|
||||
if (id === 'open') return open.length;
|
||||
if (id === 'triggered') return open.filter((i) => i.status === 'triggered').length;
|
||||
if (id === 'acknowledged') return open.filter((i) => i.status === 'acknowledged').length;
|
||||
@@ -144,6 +212,30 @@ function renderChips() {
|
||||
chipsFade();
|
||||
}
|
||||
|
||||
// The cluster dropdown, under the status chips. Not shown at all until there
|
||||
// are two clusters to choose between, the same rule the team selector follows.
|
||||
function renderOrigin() {
|
||||
const el = document.getElementById('queue-origin');
|
||||
if (clusters.length < 2) {
|
||||
el.hidden = true;
|
||||
return;
|
||||
}
|
||||
const select = h('select', { class: 'origin-select', 'aria-label': 'Cluster' },
|
||||
h('option', { value: '', text: 'All clusters', selected: !origin }),
|
||||
clusters.map((c) => h('option', { value: c, text: c, selected: c === origin })));
|
||||
select.addEventListener('change', () => {
|
||||
origin = select.value || null;
|
||||
saveOrigin();
|
||||
items = null;
|
||||
cursor = -1;
|
||||
renderChips();
|
||||
renderList();
|
||||
refresh({ fresh: true });
|
||||
});
|
||||
clear(el, icon('server', 'icon origin-select-icon'), select);
|
||||
el.hidden = false;
|
||||
}
|
||||
|
||||
function renderList() {
|
||||
const el = document.getElementById('queue-list');
|
||||
const checklist = onboarding.card();
|
||||
@@ -201,15 +293,18 @@ function row(inc, index) {
|
||||
? h('span', { class: 'row-team', text: inc.team_name })
|
||||
: null;
|
||||
|
||||
const rowOrigin = originOf(inc.group_labels);
|
||||
|
||||
return h('a', {
|
||||
class: `row ${severityClass(inc.severity)} ${resolved ? 'resolved' : ''} ${index === cursor ? 'kbd-focus' : ''}`,
|
||||
href: `/incidents/${inc.id}`,
|
||||
'aria-current': inc.id === selected ? 'true' : null,
|
||||
dataset: { index: String(index) },
|
||||
},
|
||||
h('div', { class: 'row-title', text: inc.title }),
|
||||
h('div', { class: 'row-title', text: titleWithoutOrigin(inc.title, rowOrigin) }),
|
||||
h('div', { class: 'row-age', title: inc.triggered_at, text: `Triggered ${ago(inc.triggered_at)}` }),
|
||||
h('div', { class: 'row-meta' },
|
||||
rowOrigin && !origin && originChip(rowOrigin),
|
||||
!impliedStatus && status,
|
||||
// The left-border colour alone doesn't say what it means; spell it out
|
||||
// too, same badge the incident detail page uses for severity.
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
// DOM helpers, the bottom sheet, confirmation and toasts.
|
||||
|
||||
import { severityClass } from './format.js';
|
||||
import { severityClass, originClass } from './format.js';
|
||||
|
||||
// h builds an element. attrs: class, text, on<event>, dataset, aria/other
|
||||
// attributes; boolean true sets an empty attribute, false/null skips it.
|
||||
@@ -49,6 +49,7 @@ export function fadeOnOverflow(el) {
|
||||
|
||||
// Stroke icons, 24×24. Built as SVG nodes so the CSP needs no inline anything.
|
||||
const ICONS = {
|
||||
server: ['rect:3,4,18,6,2', 'rect:3,14,18,6,2', 'M7 7h.01', 'M7 17h.01'],
|
||||
alertTriangle: ['M12 4l9.5 16.5h-19z', 'M12 10v4.5', 'M12 17.5h.01'],
|
||||
alertCircle: ['circle:12,12,9', 'M12 7.5v5', 'M12 16h.01'],
|
||||
infoCircle: ['circle:12,12,9', 'M12 11v5.5', 'M12 7.5h.01'],
|
||||
@@ -193,6 +194,13 @@ export function badge(text, cls = '') {
|
||||
return h('span', { class: `badge ${cls}`, text });
|
||||
}
|
||||
|
||||
// Where it came from (the cluster): a chip in that origin's colour, with a
|
||||
// server icon so it reads as a place and not as a status.
|
||||
export function originChip(value) {
|
||||
return h('span', { class: `origin-chip ${originClass(value)}`, title: `Origin: ${value}` },
|
||||
icon('server', 'icon badge-icon'), value);
|
||||
}
|
||||
|
||||
// The severity as a badge with a shape as well as a colour: a triangle for
|
||||
// critical, a circle with ! for warning, a circle with i for anything else, so
|
||||
// the coloured bar and badge are not the only thing that says it.
|
||||
|
||||
Reference in New Issue
Block a user