Files

T

Niklas Ye 6e95c86521 Draw boolean alert metrics as an on/off timeline

A metric that only ever reads 0 or 1 (up == 0, anything with bool) gave the
line chart nothing to show: a flat line or a square wave on a y axis padded
to -1..2. When every drawn series is 0/1 the incident page now draws one row
per series instead, filled where the value was 1, a thin mark where it was
0, and the bare track where Prometheus returned no samples, which for these
expressions means the condition did not hold.

The decision is per chart, so one chart never mixes styles; any other value
keeps the line chart as before. Frontend only: the endpoint already returns
the points and step_seconds, which pick() now passes on so the last sample
gets a width and a gap in the samples shows as a gap.

Claude-Session: https://claude.ai/code/session_016mBLURvJoMuUEr9cB2RpUN

2026-10-25 19:00:24 +01:00

4.0 KiB

Raw Permalink Blame History

Metrics on incidents

Alertmanager sends each alert with a generatorURL: the link to the rule's graph in the Prometheus that fired it. With the feature on, terdut-server parses that link, queries the same Prometheus for the rule's expression, and shows a graph and the current value under each alert on the incident page. The browser never talks to Prometheus; the server does.

Turning it on

Set TERDUT_PROMETHEUS_ALLOWED_HOSTS (Helm: prometheus.allowedHosts). The URL arrives in a webhook payload, so whoever holds an integration key chooses it; without an allowlist the server would fetch whatever it was told to. Empty, the default, means off.

Entry	Matches
`prometheus.monitoring.svc`	that host, any port
`prometheus.monitoring.svc:9090`	that host and port
`*.example.com`	`a.example.com`; not `example.com`, not `a.b.example.com`
`prom-*.monitoring.svc`	`prom-0.monitoring.svc`
`**.example.com`	`a.example.com` and `a.b.example.com`
`10.0.0.0/8`, `[fd00::]/8:9090`	addresses in the range

* stands for one DNS label (or part of one); ** is allowed only as the first label and means one or more. Matching ignores case. A bare * or **, or any entry whose last label is only a wildcard, is refused at startup.

The check is made on the address the connection is about to use, not just on the name, so a name that resolves somewhere else (DNS rebinding) is caught. A host admitted by name is still refused if it resolves to a loopback, link-local (which includes the cloud metadata address), unspecified or multicast address; to allow one deliberately, list it as an IP or CIDR. Redirects and proxy environment variables are ignored.

What is captured

Snapshots. A background worker stores each alert's series when the incident opens (opened) and when it closes (closed), in Postgres, so the graph outlives Prometheus' own retention. Incidents closed more than 24 hours before the feature was switched on are not backfilled. A failed capture is retried a few times; a host that is not allowed, or a link that is not a Prometheus graph link, is recorded as failed once.
Live data. While an incident is open, GET /api/incidents/{id}/metrics also queries Prometheus (answers are reused for 15 seconds) and returns the current value.
Window. From TERDUT_PROMETHEUS_PREROLL (default 1h) before the alert started until now, or until the incident was resolved.

Reading the graph

The expression is the alert rule's own, threshold included (latency > 1), so Prometheus only returns data while the condition is true. The graph therefore shows the firing period, and "no data" for the current value means the condition no longer holds, not that Prometheus is down. A rule expression matches every series, not just the one that fired; only series that agree with the alert's labels are kept, at most five per alert.

When every series only ever reads 0 or 1 (a boolean such as up == 0), a line is not informative, so the UI draws an on/off timeline instead: one row per series, filled where the value was 1, a thin mark where it was 0, and the bare track where there were no samples.

Endpoint

GET /api/incidents/{id}/metrics — same team scoping as the other incident routes.

{
  "enabled": true,
  "alerts": [{
    "alert_id": 12, "alert_name": "HighLatency", "expr": "latency>1",
    "available": true,
    "snapshots": { "opened": { "window_start": 1, "window_end": 2, "step_seconds": 15,
                               "series": [{"labels": {"instance": "web-1"}, "points": [[100, 1.5]]}],
                               "captured_at": 3 } },
    "live": { "window_start": 1, "window_end": 2, "step_seconds": 15,
              "series": [], "current": [] }
  }]
}

enabled is false when no host is allowed. available is false, with a reason, when this alert has no Prometheus link or its host is not allowed. A snapshot that failed carries error. live is absent once the incident is resolved. Points are [unix seconds, value].

4.0 KiB Raw Permalink Blame History