Files

T

Niklas Ye f344bb707b Graph the metric behind each alert on the incident page

Each alert carries a generatorURL pointing at the Prometheus rule that fired
it. The server now parses that link, queries the same Prometheus for the
rule's expression and draws it under the alert, with the current value
beside it. The browser never talks to Prometheus, so the CSP stays at
connect-src 'self'.

The URL comes out of a webhook payload, so whoever holds an integration key
chooses it. Querying it is therefore opt-in and bounded by an allowlist,
TERDUT_PROMETHEUS_ALLOWED_HOSTS (names, "*" as one label, a leading "**" as
several, IPs, CIDRs). The check runs on the address the dialer is about to
use, a name that resolves to loopback or link-local is refused even when it
matches, and redirects and proxy variables are ignored.

A worker takes a snapshot when an incident opens and another when it
closes, so the graph outlives Prometheus' retention. It finds its work in
the database rather than being triggered from the webhook, so ingestion is
untouched and a restart loses nothing. While an incident is open the
metrics endpoint also queries live, cached for 15 seconds.

The rule expression keeps its threshold, so Prometheus returns data only
while the alert condition holds; "no data" for the current value means the
condition cleared. Stripping the threshold needs a PromQL parser and is left
out.

Claude-Session: https://claude.ai/code/session_016mBLURvJoMuUEr9cB2RpUN

2026-10-22 14:10:34 +02:00

3.7 KiB

Raw Blame History

Metrics on incidents

Alertmanager sends each alert with a generatorURL: the link to the rule's graph in the Prometheus that fired it. With the feature on, terdut-server parses that link, queries the same Prometheus for the rule's expression, and shows a graph and the current value under each alert on the incident page. The browser never talks to Prometheus; the server does.

Turning it on

Set TERDUT_PROMETHEUS_ALLOWED_HOSTS (Helm: prometheus.allowedHosts). The URL arrives in a webhook payload, so whoever holds an integration key chooses it; without an allowlist the server would fetch whatever it was told to. Empty, the default, means off.

Entry	Matches
`prometheus.monitoring.svc`	that host, any port
`prometheus.monitoring.svc:9090`	that host and port
`*.example.com`	`a.example.com`; not `example.com`, not `a.b.example.com`
`prom-*.monitoring.svc`	`prom-0.monitoring.svc`
`**.example.com`	`a.example.com` and `a.b.example.com`
`10.0.0.0/8`, `[fd00::]/8:9090`	addresses in the range

* stands for one DNS label (or part of one); ** is allowed only as the first label and means one or more. Matching ignores case. A bare * or **, or any entry whose last label is only a wildcard, is refused at startup.

The check is made on the address the connection is about to use, not just on the name, so a name that resolves somewhere else (DNS rebinding) is caught. A host admitted by name is still refused if it resolves to a loopback, link-local (which includes the cloud metadata address), unspecified or multicast address; to allow one deliberately, list it as an IP or CIDR. Redirects and proxy environment variables are ignored.

What is captured

Snapshots. A background worker stores each alert's series when the incident opens (opened) and when it closes (closed), in Postgres, so the graph outlives Prometheus' own retention. Incidents closed more than 24 hours before the feature was switched on are not backfilled. A failed capture is retried a few times; a host that is not allowed, or a link that is not a Prometheus graph link, is recorded as failed once.
Live data. While an incident is open, GET /api/incidents/{id}/metrics also queries Prometheus (answers are reused for 15 seconds) and returns the current value.
Window. From TERDUT_PROMETHEUS_PREROLL (default 1h) before the alert started until now, or until the incident was resolved.

Reading the graph

The expression is the alert rule's own, threshold included (latency > 1), so Prometheus only returns data while the condition is true. The graph therefore shows the firing period, and "no data" for the current value means the condition no longer holds, not that Prometheus is down. A rule expression matches every series, not just the one that fired; only series that agree with the alert's labels are kept, at most five per alert.

Endpoint

GET /api/incidents/{id}/metrics — same team scoping as the other incident routes.

{
  "enabled": true,
  "alerts": [{
    "alert_id": 12, "alert_name": "HighLatency", "expr": "latency>1",
    "available": true,
    "snapshots": { "opened": { "window_start": 1, "window_end": 2, "step_seconds": 15,
                               "series": [{"labels": {"instance": "web-1"}, "points": [[100, 1.5]]}],
                               "captured_at": 3 } },
    "live": { "window_start": 1, "window_end": 2, "step_seconds": 15,
              "series": [], "current": [] }
  }]
}

enabled is false when no host is allowed. available is false, with a reason, when this alert has no Prometheus link or its host is not allowed. A snapshot that failed carries error. live is absent once the incident is resolved. Points are [unix seconds, value].

3.7 KiB Raw Blame History