Niklas Ye
f344bb707b
Each alert carries a generatorURL pointing at the Prometheus rule that fired it. The server now parses that link, queries the same Prometheus for the rule's expression and draws it under the alert, with the current value beside it. The browser never talks to Prometheus, so the CSP stays at connect-src 'self'. The URL comes out of a webhook payload, so whoever holds an integration key chooses it. Querying it is therefore opt-in and bounded by an allowlist, TERDUT_PROMETHEUS_ALLOWED_HOSTS (names, "*" as one label, a leading "**" as several, IPs, CIDRs). The check runs on the address the dialer is about to use, a name that resolves to loopback or link-local is refused even when it matches, and redirects and proxy variables are ignored. A worker takes a snapshot when an incident opens and another when it closes, so the graph outlives Prometheus' retention. It finds its work in the database rather than being triggered from the webhook, so ingestion is untouched and a restart loses nothing. While an incident is open the metrics endpoint also queries live, cached for 15 seconds. The rule expression keeps its threshold, so Prometheus returns data only while the alert condition holds; "no data" for the current value means the condition cleared. Stripping the threshold needs a PromQL parser and is left out. Claude-Session: https://claude.ai/code/session_016mBLURvJoMuUEr9cB2RpUN
3.7 KiB
Metrics on incidents
Alertmanager sends each alert with a generatorURL: the link to the rule's graph in
the Prometheus that fired it. With the feature on, terdut-server parses that link,
queries the same Prometheus for the rule's expression, and shows a graph and the
current value under each alert on the incident page. The browser never talks to
Prometheus; the server does.
Turning it on
Set TERDUT_PROMETHEUS_ALLOWED_HOSTS (Helm: prometheus.allowedHosts). The URL arrives in a
webhook payload, so whoever holds an integration key chooses it; without an allowlist the
server would fetch whatever it was told to. Empty, the default, means off.
| Entry | Matches |
|---|---|
prometheus.monitoring.svc |
that host, any port |
prometheus.monitoring.svc:9090 |
that host and port |
*.example.com |
a.example.com; not example.com, not a.b.example.com |
prom-*.monitoring.svc |
prom-0.monitoring.svc |
**.example.com |
a.example.com and a.b.example.com |
10.0.0.0/8, [fd00::]/8:9090 |
addresses in the range |
* stands for one DNS label (or part of one); ** is allowed only as the first label and means
one or more. Matching ignores case. A bare * or **, or any entry whose last label is only a
wildcard, is refused at startup.
The check is made on the address the connection is about to use, not just on the name, so a name that resolves somewhere else (DNS rebinding) is caught. A host admitted by name is still refused if it resolves to a loopback, link-local (which includes the cloud metadata address), unspecified or multicast address; to allow one deliberately, list it as an IP or CIDR. Redirects and proxy environment variables are ignored.
What is captured
- Snapshots. A background worker stores each alert's series when the incident opens
(
opened) and when it closes (closed), in Postgres, so the graph outlives Prometheus' own retention. Incidents closed more than 24 hours before the feature was switched on are not backfilled. A failed capture is retried a few times; a host that is not allowed, or a link that is not a Prometheus graph link, is recorded as failed once. - Live data. While an incident is open,
GET /api/incidents/{id}/metricsalso queries Prometheus (answers are reused for 15 seconds) and returns the current value. - Window. From
TERDUT_PROMETHEUS_PREROLL(default 1h) before the alert started until now, or until the incident was resolved.
Reading the graph
The expression is the alert rule's own, threshold included (latency > 1), so Prometheus only
returns data while the condition is true. The graph therefore shows the firing period, and "no
data" for the current value means the condition no longer holds, not that Prometheus is down.
A rule expression matches every series, not just the one that fired; only series that agree with the
alert's labels are kept, at most five per alert.
Endpoint
GET /api/incidents/{id}/metrics — same team scoping as the other incident routes.
{
"enabled": true,
"alerts": [{
"alert_id": 12, "alert_name": "HighLatency", "expr": "latency>1",
"available": true,
"snapshots": { "opened": { "window_start": 1, "window_end": 2, "step_seconds": 15,
"series": [{"labels": {"instance": "web-1"}, "points": [[100, 1.5]]}],
"captured_at": 3 } },
"live": { "window_start": 1, "window_end": 2, "step_seconds": 15,
"series": [], "current": [] }
}]
}
enabled is false when no host is allowed. available is false, with a reason, when this
alert has no Prometheus link or its host is not allowed. A snapshot that failed carries error.
live is absent once the incident is resolved. Points are [unix seconds, value].