Niklas Ye
6e95c86521
A metric that only ever reads 0 or 1 (up == 0, anything with bool) gave the line chart nothing to show: a flat line or a square wave on a y axis padded to -1..2. When every drawn series is 0/1 the incident page now draws one row per series instead, filled where the value was 1, a thin mark where it was 0, and the bare track where Prometheus returned no samples, which for these expressions means the condition did not hold. The decision is per chart, so one chart never mixes styles; any other value keeps the line chart as before. Frontend only: the endpoint already returns the points and step_seconds, which pick() now passes on so the last sample gets a width and a gap in the samples shows as a gap. Claude-Session: https://claude.ai/code/session_016mBLURvJoMuUEr9cB2RpUN
80 lines
4.0 KiB
Markdown
80 lines
4.0 KiB
Markdown
# Metrics on incidents
|
|
|
|
Alertmanager sends each alert with a `generatorURL`: the link to the rule's graph in
|
|
the Prometheus that fired it. With the feature on, terdut-server parses that link,
|
|
queries the same Prometheus for the rule's expression, and shows a graph and the
|
|
current value under each alert on the incident page. The browser never talks to
|
|
Prometheus; the server does.
|
|
|
|
## Turning it on
|
|
|
|
Set `TERDUT_PROMETHEUS_ALLOWED_HOSTS` (Helm: `prometheus.allowedHosts`). The URL arrives in a
|
|
webhook payload, so whoever holds an integration key chooses it; without an allowlist the
|
|
server would fetch whatever it was told to. Empty, the default, means off.
|
|
|
|
| Entry | Matches |
|
|
|---|---|
|
|
| `prometheus.monitoring.svc` | that host, any port |
|
|
| `prometheus.monitoring.svc:9090` | that host and port |
|
|
| `*.example.com` | `a.example.com`; not `example.com`, not `a.b.example.com` |
|
|
| `prom-*.monitoring.svc` | `prom-0.monitoring.svc` |
|
|
| `**.example.com` | `a.example.com` and `a.b.example.com` |
|
|
| `10.0.0.0/8`, `[fd00::]/8:9090` | addresses in the range |
|
|
|
|
`*` stands for one DNS label (or part of one); `**` is allowed only as the first label and means
|
|
one or more. Matching ignores case. A bare `*` or `**`, or any entry whose last label is only a
|
|
wildcard, is refused at startup.
|
|
|
|
The check is made on the address the connection is about to use, not just on the name, so a
|
|
name that resolves somewhere else (DNS rebinding) is caught. A host admitted by name is still
|
|
refused if it resolves to a loopback, link-local (which includes the cloud metadata address),
|
|
unspecified or multicast address; to allow one deliberately, list it as an IP or CIDR. Redirects
|
|
and proxy environment variables are ignored.
|
|
|
|
## What is captured
|
|
|
|
- **Snapshots.** A background worker stores each alert's series when the incident opens
|
|
(`opened`) and when it closes (`closed`), in Postgres, so the graph outlives Prometheus'
|
|
own retention. Incidents closed more than 24 hours before the feature was switched on are not
|
|
backfilled. A failed capture is retried a few times; a host that is not allowed, or a link that is
|
|
not a Prometheus graph link, is recorded as failed once.
|
|
- **Live data.** While an incident is open, `GET /api/incidents/{id}/metrics` also queries
|
|
Prometheus (answers are reused for 15 seconds) and returns the current value.
|
|
- **Window.** From `TERDUT_PROMETHEUS_PREROLL` (default 1h) before the alert started until now,
|
|
or until the incident was resolved.
|
|
|
|
## Reading the graph
|
|
|
|
The expression is the alert rule's own, threshold included (`latency > 1`), so Prometheus only
|
|
returns data while the condition is true. The graph therefore shows the firing period, and **"no
|
|
data" for the current value means the condition no longer holds**, not that Prometheus is down.
|
|
A rule expression matches every series, not just the one that fired; only series that agree with the
|
|
alert's labels are kept, at most five per alert.
|
|
|
|
When every series only ever reads `0` or `1` (a boolean such as `up == 0`), a line is not
|
|
informative, so the UI draws an on/off timeline instead: one row per series, filled where the value
|
|
was `1`, a thin mark where it was `0`, and the bare track where there were no samples.
|
|
|
|
## Endpoint
|
|
|
|
`GET /api/incidents/{id}/metrics` — same team scoping as the other incident routes.
|
|
|
|
```json
|
|
{
|
|
"enabled": true,
|
|
"alerts": [{
|
|
"alert_id": 12, "alert_name": "HighLatency", "expr": "latency>1",
|
|
"available": true,
|
|
"snapshots": { "opened": { "window_start": 1, "window_end": 2, "step_seconds": 15,
|
|
"series": [{"labels": {"instance": "web-1"}, "points": [[100, 1.5]]}],
|
|
"captured_at": 3 } },
|
|
"live": { "window_start": 1, "window_end": 2, "step_seconds": 15,
|
|
"series": [], "current": [] }
|
|
}]
|
|
}
|
|
```
|
|
|
|
`enabled` is `false` when no host is allowed. `available` is `false`, with a `reason`, when this
|
|
alert has no Prometheus link or its host is not allowed. A snapshot that failed carries `error`.
|
|
`live` is absent once the incident is resolved. Points are `[unix seconds, value]`.
|