Graph the metric behind each alert on the incident page

Each alert carries a generatorURL pointing at the Prometheus rule that fired
it. The server now parses that link, queries the same Prometheus for the
rule's expression and draws it under the alert, with the current value
beside it. The browser never talks to Prometheus, so the CSP stays at
connect-src 'self'.

The URL comes out of a webhook payload, so whoever holds an integration key
chooses it. Querying it is therefore opt-in and bounded by an allowlist,
TERDUT_PROMETHEUS_ALLOWED_HOSTS (names, "*" as one label, a leading "**" as
several, IPs, CIDRs). The check runs on the address the dialer is about to
use, a name that resolves to loopback or link-local is refused even when it
matches, and redirects and proxy variables are ignored.

A worker takes a snapshot when an incident opens and another when it
closes, so the graph outlives Prometheus' retention. It finds its work in
the database rather than being triggered from the webhook, so ingestion is
untouched and a restart loses nothing. While an incident is open the
metrics endpoint also queries live, cached for 15 seconds.

The rule expression keeps its threshold, so Prometheus returns data only
while the alert condition holds; "no data" for the current value means the
condition cleared. Stripping the threshold needs a PromQL parser and is left
out.

Claude-Session: https://claude.ai/code/session_016mBLURvJoMuUEr9cB2RpUN

docs/api.md

@@ -236,6 +236,7 @@ the team gets the same `404` as anybody else.
 | `GET` | `/api/incidents/{id}` | Get single incident, with its alerts inline |
 | `GET` | `/api/incidents/{id}/alerts` | Alerts under this incident |
 | `GET` | `/api/incidents/{id}/timeline` | Full event history, chronological |
+| `GET` | `/api/incidents/{id}/metrics` | Metrics behind the incident's alerts, from their Prometheus — see [Metrics](./metrics.md) |
 | `POST` | `/api/incidents/{id}/acknowledge` | Acknowledge (stamps authed user + time) |
 | `DELETE` | `/api/incidents/{id}/acknowledge` | Clear acknowledgement, back to `triggered` |
 | `POST` | `/api/incidents/{id}/resolve` | Close by hand — **terminal**, see above |

docs/configuration.md

@@ -30,6 +30,9 @@ over an administrator's edit.
 | `TERDUT_TRUSTED_PROXIES` | `1` | How many reverse proxies in front of the server append to `X-Forwarded-For`; the per-address rate limits use the entry that many hops from the right. `0` ignores the header |
 | `TERDUT_OPERATOR_KEY` | — | At least 32 characters. When set, the instance-scoped service account `terdut-operator` is created if missing and its `seed` key replaced with this value at every start — how terdut-operator authenticates without a bootstrap handshake. An instance-scoped account acts as owner of every team (team configuration) but is not a member of any, so it reads no incidents |
 | `TERDUT_OPERATOR_MODE` | `false` | Declares this install gitops-managed: a session's or a user's own API key's writes to teams, escalation policies, dead man's switches and integrations are refused (`403 reason:"operator_managed"`); a [service account](./api.md#service-accounts)'s are not. Team membership and the schedule stay editable regardless |
+| `TERDUT_PROMETHEUS_ALLOWED_HOSTS` | — | Comma-separated Prometheus servers the [metrics on an incident](./metrics.md) may be fetched from, matched against the host in each alert's `generatorURL`. Host names (`prometheus.monitoring.svc`), wildcards (`*` is one DNS label, so `*.example.com` or `prom-*.monitoring.svc`; a leading `**` is one or more labels), IPs or CIDR ranges, each with an optional `:port`. Empty turns the feature off. A bare `*` is refused at startup |
+| `TERDUT_PROMETHEUS_PREROLL` | `1h` | How far before an alert started its metrics graph begins |
+| `TERDUT_PROMETHEUS_TIMEOUT` | `10s` | Per-request timeout for a Prometheus query |
 | `TERDUT_OIDC_ISSUER` | — | Turns single sign-on on. The provider's issuer URL; discovery is read from `<issuer>/.well-known/openid-configuration`. See [Single sign-on](./single-sign-on.md#single-sign-on-oidc) |
 | `TERDUT_OIDC_CLIENT_ID` / `TERDUT_OIDC_CLIENT_SECRET` | — | **Required with an issuer.** The confidential client registered at the provider. Keep the secret in a Secret, not in values |
 | `TERDUT_OIDC_NAME` | `SSO` | What the sign-in button calls the provider |

docs/metrics.md

@@ -0,0 +1,75 @@
+# Metrics on incidents
+
+Alertmanager sends each alert with a `generatorURL`: the link to the rule's graph in
+the Prometheus that fired it. With the feature on, terdut-server parses that link,
+queries the same Prometheus for the rule's expression, and shows a graph and the
+current value under each alert on the incident page. The browser never talks to
+Prometheus; the server does.
+
+## Turning it on
+
+Set `TERDUT_PROMETHEUS_ALLOWED_HOSTS` (Helm: `prometheus.allowedHosts`). The URL arrives in a
+webhook payload, so whoever holds an integration key chooses it; without an allowlist the
+server would fetch whatever it was told to. Empty, the default, means off.
+
+| Entry | Matches |
+|---|---|
+| `prometheus.monitoring.svc` | that host, any port |
+| `prometheus.monitoring.svc:9090` | that host and port |
+| `*.example.com` | `a.example.com`; not `example.com`, not `a.b.example.com` |
+| `prom-*.monitoring.svc` | `prom-0.monitoring.svc` |
+| `**.example.com` | `a.example.com` and `a.b.example.com` |
+| `10.0.0.0/8`, `[fd00::]/8:9090` | addresses in the range |
+
+`*` stands for one DNS label (or part of one); `**` is allowed only as the first label and means
+one or more. Matching ignores case. A bare `*` or `**`, or any entry whose last label is only a
+wildcard, is refused at startup.
+
+The check is made on the address the connection is about to use, not just on the name, so a
+name that resolves somewhere else (DNS rebinding) is caught. A host admitted by name is still
+refused if it resolves to a loopback, link-local (which includes the cloud metadata address),
+unspecified or multicast address; to allow one deliberately, list it as an IP or CIDR. Redirects
+and proxy environment variables are ignored.
+
+## What is captured
+
+- **Snapshots.** A background worker stores each alert's series when the incident opens
+  (`opened`) and when it closes (`closed`), in Postgres, so the graph outlives Prometheus'
+  own retention. Incidents closed more than 24 hours before the feature was switched on are not
+  backfilled. A failed capture is retried a few times; a host that is not allowed, or a link that is
+  not a Prometheus graph link, is recorded as failed once.
+- **Live data.** While an incident is open, `GET /api/incidents/{id}/metrics` also queries
+  Prometheus (answers are reused for 15 seconds) and returns the current value.
+- **Window.** From `TERDUT_PROMETHEUS_PREROLL` (default 1h) before the alert started until now,
+  or until the incident was resolved.
+
+## Reading the graph
+
+The expression is the alert rule's own, threshold included (`latency > 1`), so Prometheus only
+returns data while the condition is true. The graph therefore shows the firing period, and **"no
+data" for the current value means the condition no longer holds**, not that Prometheus is down.
+A rule expression matches every series, not just the one that fired; only series that agree with the
+alert's labels are kept, at most five per alert.
+
+## Endpoint
+
+`GET /api/incidents/{id}/metrics` — same team scoping as the other incident routes.
+
+```json
+{
+  "enabled": true,
+  "alerts": [{
+    "alert_id": 12, "alert_name": "HighLatency", "expr": "latency>1",
+    "available": true,
+    "snapshots": { "opened": { "window_start": 1, "window_end": 2, "step_seconds": 15,
+                               "series": [{"labels": {"instance": "web-1"}, "points": [[100, 1.5]]}],
+                               "captured_at": 3 } },
+    "live": { "window_start": 1, "window_end": 2, "step_seconds": 15,
+              "series": [], "current": [] }
+  }]
+}
+```
+
+`enabled` is `false` when no host is allowed. `available` is `false`, with a `reason`, when this
+alert has no Prometheus link or its host is not allowed. A snapshot that failed carries `error`.
+`live` is absent once the incident is resolved. Points are `[unix seconds, value]`.