# Metrics on incidents

Alertmanager sends each alert with a `generatorURL`: the link to the rule's graph in
the Prometheus that fired it. With the feature on, terdut-server parses that link,
queries the same Prometheus for the rule's expression, and shows a graph and the
current value under each alert on the incident page. The browser never talks to
Prometheus; the server does.

## Turning it on

Set `TERDUT_PROMETHEUS_ALLOWED_HOSTS` (Helm: `prometheus.allowedHosts`). The URL arrives in a
webhook payload, so whoever holds an integration key chooses it; without an allowlist the
server would fetch whatever it was told to. Empty, the default, means off.

| Entry | Matches |
|---|---|
| `prometheus.monitoring.svc` | that host, any port |
| `prometheus.monitoring.svc:9090` | that host and port |
| `*.example.com` | `a.example.com`; not `example.com`, not `a.b.example.com` |
| `prom-*.monitoring.svc` | `prom-0.monitoring.svc` |
| `**.example.com` | `a.example.com` and `a.b.example.com` |
| `10.0.0.0/8`, `[fd00::]/8:9090` | addresses in the range |

`*` stands for one DNS label (or part of one); `**` is allowed only as the first label and means
one or more. Matching ignores case. A bare `*` or `**`, or any entry whose last label is only a
wildcard, is refused at startup.

The check is made on the address the connection is about to use, not just on the name, so a
name that resolves somewhere else (DNS rebinding) is caught. A host admitted by name is still
refused if it resolves to a loopback, link-local (which includes the cloud metadata address),
unspecified or multicast address; to allow one deliberately, list it as an IP or CIDR. Redirects
and proxy environment variables are ignored.

## What is captured

- **Snapshots.** A background worker stores each alert's series when the incident opens
  (`opened`) and when it closes (`closed`), in Postgres, so the graph outlives Prometheus'
  own retention. Incidents closed more than 24 hours before the feature was switched on are not
  backfilled. A failed capture is retried a few times; a host that is not allowed, or a link that is
  not a Prometheus graph link, is recorded as failed once.
- **Live data.** While an incident is open, `GET /api/incidents/{id}/metrics` also queries
  Prometheus (answers are reused for 15 seconds) and returns the current value.
- **Window.** From `TERDUT_PROMETHEUS_PREROLL` (default 1h) before the alert started until now,
  or until the incident was resolved.

## Reading the graph

The expression is the alert rule's own, threshold included (`latency > 1`), so Prometheus only
returns data while the condition is true. The graph therefore shows the firing period, and **"no
data" for the current value means the condition no longer holds**, not that Prometheus is down.
A rule expression matches every series, not just the one that fired; only series that agree with the
alert's labels are kept, at most five per alert.

## Endpoint

`GET /api/incidents/{id}/metrics` — same team scoping as the other incident routes.

```json
{
  "enabled": true,
  "alerts": [{
    "alert_id": 12, "alert_name": "HighLatency", "expr": "latency>1",
    "available": true,
    "snapshots": { "opened": { "window_start": 1, "window_end": 2, "step_seconds": 15,
                               "series": [{"labels": {"instance": "web-1"}, "points": [[100, 1.5]]}],
                               "captured_at": 3 } },
    "live": { "window_start": 1, "window_end": 2, "step_seconds": 15,
              "series": [], "current": [] }
  }]
}
```

`enabled` is `false` when no host is allowed. `available` is `false`, with a `reason`, when this
alert has no Prometheus link or its host is not allowed. A snapshot that failed carries `error`.
`live` is absent once the incident is resolved. Points are `[unix seconds, value]`.