Each alert carries a generatorURL pointing at the Prometheus rule that fired
it. The server now parses that link, queries the same Prometheus for the
rule's expression and draws it under the alert, with the current value
beside it. The browser never talks to Prometheus, so the CSP stays at
connect-src 'self'.
The URL comes out of a webhook payload, so whoever holds an integration key
chooses it. Querying it is therefore opt-in and bounded by an allowlist,
TERDUT_PROMETHEUS_ALLOWED_HOSTS (names, "*" as one label, a leading "**" as
several, IPs, CIDRs). The check runs on the address the dialer is about to
use, a name that resolves to loopback or link-local is refused even when it
matches, and redirects and proxy variables are ignored.
A worker takes a snapshot when an incident opens and another when it
closes, so the graph outlives Prometheus' retention. It finds its work in
the database rather than being triggered from the webhook, so ingestion is
untouched and a restart loses nothing. While an incident is open the
metrics endpoint also queries live, cached for 15 seconds.
The rule expression keeps its threshold, so Prometheus returns data only
while the alert condition holds; "no data" for the current value means the
condition cleared. Stripping the threshold needs a PromQL parser and is left
out.
Claude-Session: https://claude.ai/code/session_016mBLURvJoMuUEr9cB2RpUN
The stats page could only say how many alerts fired. It now answers when
one alert fires, how often, and how fast it is picked up and fixed: each
row of the top-alerts list opens /stats/alert/{name}, with a timeline,
by-hour and by-weekday charts, time-to-resolve per occurrence and a table
of every occurrence linking to its incident. The overview gains a
weekday-by-hour heatmap and a few derived observations.
An occurrence is an incident, not an alert row. Alert rows are reused per
fingerprint and mutated in place, so they carry no history; incidents are
the only record of each time something fired and was answered. That means
/api/stats/alerts/top now counts incidents opened (filtered on
triggered_at) so the list and the drilldown agree. The response keeps name
and count and only adds fields, so existing clients keep working, but the
counts can differ from before.
No migration: the lookups use incidents_signature_idx. Label breakdown,
team and severity filters and period-over-period deltas from the design
draft are left for later.
Claude-Session: https://claude.ai/code/session_016mBLURvJoMuUEr9cB2RpUN
The server exits when the database does not answer within about ten
seconds, so while the database is down the pod crash-loops with a restart
backoff that grows to five minutes, and /healthz only answers after it has
connected. The hook gave up after a fixed two minutes per attempt, and Flux's
default five-minute timeout came first anyway. A failed hook fails the
release and nothing runs it again until the next upgrade, so a database that
was merely slow left the install without an administrator.
Each attempt now waits bootstrap.waitSeconds (default 600), logging progress,
and the Job gets an activeDeadlineSeconds derived from it so a database that
never comes up still ends the hook. No change to the binary, API or schema.
Claude-Session: https://claude.ai/code/session_016mBLURvJoMuUEr9cB2RpUN
Users get a notification_channels table (replacing users.ntfy_topic); pages fan
out to every enabled channel through a notify.Provider registry, with ntfy as
the first provider. Fallbacks are typed {type, address}. Schema edited in place:
existing databases must be recreated.
Claude-Session: https://claude.ai/code/session_016mBLURvJoMuUEr9cB2RpUN
The bootstrap hook created `admin` with no password, so the account could
only use its API key and could not sign in on the web UI or the TUI. It
also won the one-shot /api/bootstrap against whoever ran it by hand, and
failed with exit 1 when the Secret already existed, which fails the Helm
release.
The hook now generates a 32-character password, stores username, password
and api-key in the <release>-admin-key Secret, and reuses the stored
password on later runs, so recreating the database brings the same
account back. The password is written before the account is created, so a
crash in between cannot leave an administrator nobody has the password
for. When the server was bootstrapped by something else, it checks the
stored password against /api/login and removes only the password it
generated if that does not sign in, then exits cleanly. bootstrap.enabled:
false still removes the hook and its role.
Claude-Session: https://claude.ai/code/session_016mBLURvJoMuUEr9cB2RpUN
The README was 1,240 lines of reference material and still described a
SQLite quick start. It is now a short tour (highlights, screenshots of the
web UI, an accurate quick start against Postgres), and each topic has its
own page under docs/ with an index: deployment, configuration, Alertmanager,
incidents, notifications, escalation, dead man's switches, single sign-on,
web UI, API and development. SERVICE-ACCOUNTS.md is rewritten from a
proposal into a reference, and TEAM-LOOKUP.md is gone with the endpoint it
described. The "Upgrading to ..." sections for an unreleased product are
dropped.
Claude-Session: https://claude.ai/code/session_016mBLURvJoMuUEr9cB2RpUN
Niklas Ye