Whoever is on call gets paged on a phone, and until now the only ways to
act on a page were the notification's Acknowledge button or a terminal.
Tapping the notification itself opened /api/incidents/{id}, which a
browser can only answer with a 401 in JSON. The server now serves a web
UI at / covering the incident queue, each incident's alerts and timeline
with every action on it, who is on call, the alert feed, and changing
your own password. The notification link now points at /incidents/{id}
in that UI.
It is embedded in the binary and has no build step: plain HTML, CSS and
ES modules under internal/web/static, served with an ETag per file and a
CSP that allows nothing from any other origin. That is how rd-web is
built. It avoids adding a node toolchain to the Dockerfile and the
pipeline for a page this size, and it keeps the page on the same origin
as the API, so no CORS is needed and nothing else has to be deployed.
Paths without a file extension fall back to index.html, so a deep link
survives a reload. An unknown path under /api/ still gets a JSON 404
rather than the page.
Signing in uses a username and password, because pasting a 64-character
API key into a phone at 3am is not a sign-in flow. Users have no
password until one is set through PUT /api/users/{id}/password, or
optionally at bootstrap. A user without a password is exactly where they
were before this commit and can only use API keys. A login sets an
HttpOnly, SameSite=Lax session cookie. It lasts 30 days and slides
forward while in use, so an on-call phone does not sign itself out.
Only the token's hash is stored, as for API keys.
The cookie needs a CSRF guard where a bearer header does not, because
browsers attach cookies to requests other sites make. So cookie-
authenticated requests go through Go 1.25's http.CrossOriginProtection,
and bearer requests do not. A request carrying an Authorization header
is judged on that header alone and never falls back to the cookie.
Changing a password ends every other session of that user. Changing
your own requires the current password, so a phone left signed in
cannot be used to take the account over.
Failed logins are counted per username and per client address. Ten
failures for one username in 15 minutes refuse that username for the
rest of the window, even with the right password. That makes locking
somebody out possible for anyone who knows their username. It was
accepted because the alternative is unlimited guessing, and during a
lockout the notification's Acknowledge button and API keys keep
working. The address limit reads the first X-Forwarded-For hop, since
behind the gateway RemoteAddr is Envoy. It is looser, because a whole
office behind one NAT shares it.
The Secure flag follows TERDUT_PUBLIC_URL, since TLS terminates at the
gateway and the server itself only ever sees plain HTTP. The chart
already defaults that variable to https://<hostname>.
Schedule editing, statistics and user management stay in terdut-tui for
now. The API they use is unchanged, and bearer authentication behaves
exactly as before.
SUM över noll rader är NULL i SQLite, inte 0. handleStatsIncidents läste de
tre statusräknarna rakt in i int64, så i samma stund som filtret inte
matchade någon rad föll skanningen på "converting NULL to int64 is
unsupported" och hela /api/stats/incidents svarade 500. COUNT(*) ger
däremot 0 utan knot, vilket är precis varför felet inte syns förrän
tabellen töms — det är det enda uttrycket i satsen som klarar noll rader.
Filtret är alltid på: statsFilter lägger på archived_at IS NULL (923fc8b,
flyttat hit i 279ef6c). En installation som varit tyst ett tag arkiverar
därmed sig själv in i felet. Det är sluttillståndet för en lugn vecka, inte
ett kantfall, och klustret står i det nu.
Symptomet pekade åt fel håll. terdut-tui hämtar listan och statistiken i
samma uppdatering, så incidentvyn såg trasig ut medan /api/incidents
svarade 200 med []. Loggen i klustret visar de två anropen bredvid
varandra, det ena grönt och det andra rött. Ingen ändring i terdut-tui
behövs: dess ListIncidents är oförändrad sedan 0.7.2 och skickar samma
parametrar som förut.
handleStatsAlerts bar samma fel och rättas likadant, innan någon hittar
det på samma sätt. COALESCE i SQL i stället för sql.NullInt64 i Go,
eftersom jämförelserna redan bor i satserna här (severityRankSQL, 279ef6c).
mtta_seconds och mttr_seconds lämnas medvetet utan COALESCE. null betyder
"inget att mäta ännu" och 0 skulle läsas som "omedelbart" — två olika
påståenden, och testet från 279ef6c låser fast skillnaden.
Claude-Session: https://claude.ai/code/session_01S7R4gWTz5wh5xCY4nCSJjN
The rename to git.ryuvia.com/niklas/... was a plain string substitution, so it
left the import blocks in their old order. The new path sorts before
github.com/go-chi/..., where the old one sorted after, which gofmt considers
unformatted.
go vet does not look at import order, so CI had nothing to say about it.
The module path, the container image, the Helm chart and the CI pipeline all
named GitHub. They now name the Gitea instance everything else already runs on.
The workflows are rewritten rather than translated. Gitea's runner image is
ubuntu:22.04, whose nodejs is Node 12, so no JS action runs there at all --
actions/checkout@v4 dies with a SyntaxError before it does anything. Every step
is shell, checkout is a plain clone (this repo is public, so it needs no
credential), and the jobs that need docker or helm run in host mode because the
dind bridge a `container:` job gets cannot reach github.com or get.helm.sh.
Two consequences worth naming:
- upload-artifact/download-artifact are also JS actions, and there is no
artifact store here, so the job that builds the binaries is the job that
publishes them. Nothing is passed between jobs.
- setup-qemu-action is gone with the rest, and the runner has no binfmt
registration. The Dockerfile's builder stage now runs on $BUILDPLATFORM and
cross-compiles from TARGETARCH instead, which is what keeps the arm64 image
buildable -- and makes it native rather than emulated.
The chart moves from a GitHub Pages index to an OCI artifact in Gitea's
registry. Publishing stays tag-only for the reason recorded in release.yaml: a
workflow triggered by the branch push cannot know the version it is about to be
tagged with.
The GitHub repository is left in place and untouched. Nothing pushes to it any
more, but its existing release downloads and chart index keep resolving.
Everything this server does assumes alerts arrive. If Prometheus stops
evaluating, or Alertmanager cannot reach us, nothing arrives — and
silence is indistinguishable from everything being fine. The cluster has
shipped the alert for exactly this case all along: Watchdog is
expr: vector(1), so it fires permanently and is re-sent forever, and it
is worth nothing unless something downstream notices it stop. Nothing
did. It arrived, opened no incident because a repeat_interval re-send is
not a new occurrence, and when the monitoring stack died the sweeper
quietly expired it and paged nobody.
So the handling is inverted for a configurable set of alerts: receiving
one opens no incident, and the absence of one does. TERDUT_DEADMAN_MATCHERS
selects them as label matchers, defaulting to alertname=Watchdog.
The unit of monitoring is the fingerprint rather than the alert name. Two
clusters sending the same Watchdog are two independent switches, so a
healthy one can never mask a dead one. Every matcher must name an
alertname, which keeps the sweeper's candidate query on alerts_name_idx
instead of JSON-extracting labels from every row, and leaves matching
with a single implementation.
A switch is dormant until its first heartbeat: a matcher nothing has ever
sent opens nothing, so a fresh deploy or a restored database does not
page. Resolving the incident by hand sticks, exactly as it does for an
alert-backed one, so a decommissioned source is a one-time page rather
than a nag; the switch re-arms only when the heartbeat comes back, and
dying again is a new incident.
The incident has no member alerts on purpose. Linking the heartbeat would
have the settled-incident cascade close it on the very sweep that opened
it, and there is no alert describing the problem anyway — the problem is
that no alert arrived. What happened is on the timeline instead, and
recovery is the only automatic way out.
One narrow exemption in the ingest guard makes recovery possible at all.
A heartbeat we declared dead is marked resolved, and the one that proves
us wrong carries the unchanged startsAt of an alert that never stopped
firing — so "resolution is terminal within an instance" would discard it
forever and a switch could die exactly once. The exemption is scoped to
resolution_source = 'deadman', which is the only resolution this server
infers from silence on a timeout of its own, so nothing another writer
set can be undone by a stale retry. Matched alerts are also held back
from the generic staleness expiry, which would otherwise resolve a
heartbeat as 'expiry' long before its own tighter deadline.
The timeout points the opposite way to TERDUT_STALE_AFTER: staleness is a
generous grace period around a repeat_interval you do not control, while
this is a deadline you set deliberately and configure the heartbeat's
route to beat. Inheriting a 4h or 12h repeat_interval gives a dead man's
switch with a twelve hour fuse, so the README spells out the route the
heartbeat needs.
A date is held by exactly one person and POST /api/schedule plain-inserts,
so any date that was already taken came back 409. That made reassignment
impossible through the API: the only route was to delete the entry first,
and for a week that meant seven separate deletions. Worse, the reject is
all-or-nothing across the request, so assigning a week where a single day
happened to be taken failed entirely and placed none of the other six.
The refusal itself is worth keeping. Moving a shift off the person
expecting to be paged for it should not be something a plain call does by
accident, so the fix is to make it possible to ask for rather than to
remove the guard: "replace": true takes the dates anyway, and the flag
defaults to off so every existing caller behaves exactly as before.
The delete and the insert share the transaction that was already there.
That matters more than the flag does — a week of free and taken days now
lands as a unit, and a failure part way through leaves the rota as it was
instead of with a shift deleted and nothing put back. A rota with a hole
in it is worse than a rota that refused to change.
One consequence worth naming: under replace a date repeated inside one
request is idempotent rather than a conflict, because the second pass
clears what the first wrote.
An incident's history went quiet after "Incident opened": nothing said
that anybody had been paged, reminded, or told it resolved. Delivery
lived only in the notifications outbox, which no API exposes, so when a
page failed to arrive there was nothing in the product that said whether
it had been sent.
The notifier now writes two event types. A notified event once ntfy
accepts the publish, carrying the kind in detail and the paged user in
user_id — absent when the page went to the shared fallback topic, which
belongs to nobody. And a notify_failed event when a notification
exhausts its retries, which is the one worth having: without it a page
that never landed leaves the timeline identical to one that did.
Both are written from the delivery result rather than at enqueue. A
queued notification is an intention, and the timeline is append-only, so
claiming somebody was told before ntfy accepted it would be a lie that
stays there. A failed timeline write is logged rather than returned, so
it cannot make a delivered row look unsent and send the page twice.
The topic is deliberately in neither: it is a shared secret with the
ntfy server, and every API key can read the timeline.
No migration — incident_events.type is free text, unlike
notifications.kind.
The handler never deletes the token: it stays valid until expires_at and
is purged by the sweeper, so a second tap is an idempotent no-op rather
than a rejection. Caught by pressing Acknowledge twice against the live
server. What bounds the token is scope -- one incident, one action, one
day -- not a use count.
An incident opened, got assigned to whoever held today's schedule entry,
and then sat there silently until somebody thought to look. The schedule
and the incident model were both built; nothing reached the person
holding the pager.
Notifications go out through ntfy, over plain HTTP with no new
dependencies. Delivery is an outbox rather than an inline call: the pool
is limited to a single connection, so a POST made while holding the
webhook's transaction would stall every other request behind it. The
webhook inserts a row and a notifier goroutine sends it within a tick,
retrying with exponential backoff.
Only opening an incident has to resolve a topic from scratch. Reminders
and all-clears reuse whatever that first notification chose, which keeps
configuration out of resolveIfSettled and gives the right rule for free:
you only hear that something resolved if you were told it started.
Each push carries an Acknowledge button, because the useful thing to do
at 3am is stop the pager without unlocking anything. It POSTs to an
unauthenticated /api/notify/ack/{token} — a notification body lives on
the ntfy server and in the device cache, so a real API key must never
appear in one. The token is minted per delivery, scoped to one incident
and one action, and expires in a day.
Reminders repeat until the incident stops being untouched. The stop
conditions are the states that already mean somebody has it: acknowledged,
snoozed, resolved, archived. Snooze is the mute button, so there is no
separate reminder cap.
Notifications sent to the fallback topic carry no Acknowledge button. The
topic is shared, and a button on it would let any subscriber acknowledge
as somebody else.
The alerts row was both Alertmanager's record and the human work queue, and
the two have different owners. The webhook upsert rewrites that row on every
notification; acknowledgement, comments and archiving were columns on it that
the upsert happened not to touch. So an alert that resolved and re-fired days
later still read as acknowledged by whoever acked the first occurrence — the
ack outlived the thing it referred to. Nothing recorded transitions either:
rows are mutated in place, so there was no timeline and no way to compute how
long anything took.
Alerts are now read-only signal records with two states, and incidents are
the work item: triggered, acknowledged or resolved, with an assignee, a
snooze, notes and an append-only timeline. Many alerts map to one incident,
and a new occurrence opens a new incident, which is what makes a stale ack
impossible rather than merely unlikely.
Correlation uses Alertmanager's own groupKey. It already grouped the alerts
according to the group_by routing tree the operator configured and sends the
result on every webhook, where it was being discarded; adopting it means
changing group_by in alertmanager.yml changes correlation here, with no
second grouping scheme to configure and keep in sync.
An incident opens only when an alert transitions into firing — an unseen
fingerprint, a newer startsAt, or a resolved alert starting again. The
unchanged notifications Alertmanager re-sends every repeat_interval are none
of those. That rule is what lets manual resolution be terminal: without it,
closing an incident by hand would be undone by the next re-send of an alert
that never stopped firing, and the button would be a lie. Snooze covers the
"not now" case instead. Incidents otherwise resolve by cascade, once every
alert under them has stopped firing, whether by webhook or by expiry.
New incidents are assigned to whoever holds today's schedule entry. The
schedule table has existed since the first release with nothing reading it.
Also here, following from the split:
- Incident severity is a high-water mark over its alerts, never lowered.
An incident that hit critical was a critical incident, and downgrading a
live one would demote it in the queue while the work is still open.
- /api/stats/incidents reports MTTA and MTTR, null rather than zero until
there is something to average. Neither was computable before.
- Alert archiving becomes sweeper-only housekeeping; the archive people
interact with is the incident's.
Breaking: the alert acknowledge, archive and comment endpoints are gone, and
the alert object drops the acknowledgement fields and gains incident_id. The
README maps each removed endpoint to its replacement. Migration 008 backfills
an incident per existing alert, archived ones included so no comment is
orphaned, carrying acknowledgements across and turning comments into timeline
notes.
Both documented alert contracts are untouched: received_at still advances on
every accepted payload, re-sends included, and resolution_source still says
how much to trust ends_at. The upsert is byte-for-byte what it was, now
running inside the ingest transaction.
The API reference listed endpoints but never the alert object's fields, so
two of them were load-bearing for clients while being described nowhere.
received_at appeared only in passing, as a stats filter; resolution_source
only inside the stale-expiry prose.
Both carry meaning a client cannot derive on its own. starts_at comes from
Prometheus and never changes for an alert instance, so received_at is the
only signal that a firing alert is still being refreshed — it advances on
every accepted webhook, including the unchanged notifications Alertmanager
re-sends every repeat_interval. resolution_source then says how much to
trust ends_at: under 'alertmanager' it is an end time somebody reported,
but under 'expiry' nothing ever reported one, so it is either a stale
watermark or the sweep timestamp, and only an upper bound.
README gains an alert object field table plus a contract section for each,
including the nullability rules and the advice to tolerate unrecognised
resolution_source values. The field comments in models.Alert now say these
are public API rather than ingest details, and the upsert carries a note at
the received_at line, which is where a regression would be introduced.
Three tests lock the newly documented behaviour, none of which was covered
before — the whole suite passed with the received_at bump deleted from the
upsert, because the expiry tests only ever set that column via SQL:
- a re-send advances received_at and leaves starts_at alone
- a discarded out-of-order retry does not count as a heartbeat
- an expiry resolve preserves a reported ends_at watermark and stamps
sweep time only when none was known
A resolved webhook was the only path out of the firing state, so a
notification that was dropped, silenced, or lost to a restart pinned an
alert as firing forever — Prometheus showed it resolved while
terdut-server kept listing it. The archiver only ever touched resolved
alerts, and both the list and stats queries compared status with plain
equality, so a stale row was indistinguishable from a live one.
A sweeper pass now resolves firing alerts on either of two signals: the
ends_at watermark Alertmanager sets on outgoing firing notifications has
passed (plus a grace period for clock skew), or no webhook has refreshed
the alert within TERDUT_STALE_AFTER (default 6h, above Alertmanager's 4h
repeat_interval). Such alerts get resolution_source = 'expiry',
distinguishing them from a real 'alertmanager' resolve.
Two related webhook bugs fixed alongside:
- The upsert had no ordering guard, so a retried firing notification
arriving after the resolved one resurrected the alert. Payloads for
an older alert instance are now discarded: a stale retry carries the
same startsAt, a genuine re-fire a newer one.
- archived_at was never cleared on re-fire, leaving a re-fired alert
archived and invisible in the default list.
Stats now exclude archived alerts to match the default list view; this
lowers historical firing/resolved totals.
The chart exposes both sweeper durations via sweeper.staleAfter and
sweeper.archiveAfter.
Alerts can be manually archived (POST /api/alerts/{id}/archive) or
unarchived (DELETE /api/alerts/{id}/archive). A background goroutine
auto-archives resolved alerts older than TERDUT_ARCHIVE_AFTER (default 7d).
GET /api/alerts hides archived alerts by default; ?archived=true shows them.
- GET /api/stats/alerts — total/firing/resolved counts
- GET /api/stats/alerts/top — most frequent alert names (?limit, default 10)
- GET /api/stats/alerts/by-hour — counts for all 24 hours (zeros filled in)
- GET /api/stats/alerts/by-day — counts for all 7 days with names (zeros filled in)
- All endpoints accept optional ?from/?to (YYYY-MM-DD) to filter by received_at
- Migration 005: schedule_entries table (date TEXT UNIQUE, one person per day)
- POST /api/schedule — assign user to one or more dates in a single
transaction; any date conflict rejects the whole request (409)
- GET /api/schedule — list all entries ordered by date, optional ?from/?to
- GET /api/schedule/current — today's on-call user (UTC date), 404 if none
- DELETE /api/schedule/{id} — remove an entry (204)
- Migration 004: acknowledged_by/acknowledged_at columns on alerts,
alert_comments table (FK cascade on delete)
- POST /api/alerts/{id}/acknowledge — stamps authed user + timestamp,
returns updated alert with acknowledged_by username
- DELETE /api/alerts/{id}/acknowledge — clears ack (204)
- GET /api/alerts/{id}/comments — list in chronological order
- POST /api/alerts/{id}/comments — add comment (returns 201)
- DELETE /api/alerts/{id}/comments/{commentID} — own comments only (204)
- All alert queries now LEFT JOIN users for ack username
- Migration 003: alerts table with fingerprint UNIQUE, JSON label/annotation
columns, nullable ends_at, and indexed status/name/received_at
- POST /api/alertmanager/webhook — upserts each alert by fingerprint;
zero endsAt ("0001-01-01") stored as NULL (still firing)
- GET /api/alerts — filtered list (?status, ?name, ?from, ?to, ?limit)
- GET /api/alerts/{id} — single alert lookup
- Migration 002: users and api_keys tables (Unix timestamps, FK cascade)
- POST /api/bootstrap — creates first user + key when DB is empty
- POST/GET/DELETE /api/users — user CRUD
- POST/DELETE /api/users/{id}/api-keys — key issuance and revocation
- AuthMiddleware: SHA-256 bearer token lookup, last_used_at tracking
- Raw key returned once on creation; only SHA-256 hash stored
- go mod init with chi, modernc.org/sqlite, golang.org/x/crypto
- Custom embedded migration runner (no CGO dependency)
- Config from TERDUT_ADDR / TERDUT_DB_PATH env vars
- chi router with /healthz endpoint
- Graceful shutdown on SIGINT/SIGTERM
Niklas Ye