Until now every authenticated caller could create and delete users, set
anybody's password and mint anybody's API keys -- auth.go said so in a
comment. Defensible with one operator and a hand-made account; not once
people sign themselves up (#7), and not in a multi-tenant install (#4),
where the user list is no longer everybody who works here.
users.is_admin is the flag. AdminOnly gates creating and deleting users
and granting the flag itself. The endpoints that are self-service for
your own account and administration for somebody else's -- password,
ntfy topic, API keys -- go through requireSelfOrAdmin instead, because
which rule applies depends on the {id} in the path rather than on the
route.
Minting your own API key stays self-service. A key carries exactly the
rights of the user it belongs to, so issuing one is no more than signing
in again; requiring an admin for it would mean a responder cannot set up
the TUI without somebody else in the room.
/api/users stays readable by everybody. The queue's assignment control
and the on-call schedule both have to name people, and hiding the roster
from the people on it buys nothing.
THE MIGRATION MAKES EVERY EXISTING USER AN ADMINISTRATOR. They already
hold these powers, so nobody's access changes on upgrade: it names what
is already true and leaves demotion as a deliberate act. Promoting only
user 1 would silently strip the others, and could leave an install whose
only administrator is an account nobody has a password for.
Two guards keep an install administrable: the last administrator can be
neither deleted nor demoted, and nobody can delete or demote themselves
-- the likelier accident, where the only admin clears their own flag
while tidying up and locks the door behind them.
No UI changes: there are no account-management screens yet. models.User
carries is_admin (not omitempty, so a client can tell false from an old
server), which is what #5's admin page will render from.
An incident opened, got assigned to whoever held today's schedule entry,
and then sat there silently until somebody thought to look. The schedule
and the incident model were both built; nothing reached the person
holding the pager.
Notifications go out through ntfy, over plain HTTP with no new
dependencies. Delivery is an outbox rather than an inline call: the pool
is limited to a single connection, so a POST made while holding the
webhook's transaction would stall every other request behind it. The
webhook inserts a row and a notifier goroutine sends it within a tick,
retrying with exponential backoff.
Only opening an incident has to resolve a topic from scratch. Reminders
and all-clears reuse whatever that first notification chose, which keeps
configuration out of resolveIfSettled and gives the right rule for free:
you only hear that something resolved if you were told it started.
Each push carries an Acknowledge button, because the useful thing to do
at 3am is stop the pager without unlocking anything. It POSTs to an
unauthenticated /api/notify/ack/{token} — a notification body lives on
the ntfy server and in the device cache, so a real API key must never
appear in one. The token is minted per delivery, scoped to one incident
and one action, and expires in a day.
Reminders repeat until the incident stops being untouched. The stop
conditions are the states that already mean somebody has it: acknowledged,
snoozed, resolved, archived. Snooze is the mute button, so there is no
separate reminder cap.
Notifications sent to the fallback topic carry no Acknowledge button. The
topic is shared, and a button on it would let any subscriber acknowledge
as somebody else.
The alerts row was both Alertmanager's record and the human work queue, and
the two have different owners. The webhook upsert rewrites that row on every
notification; acknowledgement, comments and archiving were columns on it that
the upsert happened not to touch. So an alert that resolved and re-fired days
later still read as acknowledged by whoever acked the first occurrence — the
ack outlived the thing it referred to. Nothing recorded transitions either:
rows are mutated in place, so there was no timeline and no way to compute how
long anything took.
Alerts are now read-only signal records with two states, and incidents are
the work item: triggered, acknowledged or resolved, with an assignee, a
snooze, notes and an append-only timeline. Many alerts map to one incident,
and a new occurrence opens a new incident, which is what makes a stale ack
impossible rather than merely unlikely.
Correlation uses Alertmanager's own groupKey. It already grouped the alerts
according to the group_by routing tree the operator configured and sends the
result on every webhook, where it was being discarded; adopting it means
changing group_by in alertmanager.yml changes correlation here, with no
second grouping scheme to configure and keep in sync.
An incident opens only when an alert transitions into firing — an unseen
fingerprint, a newer startsAt, or a resolved alert starting again. The
unchanged notifications Alertmanager re-sends every repeat_interval are none
of those. That rule is what lets manual resolution be terminal: without it,
closing an incident by hand would be undone by the next re-send of an alert
that never stopped firing, and the button would be a lie. Snooze covers the
"not now" case instead. Incidents otherwise resolve by cascade, once every
alert under them has stopped firing, whether by webhook or by expiry.
New incidents are assigned to whoever holds today's schedule entry. The
schedule table has existed since the first release with nothing reading it.
Also here, following from the split:
- Incident severity is a high-water mark over its alerts, never lowered.
An incident that hit critical was a critical incident, and downgrading a
live one would demote it in the queue while the work is still open.
- /api/stats/incidents reports MTTA and MTTR, null rather than zero until
there is something to average. Neither was computable before.
- Alert archiving becomes sweeper-only housekeeping; the archive people
interact with is the incident's.
Breaking: the alert acknowledge, archive and comment endpoints are gone, and
the alert object drops the acknowledgement fields and gains incident_id. The
README maps each removed endpoint to its replacement. Migration 008 backfills
an incident per existing alert, archived ones included so no comment is
orphaned, carrying acknowledgements across and turning comments into timeline
notes.
Both documented alert contracts are untouched: received_at still advances on
every accepted payload, re-sends included, and resolution_source still says
how much to trust ends_at. The upsert is byte-for-byte what it was, now
running inside the ingest transaction.
The API reference listed endpoints but never the alert object's fields, so
two of them were load-bearing for clients while being described nowhere.
received_at appeared only in passing, as a stats filter; resolution_source
only inside the stale-expiry prose.
Both carry meaning a client cannot derive on its own. starts_at comes from
Prometheus and never changes for an alert instance, so received_at is the
only signal that a firing alert is still being refreshed — it advances on
every accepted webhook, including the unchanged notifications Alertmanager
re-sends every repeat_interval. resolution_source then says how much to
trust ends_at: under 'alertmanager' it is an end time somebody reported,
but under 'expiry' nothing ever reported one, so it is either a stale
watermark or the sweep timestamp, and only an upper bound.
README gains an alert object field table plus a contract section for each,
including the nullability rules and the advice to tolerate unrecognised
resolution_source values. The field comments in models.Alert now say these
are public API rather than ingest details, and the upsert carries a note at
the received_at line, which is where a regression would be introduced.
Three tests lock the newly documented behaviour, none of which was covered
before — the whole suite passed with the received_at bump deleted from the
upsert, because the expiry tests only ever set that column via SQL:
- a re-send advances received_at and leaves starts_at alone
- a discarded out-of-order retry does not count as a heartbeat
- an expiry resolve preserves a reported ends_at watermark and stamps
sweep time only when none was known
A resolved webhook was the only path out of the firing state, so a
notification that was dropped, silenced, or lost to a restart pinned an
alert as firing forever — Prometheus showed it resolved while
terdut-server kept listing it. The archiver only ever touched resolved
alerts, and both the list and stats queries compared status with plain
equality, so a stale row was indistinguishable from a live one.
A sweeper pass now resolves firing alerts on either of two signals: the
ends_at watermark Alertmanager sets on outgoing firing notifications has
passed (plus a grace period for clock skew), or no webhook has refreshed
the alert within TERDUT_STALE_AFTER (default 6h, above Alertmanager's 4h
repeat_interval). Such alerts get resolution_source = 'expiry',
distinguishing them from a real 'alertmanager' resolve.
Two related webhook bugs fixed alongside:
- The upsert had no ordering guard, so a retried firing notification
arriving after the resolved one resurrected the alert. Payloads for
an older alert instance are now discarded: a stale retry carries the
same startsAt, a genuine re-fire a newer one.
- archived_at was never cleared on re-fire, leaving a re-fired alert
archived and invisible in the default list.
Stats now exclude archived alerts to match the default list view; this
lowers historical firing/resolved totals.
The chart exposes both sweeper durations via sweeper.staleAfter and
sweeper.archiveAfter.
Alerts can be manually archived (POST /api/alerts/{id}/archive) or
unarchived (DELETE /api/alerts/{id}/archive). A background goroutine
auto-archives resolved alerts older than TERDUT_ARCHIVE_AFTER (default 7d).
GET /api/alerts hides archived alerts by default; ?archived=true shows them.
- Migration 005: schedule_entries table (date TEXT UNIQUE, one person per day)
- POST /api/schedule — assign user to one or more dates in a single
transaction; any date conflict rejects the whole request (409)
- GET /api/schedule — list all entries ordered by date, optional ?from/?to
- GET /api/schedule/current — today's on-call user (UTC date), 404 if none
- DELETE /api/schedule/{id} — remove an entry (204)
- Migration 004: acknowledged_by/acknowledged_at columns on alerts,
alert_comments table (FK cascade on delete)
- POST /api/alerts/{id}/acknowledge — stamps authed user + timestamp,
returns updated alert with acknowledged_by username
- DELETE /api/alerts/{id}/acknowledge — clears ack (204)
- GET /api/alerts/{id}/comments — list in chronological order
- POST /api/alerts/{id}/comments — add comment (returns 201)
- DELETE /api/alerts/{id}/comments/{commentID} — own comments only (204)
- All alert queries now LEFT JOIN users for ack username
- Migration 003: alerts table with fingerprint UNIQUE, JSON label/annotation
columns, nullable ends_at, and indexed status/name/received_at
- POST /api/alertmanager/webhook — upserts each alert by fingerprint;
zero endsAt ("0001-01-01") stored as NULL (still firing)
- GET /api/alerts — filtered list (?status, ?name, ?from, ?to, ?limit)
- GET /api/alerts/{id} — single alert lookup
- Migration 002: users and api_keys tables (Unix timestamps, FK cascade)
- POST /api/bootstrap — creates first user + key when DB is empty
- POST/GET/DELETE /api/users — user CRUD
- POST/DELETE /api/users/{id}/api-keys — key issuance and revocation
- AuthMiddleware: SHA-256 bearer token lookup, last_used_at tracking
- Raw key returned once on creation; only SHA-256 hash stored
Niklas Ye