The API reference listed endpoints but never the alert object's fields, so
two of them were load-bearing for clients while being described nowhere.
received_at appeared only in passing, as a stats filter; resolution_source
only inside the stale-expiry prose.
Both carry meaning a client cannot derive on its own. starts_at comes from
Prometheus and never changes for an alert instance, so received_at is the
only signal that a firing alert is still being refreshed — it advances on
every accepted webhook, including the unchanged notifications Alertmanager
re-sends every repeat_interval. resolution_source then says how much to
trust ends_at: under 'alertmanager' it is an end time somebody reported,
but under 'expiry' nothing ever reported one, so it is either a stale
watermark or the sweep timestamp, and only an upper bound.
README gains an alert object field table plus a contract section for each,
including the nullability rules and the advice to tolerate unrecognised
resolution_source values. The field comments in models.Alert now say these
are public API rather than ingest details, and the upsert carries a note at
the received_at line, which is where a regression would be introduced.
Three tests lock the newly documented behaviour, none of which was covered
before — the whole suite passed with the received_at bump deleted from the
upsert, because the expiry tests only ever set that column via SQL:
- a re-send advances received_at and leaves starts_at alone
- a discarded out-of-order retry does not count as a heartbeat
- an expiry resolve preserves a reported ends_at watermark and stamps
sweep time only when none was known
chart-release.yml fires on any charts/** push to main and ran
chart-releaser against the committed Chart.yaml version, failing with
422 already_exists whenever that version was already published. A tagged
release also publishes the chart from release.yml, which seds the
version from the tag, so the two workflows raced and this one lost.
Brings it to parity with the chart job in release.yml, which has carried
skip_existing since 1451682.
A resolved webhook was the only path out of the firing state, so a
notification that was dropped, silenced, or lost to a restart pinned an
alert as firing forever — Prometheus showed it resolved while
terdut-server kept listing it. The archiver only ever touched resolved
alerts, and both the list and stats queries compared status with plain
equality, so a stale row was indistinguishable from a live one.
A sweeper pass now resolves firing alerts on either of two signals: the
ends_at watermark Alertmanager sets on outgoing firing notifications has
passed (plus a grace period for clock skew), or no webhook has refreshed
the alert within TERDUT_STALE_AFTER (default 6h, above Alertmanager's 4h
repeat_interval). Such alerts get resolution_source = 'expiry',
distinguishing them from a real 'alertmanager' resolve.
Two related webhook bugs fixed alongside:
- The upsert had no ordering guard, so a retried firing notification
arriving after the resolved one resurrected the alert. Payloads for
an older alert instance are now discarded: a stale retry carries the
same startsAt, a genuine re-fire a newer one.
- archived_at was never cleared on re-fire, leaving a re-fired alert
archived and invisible in the default list.
Stats now exclude archived alerts to match the default list view; this
lowers historical firing/resolved totals.
The chart exposes both sweeper durations via sweeper.staleAfter and
sweeper.archiveAfter.
Alerts can be manually archived (POST /api/alerts/{id}/archive) or
unarchived (DELETE /api/alerts/{id}/archive). A background goroutine
auto-archives resolved alerts older than TERDUT_ARCHIVE_AFTER (default 7d).
GET /api/alerts hides archived alerts by default; ?archived=true shows them.
Post-install/post-upgrade Job that calls /api/bootstrap on first deploy
and stores the admin API key in a Secret (<release>-admin-key by default).
Exits cleanly on subsequent upgrades when bootstrap is already complete.
Adds ServiceAccount, Role (secrets:create), and RoleBinding as hook resources.
Tag-triggered workflow builds multi-platform binaries, pushes a multi-arch
Docker image to GHCR, bumps and releases the Helm chart, and creates a
GitHub release with all binary artifacts. Adds GPL-3.0 LICENSE and version
variable stamped at build time via ldflags.
charts/terdut-server/ — Helm chart for Kubernetes deployment:
- Deployment (replicas=1, /healthz probes, TERDUT_DB_PATH=/data/terdut.db)
- Service (ClusterIP :8080)
- PVC (1Gi, synology-iscsi) mounted at /data
- HTTPRoute via envoy-main gateway
.github/workflows/chart-release.yml — packages and publishes the chart to
gh-pages branch on any push to main that touches charts/; repo URL will be
https://yeniklas.github.io/terdut-server once the repo is made public
- GET /api/stats/alerts — total/firing/resolved counts
- GET /api/stats/alerts/top — most frequent alert names (?limit, default 10)
- GET /api/stats/alerts/by-hour — counts for all 24 hours (zeros filled in)
- GET /api/stats/alerts/by-day — counts for all 7 days with names (zeros filled in)
- All endpoints accept optional ?from/?to (YYYY-MM-DD) to filter by received_at
- Migration 005: schedule_entries table (date TEXT UNIQUE, one person per day)
- POST /api/schedule — assign user to one or more dates in a single
transaction; any date conflict rejects the whole request (409)
- GET /api/schedule — list all entries ordered by date, optional ?from/?to
- GET /api/schedule/current — today's on-call user (UTC date), 404 if none
- DELETE /api/schedule/{id} — remove an entry (204)
- Migration 004: acknowledged_by/acknowledged_at columns on alerts,
alert_comments table (FK cascade on delete)
- POST /api/alerts/{id}/acknowledge — stamps authed user + timestamp,
returns updated alert with acknowledged_by username
- DELETE /api/alerts/{id}/acknowledge — clears ack (204)
- GET /api/alerts/{id}/comments — list in chronological order
- POST /api/alerts/{id}/comments — add comment (returns 201)
- DELETE /api/alerts/{id}/comments/{commentID} — own comments only (204)
- All alert queries now LEFT JOIN users for ack username
- Migration 003: alerts table with fingerprint UNIQUE, JSON label/annotation
columns, nullable ends_at, and indexed status/name/received_at
- POST /api/alertmanager/webhook — upserts each alert by fingerprint;
zero endsAt ("0001-01-01") stored as NULL (still firing)
- GET /api/alerts — filtered list (?status, ?name, ?from, ?to, ?limit)
- GET /api/alerts/{id} — single alert lookup
- Migration 002: users and api_keys tables (Unix timestamps, FK cascade)
- POST /api/bootstrap — creates first user + key when DB is empty
- POST/GET/DELETE /api/users — user CRUD
- POST/DELETE /api/users/{id}/api-keys — key issuance and revocation
- AuthMiddleware: SHA-256 bearer token lookup, last_used_at tracking
- Raw key returned once on creation; only SHA-256 hash stored
- go mod init with chi, modernc.org/sqlite, golang.org/x/crypto
- Custom embedded migration runner (no CGO dependency)
- Config from TERDUT_ADDR / TERDUT_DB_PATH env vars
- chi router with /healthz endpoint
- Graceful shutdown on SIGINT/SIGTERM
Niklas Ye