A resolved webhook was the only path out of the firing state, so a
notification that was dropped, silenced, or lost to a restart pinned an
alert as firing forever — Prometheus showed it resolved while
terdut-server kept listing it. The archiver only ever touched resolved
alerts, and both the list and stats queries compared status with plain
equality, so a stale row was indistinguishable from a live one.
A sweeper pass now resolves firing alerts on either of two signals: the
ends_at watermark Alertmanager sets on outgoing firing notifications has
passed (plus a grace period for clock skew), or no webhook has refreshed
the alert within TERDUT_STALE_AFTER (default 6h, above Alertmanager's 4h
repeat_interval). Such alerts get resolution_source = 'expiry',
distinguishing them from a real 'alertmanager' resolve.
Two related webhook bugs fixed alongside:
- The upsert had no ordering guard, so a retried firing notification
arriving after the resolved one resurrected the alert. Payloads for
an older alert instance are now discarded: a stale retry carries the
same startsAt, a genuine re-fire a newer one.
- archived_at was never cleared on re-fire, leaving a re-fired alert
archived and invisible in the default list.
Stats now exclude archived alerts to match the default list view; this
lowers historical firing/resolved totals.
The chart exposes both sweeper durations via sweeper.staleAfter and
sweeper.archiveAfter.
Alerts can be manually archived (POST /api/alerts/{id}/archive) or
unarchived (DELETE /api/alerts/{id}/archive). A background goroutine
auto-archives resolved alerts older than TERDUT_ARCHIVE_AFTER (default 7d).
GET /api/alerts hides archived alerts by default; ?archived=true shows them.
Post-install/post-upgrade Job that calls /api/bootstrap on first deploy
and stores the admin API key in a Secret (<release>-admin-key by default).
Exits cleanly on subsequent upgrades when bootstrap is already complete.
Adds ServiceAccount, Role (secrets:create), and RoleBinding as hook resources.
Tag-triggered workflow builds multi-platform binaries, pushes a multi-arch
Docker image to GHCR, bumps and releases the Helm chart, and creates a
GitHub release with all binary artifacts. Adds GPL-3.0 LICENSE and version
variable stamped at build time via ldflags.
charts/terdut-server/ — Helm chart for Kubernetes deployment:
- Deployment (replicas=1, /healthz probes, TERDUT_DB_PATH=/data/terdut.db)
- Service (ClusterIP :8080)
- PVC (1Gi, synology-iscsi) mounted at /data
- HTTPRoute via envoy-main gateway
.github/workflows/chart-release.yml — packages and publishes the chart to
gh-pages branch on any push to main that touches charts/; repo URL will be
https://yeniklas.github.io/terdut-server once the repo is made public
- GET /api/stats/alerts — total/firing/resolved counts
- GET /api/stats/alerts/top — most frequent alert names (?limit, default 10)
- GET /api/stats/alerts/by-hour — counts for all 24 hours (zeros filled in)
- GET /api/stats/alerts/by-day — counts for all 7 days with names (zeros filled in)
- All endpoints accept optional ?from/?to (YYYY-MM-DD) to filter by received_at
- Migration 005: schedule_entries table (date TEXT UNIQUE, one person per day)
- POST /api/schedule — assign user to one or more dates in a single
transaction; any date conflict rejects the whole request (409)
- GET /api/schedule — list all entries ordered by date, optional ?from/?to
- GET /api/schedule/current — today's on-call user (UTC date), 404 if none
- DELETE /api/schedule/{id} — remove an entry (204)
- Migration 004: acknowledged_by/acknowledged_at columns on alerts,
alert_comments table (FK cascade on delete)
- POST /api/alerts/{id}/acknowledge — stamps authed user + timestamp,
returns updated alert with acknowledged_by username
- DELETE /api/alerts/{id}/acknowledge — clears ack (204)
- GET /api/alerts/{id}/comments — list in chronological order
- POST /api/alerts/{id}/comments — add comment (returns 201)
- DELETE /api/alerts/{id}/comments/{commentID} — own comments only (204)
- All alert queries now LEFT JOIN users for ack username
- Migration 003: alerts table with fingerprint UNIQUE, JSON label/annotation
columns, nullable ends_at, and indexed status/name/received_at
- POST /api/alertmanager/webhook — upserts each alert by fingerprint;
zero endsAt ("0001-01-01") stored as NULL (still firing)
- GET /api/alerts — filtered list (?status, ?name, ?from, ?to, ?limit)
- GET /api/alerts/{id} — single alert lookup
- Migration 002: users and api_keys tables (Unix timestamps, FK cascade)
- POST /api/bootstrap — creates first user + key when DB is empty
- POST/GET/DELETE /api/users — user CRUD
- POST/DELETE /api/users/{id}/api-keys — key issuance and revocation
- AuthMiddleware: SHA-256 bearer token lookup, last_used_at tracking
- Raw key returned once on creation; only SHA-256 hash stored
- go mod init with chi, modernc.org/sqlite, golang.org/x/crypto
- Custom embedded migration runner (no CGO dependency)
- Config from TERDUT_ADDR / TERDUT_DB_PATH env vars
- chi router with /healthz endpoint
- Graceful shutdown on SIGINT/SIGTERM
Niklas Ye