niklas/terdut-server
1
0
Fork 0
Code Issues 2 Pull Requests Actions Packages Projects Releases 35 Wiki Activity
Files
a602ff3efca32011269a4a62f5eaa9bd5c04c79f
terdut-server/README.md
T
Niklas Ye a602ff3efc Document received_at and resolution_source as public contract 
The API reference listed endpoints but never the alert object's fields, so
two of them were load-bearing for clients while being described nowhere.
received_at appeared only in passing, as a stats filter; resolution_source
only inside the stale-expiry prose.

Both carry meaning a client cannot derive on its own. starts_at comes from
Prometheus and never changes for an alert instance, so received_at is the
only signal that a firing alert is still being refreshed — it advances on
every accepted webhook, including the unchanged notifications Alertmanager
re-sends every repeat_interval. resolution_source then says how much to
trust ends_at: under 'alertmanager' it is an end time somebody reported,
but under 'expiry' nothing ever reported one, so it is either a stale
watermark or the sweep timestamp, and only an upper bound.

README gains an alert object field table plus a contract section for each,
including the nullability rules and the advice to tolerate unrecognised
resolution_source values. The field comments in models.Alert now say these
are public API rather than ingest details, and the upsert carries a note at
the received_at line, which is where a regression would be introduced.

Three tests lock the newly documented behaviour, none of which was covered
before — the whole suite passed with the received_at bump deleted from the
upsert, because the expiry tests only ever set that column via SQL:

  - a re-send advances received_at and leaves starts_at alone
  - a discarded out-of-order retry does not count as a heartbeat
  - an expiry resolve preserves a reported ends_at watermark and stamps
    sweep time only when none was known
2026-07-30 09:02:44 +02:00

259 lines
10 KiB
Markdown
Raw Blame History

# Terminal Duty (terdut-server)
On-call alert management server for teams using Prometheus Alertmanager.
- Receives Alertmanager webhooks directly — no adapter needed
- Stores and queries alerts (acknowledge, comment)
- On-call schedule management (user-to-day assignments)
- Alert statistics (by status, by hour, by day)
- REST API with per-user API key authentication
- Single binary, SQLite storage — trivial to self-host
---
## Quick start
**Prerequisites:** Go 1.21+
```bash
git clone https://github.com/yeniklas/terdut-server
cd terdut-server
go run ./cmd/terdut
```
The server starts on `:8080` with a `terdut.db` file in the working directory.
### Create the first user
```bash
curl -X POST http://localhost:8080/api/bootstrap \
-H "Content-Type: application/json" \
-d '{"username": "admin", "email": "admin@example.com"}'
```
Save the `api_key.key` value from the response — it is shown **once only**.
Use it as a bearer token for all subsequent requests:
```bash
export KEY=<your-key>
curl -H "Authorization: Bearer $KEY" http://localhost:8080/api/users
```
### Docker
```bash
docker build -t terdut-server .
docker run -p 8080:8080 -v $(pwd)/data:/data \
-e TERDUT_DB_PATH=/data/terdut.db \
terdut-server
```
---
## Configuration
| Variable | Default | Description |
|---|---|---|
| `TERDUT_ADDR` | `:8080` | TCP address to listen on |
| `TERDUT_DB_PATH` | `terdut.db` | Path to the SQLite database file |
| `TERDUT_ARCHIVE_AFTER` | `168h` (7d) | How long a resolved alert stays in the default list before being auto-archived |
| `TERDUT_STALE_AFTER` | `6h` | How long a firing alert may go without a refreshing webhook before it is treated as resolved — **must exceed your Alertmanager `repeat_interval`** |
Durations use Go syntax (`30m`, `12h`, `168h`). An unparseable value falls back to the default.
In the Helm chart the two sweeper durations are set via `sweeper.staleAfter` and `sweeper.archiveAfter`.
---
## Alertmanager configuration
Add terdut-server as a webhook receiver in your `alertmanager.yml`:
```yaml
receivers:
- name: terdut
webhook_configs:
- url: http://terdut-server:8080/api/alertmanager/webhook
send_resolved: true
route:
receiver: terdut
```
The webhook endpoint requires no authentication.
### Stale alert expiry
A resolved webhook is the only signal that an alert has stopped firing, so a
notification that is dropped, silenced, or lost to a restart would otherwise pin
that alert as firing forever. A background sweeper resolves firing alerts that
Alertmanager has stopped refreshing, using either signal:
- the `endsAt` watermark on the last notification has passed, or
- no webhook has refreshed the alert within `TERDUT_STALE_AFTER`.
Alertmanager re-sends firing notifications every `repeat_interval`, which is what
keeps a live alert fresh — so `TERDUT_STALE_AFTER` must be comfortably larger
than your `repeat_interval` (default 4h), or live alerts will be resolved
prematurely. Alerts resolved this way are marked `"resolution_source": "expiry"`
to distinguish them from a real Alertmanager resolve (`"alertmanager"`).
---
## API reference
### Authentication
All endpoints except `/api/bootstrap` and `/api/alertmanager/webhook` require:
```
Authorization: Bearer <api-key>
```
### Users
| Method | Path | Description |
|---|---|---|
| `POST` | `/api/bootstrap` | Create first user + API key (only works on empty DB) |
| `GET` | `/api/users` | List users |
| `POST` | `/api/users` | Create user `{"username","email"}` |
| `DELETE` | `/api/users/{id}` | Delete user (cascades to keys) |
| `POST` | `/api/users/{id}/api-keys` | Issue API key `{"name"}` — key shown once |
| `DELETE` | `/api/users/{id}/api-keys/{keyID}` | Revoke API key |
### Alert ingestion
| Method | Path | Description |
|---|---|---|
| `POST` | `/api/alertmanager/webhook` | Alertmanager v4 webhook receiver (no auth) |
### Alerts
| Method | Path | Description |
|---|---|---|
| `GET` | `/api/alerts` | List alerts. Filters: `?status=firing\|resolved`, `?name=`, `?archived=true`, `?from=YYYY-MM-DD`, `?to=YYYY-MM-DD`, `?limit=` (default 50, max 500) |
| `GET` | `/api/alerts/{id}` | Get single alert |
| `POST` | `/api/alerts/{id}/acknowledge` | Acknowledge alert (stamps authed user + time) |
| `DELETE` | `/api/alerts/{id}/acknowledge` | Clear acknowledgement |
| `POST` | `/api/alerts/{id}/archive` | Archive alert (hides it from the default list) |
| `DELETE` | `/api/alerts/{id}/archive` | Un-archive alert |
| `GET` | `/api/alerts/{id}/comments` | List comments (chronological) |
| `POST` | `/api/alerts/{id}/comments` | Add comment `{"content"}` |
| `DELETE` | `/api/alerts/{id}/comments/{commentID}` | Delete own comment |
Archived alerts are hidden from `GET /api/alerts` unless `?archived=true` is
passed. Resolved alerts carry `resolution_source`: `"alertmanager"` for a real
resolved webhook, `"expiry"` when the sweeper inferred it (see
[Stale alert expiry](#stale-alert-expiry)).
#### The alert object
Returned by `GET /api/alerts` (as an array) and `GET /api/alerts/{id}`.
Timestamps are RFC 3339 in UTC. Fields marked *optional* are omitted entirely
when unset, so clients must treat them as nullable.
| Field | Type | Notes |
|---|---|---|
| `id` | integer | Server-assigned; stable for the life of the row |
| `fingerprint` | string | Alertmanager's fingerprint — the upsert key |
| `name` | string | From the `alertname` label |
| `status` | string | `"firing"` or `"resolved"` |
| `labels` | object | String→string, as sent by Alertmanager |
| `annotations` | object | String→string, as sent by Alertmanager |
| `starts_at` | timestamp | When the alert instance began, **per Prometheus** |
| `ends_at` | timestamp | *optional* — absent while no end is known |
| `generator_url` | string | Link back to the originating Prometheus |
| `received_at` | timestamp | When the server last accepted a webhook for this alert — see below |
| `acknowledged_by_id` | integer | *optional* — user id |
| `acknowledged_by` | string | *optional* — username |
| `acknowledged_at` | timestamp | *optional* |
| `resolution_source` | string | *optional* — `"alertmanager"` or `"expiry"` |
| `archived_at` | timestamp | *optional* — set while archived |
##### `received_at` is a liveness heartbeat
`starts_at` comes from Prometheus and **never changes** for the lifetime of an
alert instance. It says when the problem began, not whether it is still
happening — an alert that started twelve days ago looks identical whether
Alertmanager refreshed it a minute ago or went silent a week ago.
`received_at` is the field that answers "is this still live". It is set to the
server's clock on **every accepted webhook** for that fingerprint, including the
unchanged firing notifications Alertmanager re-sends every `repeat_interval`.
Clients may rely on this:
- **A firing alert whose `received_at` is advancing is still being refreshed.**
Stale-dating it against `repeat_interval` is a valid liveness check, and it is
what the built-in sweeper does (see
[Stale alert expiry](#stale-alert-expiry)).
- **`received_at` tracks accepted payloads, not delivery attempts.** A retry
that describes an older instance than the stored one is discarded, and a
discarded payload does not move `received_at`.
- **It stops advancing once the alert resolves,** because Alertmanager stops
re-sending. On an alert resolved by the sweeper
(`"resolution_source": "expiry"`) it therefore marks the last time
Alertmanager was actually heard from, which is earlier than `ends_at`.
`GET /api/alerts` is ordered by `received_at` descending — most recently
refreshed first — and the `?from=` / `?to=` filters on both the alert and stats
endpoints select on `received_at`, not `starts_at`.
##### `resolution_source` says how much to trust `ends_at`
An alert can leave the firing state two ways, and `resolution_source` records
which happened. Clients may rely on this:
- **Absent while firing.** It is set only on resolve, and a re-fire under the
same fingerprint clears it again, so its presence always agrees with
`"status": "resolved"`.
- **`"alertmanager"` — a real resolved webhook arrived.** `ends_at` is the end
time Alertmanager reported. It is an observed value and can be displayed as
fact.
- **`"expiry"` — the sweeper inferred the resolve** because Alertmanager stopped
refreshing the alert (see [Stale alert expiry](#stale-alert-expiry)). Nothing
ever reported an end, so **`ends_at` is approximate**: it is either the stale
`endsAt` watermark from the last notification, or — when that notification
carried none — the time the sweep ran, which lags the last real contact by up
to `TERDUT_STALE_AFTER` plus a sweep interval. Treat it as "no later than",
not as when the problem stopped.
On these alerts `received_at` is the more truthful signal: it marks the last
time Alertmanager was actually heard from. Surfacing the distinction is
worthwhile, since `"expiry"` can also mean the alert is still firing and the
notification path broke.
Treat the value as an open set and tolerate ones you do not recognise — new
sources may be added, and unknown values should degrade to "resolved, reason
unknown" rather than being rejected.
### On-call schedule
| Method | Path | Description |
|---|---|---|
| `POST` | `/api/schedule` | Assign user to dates `{"user_id", "dates":["YYYY-MM-DD",...]}` — all-or-nothing |
| `GET` | `/api/schedule` | List entries. Filters: `?from=YYYY-MM-DD`, `?to=YYYY-MM-DD` |
| `GET` | `/api/schedule/current` | Today's on-call user (UTC), 404 if none |
| `DELETE` | `/api/schedule/{id}` | Remove schedule entry |
### Statistics
All stat endpoints accept optional `?from=YYYY-MM-DD` and `?to=YYYY-MM-DD` to filter by `received_at`. Archived alerts are excluded, matching the default alert list.
| Method | Path | Description |
|---|---|---|
| `GET` | `/api/stats/alerts` | `{total, firing, resolved}` counts |
| `GET` | `/api/stats/alerts/top` | Most frequent alert names. `?limit=` (default 10, max 100) |
| `GET` | `/api/stats/alerts/by-hour` | Count per hour-of-day (UTC), all 24 slots returned |
| `GET` | `/api/stats/alerts/by-day` | Count per day-of-week, all 7 slots with names returned |
---
## Development
```bash
go test ./... # run all tests
go build ./... # compile all packages
go run ./cmd/terdut # run locally
```
Reference in New Issue View Git Blame Copy Permalink