Terminal Duty (terdut-server)

On-call alert management server for teams using Prometheus Alertmanager.

• Receives Alertmanager webhooks directly — no adapter needed
• Stores and queries alerts (acknowledge, comment)
• On-call schedule management (user-to-day assignments)
• Alert statistics (by status, by hour, by day)
• REST API with per-user API key authentication
• Single binary, SQLite storage — trivial to self-host

---

Quick start

Prerequisites: Go 1.21+

git clone https://github.com/yeniklas/terdut-server
cd terdut-server
go run ./cmd/terdut

The server starts on :8080 with a terdut.db file in the working directory.

Create the first user

curl -X POST http://localhost:8080/api/bootstrap \
  -H "Content-Type: application/json" \
  -d '{"username": "admin", "email": "admin@example.com"}'

Save the api_key.key value from the response — it is shown once only.

Use it as a bearer token for all subsequent requests:

export KEY=<your-key>
curl -H "Authorization: Bearer $KEY" http://localhost:8080/api/users

Docker

docker build -t terdut-server .
docker run -p 8080:8080 -v $(pwd)/data:/data \
  -e TERDUT_DB_PATH=/data/terdut.db \
  terdut-server

---

Configuration

Variable	Default	Description
TERDUT_ADDR	:8080	TCP address to listen on
TERDUT_DB_PATH	terdut.db	Path to the SQLite database file
TERDUT_ARCHIVE_AFTER	168h (7d)	How long a resolved alert stays in the default list before being auto-archived
TERDUT_STALE_AFTER	6h	How long a firing alert may go without a refreshing webhook before it is treated as resolved — must exceed your Alertmanager repeat_interval

Durations use Go syntax (30m, 12h, 168h). An unparseable value falls back to the default.

In the Helm chart the two sweeper durations are set via sweeper.staleAfter and sweeper.archiveAfter.

---

Alertmanager configuration

Add terdut-server as a webhook receiver in your alertmanager.yml:

receivers:
  - name: terdut
    webhook_configs:
      - url: http://terdut-server:8080/api/alertmanager/webhook
        send_resolved: true

route:
  receiver: terdut

The webhook endpoint requires no authentication.

Stale alert expiry

A resolved webhook is the only signal that an alert has stopped firing, so a notification that is dropped, silenced, or lost to a restart would otherwise pin that alert as firing forever. A background sweeper resolves firing alerts that Alertmanager has stopped refreshing, using either signal:

• the endsAt watermark on the last notification has passed, or
• no webhook has refreshed the alert within TERDUT_STALE_AFTER.

Alertmanager re-sends firing notifications every repeat_interval, which is what keeps a live alert fresh — so TERDUT_STALE_AFTER must be comfortably larger than your repeat_interval (default 4h), or live alerts will be resolved prematurely. Alerts resolved this way are marked "resolution_source": "expiry" to distinguish them from a real Alertmanager resolve ("alertmanager").

---

API reference

Authentication

All endpoints except /api/bootstrap and /api/alertmanager/webhook require:

Authorization: Bearer <api-key>

Users

Method	Path	Description
POST	/api/bootstrap	Create first user + API key (only works on empty DB)
GET	/api/users	List users
POST	/api/users	Create user {"username","email"}
DELETE	/api/users/{id}	Delete user (cascades to keys)
POST	/api/users/{id}/api-keys	Issue API key {"name"} — key shown once
DELETE	/api/users/{id}/api-keys/{keyID}	Revoke API key

Alert ingestion

Method	Path	Description
POST	/api/alertmanager/webhook	Alertmanager v4 webhook receiver (no auth)

Alerts

Method	Path	Description
GET	/api/alerts	List alerts. Filters: ?status=firing|resolved, ?name=, ?archived=true, ?from=YYYY-MM-DD, ?to=YYYY-MM-DD, ?limit= (default 50, max 500)
GET	/api/alerts/{id}	Get single alert
POST	/api/alerts/{id}/acknowledge	Acknowledge alert (stamps authed user + time)
DELETE	/api/alerts/{id}/acknowledge	Clear acknowledgement
POST	/api/alerts/{id}/archive	Archive alert (hides it from the default list)
DELETE	/api/alerts/{id}/archive	Un-archive alert
GET	/api/alerts/{id}/comments	List comments (chronological)
POST	/api/alerts/{id}/comments	Add comment {"content"}
DELETE	/api/alerts/{id}/comments/{commentID}	Delete own comment

Archived alerts are hidden from GET /api/alerts unless ?archived=true is passed. Resolved alerts carry resolution_source: "alertmanager" for a real resolved webhook, "expiry" when the sweeper inferred it (see Stale alert expiry).

The alert object

Returned by GET /api/alerts (as an array) and GET /api/alerts/{id}. Timestamps are RFC 3339 in UTC. Fields marked optional are omitted entirely when unset, so clients must treat them as nullable.

Field	Type	Notes
id	integer	Server-assigned; stable for the life of the row
fingerprint	string	Alertmanager's fingerprint — the upsert key
name	string	From the alertname label
status	string	"firing" or "resolved"
labels	object	String→string, as sent by Alertmanager
annotations	object	String→string, as sent by Alertmanager
starts_at	timestamp	When the alert instance began, per Prometheus
ends_at	timestamp	optional — absent while no end is known
generator_url	string	Link back to the originating Prometheus
received_at	timestamp	When the server last accepted a webhook for this alert — see below
acknowledged_by_id	integer	optional — user id
acknowledged_by	string	optional — username
acknowledged_at	timestamp	optional
resolution_source	string	optional — "alertmanager" or "expiry"
archived_at	timestamp	optional — set while archived

received_at is a liveness heartbeat

starts_at comes from Prometheus and never changes for the lifetime of an alert instance. It says when the problem began, not whether it is still happening — an alert that started twelve days ago looks identical whether Alertmanager refreshed it a minute ago or went silent a week ago.

received_at is the field that answers "is this still live". It is set to the server's clock on every accepted webhook for that fingerprint, including the unchanged firing notifications Alertmanager re-sends every repeat_interval. Clients may rely on this:

• A firing alert whose received_at is advancing is still being refreshed. Stale-dating it against repeat_interval is a valid liveness check, and it is what the built-in sweeper does (see Stale alert expiry).
• received_at tracks accepted payloads, not delivery attempts. A retry that describes an older instance than the stored one is discarded, and a discarded payload does not move received_at.
• It stops advancing once the alert resolves, because Alertmanager stops re-sending. On an alert resolved by the sweeper ("resolution_source": "expiry") it therefore marks the last time Alertmanager was actually heard from, which is earlier than ends_at.

GET /api/alerts is ordered by received_at descending — most recently refreshed first — and the ?from= / ?to= filters on both the alert and stats endpoints select on received_at, not starts_at.

resolution_source says how much to trust ends_at

An alert can leave the firing state two ways, and resolution_source records which happened. Clients may rely on this:

• Absent while firing. It is set only on resolve, and a re-fire under the same fingerprint clears it again, so its presence always agrees with "status": "resolved".

• "alertmanager" — a real resolved webhook arrived. ends_at is the end time Alertmanager reported. It is an observed value and can be displayed as fact.

• "expiry" — the sweeper inferred the resolve because Alertmanager stopped refreshing the alert (see Stale alert expiry). Nothing ever reported an end, so ends_at is approximate: it is either the stale endsAt watermark from the last notification, or — when that notification carried none — the time the sweep ran, which lags the last real contact by up to TERDUT_STALE_AFTER plus a sweep interval. Treat it as "no later than", not as when the problem stopped.

  On these alerts received_at is the more truthful signal: it marks the last time Alertmanager was actually heard from. Surfacing the distinction is worthwhile, since "expiry" can also mean the alert is still firing and the notification path broke.

Treat the value as an open set and tolerate ones you do not recognise — new sources may be added, and unknown values should degrade to "resolved, reason unknown" rather than being rejected.

On-call schedule

Method	Path	Description
POST	/api/schedule	Assign user to dates {"user_id", "dates":["YYYY-MM-DD",...]} — all-or-nothing
GET	/api/schedule	List entries. Filters: ?from=YYYY-MM-DD, ?to=YYYY-MM-DD
GET	/api/schedule/current	Today's on-call user (UTC), 404 if none
DELETE	/api/schedule/{id}	Remove schedule entry

Statistics

All stat endpoints accept optional ?from=YYYY-MM-DD and ?to=YYYY-MM-DD to filter by received_at. Archived alerts are excluded, matching the default alert list.

Method	Path	Description
GET	/api/stats/alerts	{total, firing, resolved} counts
GET	/api/stats/alerts/top	Most frequent alert names. ?limit= (default 10, max 100)
GET	/api/stats/alerts/by-hour	Count per hour-of-day (UTC), all 24 slots returned
GET	/api/stats/alerts/by-day	Count per day-of-week, all 7 slots with names returned

---

Development

go test ./...        # run all tests
go build ./...       # compile all packages
go run ./cmd/terdut  # run locally