Terminal Duty (terdut-server)

Incident management server for teams using Prometheus Alertmanager.

• Receives Alertmanager webhooks directly — no adapter needed
• Turns alerts into incidents, correlated by Alertmanager's own groupKey
• Incident workflow: acknowledge, assign, snooze, note, resolve, with a full timeline
• On-call schedule management, with new incidents auto-assigned to whoever is on call
• Alert and incident statistics, including MTTA and MTTR
• Web UI for phones and desktops, served by the same binary
• REST API with per-user API key authentication
• Single binary plus a Postgres — straightforward to self-host

---

Quick start

Prerequisites: Go 1.21+

git clone https://git.ryuvia.com/niklas/terdut-server
cd terdut-server
go run ./cmd/terdut

The server starts on :8080 with a terdut.db file in the working directory.

Create the first user

curl -X POST http://localhost:8080/api/bootstrap \
  -H "Content-Type: application/json" \
  -d '{"username": "admin", "email": "admin@example.com", "password": "<at least 10 characters>"}'

Save the api_key.key value from the response — it is shown once only. The password is optional and is what signs you in to the web UI.

Use it as a bearer token for all subsequent requests:

export KEY=<your-key>
curl -H "Authorization: Bearer $KEY" http://localhost:8080/api/users

Web UI

The server serves a web UI at /: the incident queue, each incident's alerts and timeline with every action (acknowledge, assign, snooze, note, resolve, archive), who is on call, the alert feed, and an Account tab for your own password and the ntfy topic your pages go to. It is built for a phone first. On a phone it has a bottom tab bar and a sticky action bar, it follows the system's dark mode, and it can be added to the home screen. From 900px wide it switches to a sidebar with the queue and the incident side by side. Statistics remain in terdut-tui for now.

You sign in with a username and password. Users have no password until one is set, and a user without one can only use API keys:

# an admin sets someone's first password with their API key
curl -X PUT http://localhost:8080/api/users/2/password \
  -H "Authorization: Bearer $KEY" -H "Content-Type: application/json" \
  -d '{"password": "<at least 10 characters>"}'

After that, users change it themselves under Account. Changing your own password requires the current one.

How a browser stays signed in:

• A successful login sets an HttpOnly, SameSite=Lax session cookie. It lasts 30 days and slides forward while it is used, so an on-call phone stays signed in.
• The cookie is marked Secure when TERDUT_PUBLIC_URL starts with https://, so set it to the HTTPS address. TLS terminates at the gateway and the server itself only ever sees plain HTTP.
• Requests authenticated by the cookie are checked for cross-origin use (Go's http.CrossOriginProtection). That is the CSRF guard. Bearer-key clients are not affected.
• Setting a password signs that user out everywhere else.
• Ten failed logins for one username within 15 minutes lock that username for the rest of the window.

With TERDUT_PUBLIC_URL set, tapping a push notification opens the incident in the web UI (/incidents/{id}).

A Team tab holds everything a team owns: the on-call rota, the escalation ladder, the alert sources with their keys, the dead man's switches and the membership. An owner edits it; a member sees the same page read-only, because the server refuses their writes anyway. Somebody in more than one team picks between them at the top.

The Admin tab appears only for a system administrator, and holds what belongs to the whole server rather than to one team: every team, every user, and the settings that used to be environment variables. Adding somebody is minting them an invite link into a team, rather than creating a bare account: the person who accepts it picks their own password, so one never passes through an administrator, and the link carries the team, so they land somewhere with a queue in it.

A name in that list opens that person's page, at /admin/users/{id}: their email and when they joined, where their notifications go, whether they are an administrator, whether the account is disabled, the teams they are in with their role in each, a password field for a first or forgotten one, and deletion. It is the one place membership is edited from the person's side — the Team tab answers "who is in this team", and answering "which teams is this person in" there means visiting each team in turn.

Docker

docker build -t terdut-server .
docker run -p 8080:8080 \
  -e TERDUT_DB_DSN='postgres://terdut:secret@host.docker.internal:5432/terdut?sslmode=disable' \
  terdut-server

The server creates its own schema on startup and needs a reachable Postgres; it stores nothing on disk, so there is no volume to mount.

Kubernetes

A Helm chart is published from this repository as an OCI artifact, versioned in lockstep with the app — chart x.y.z is always app vx.y.z:

helm upgrade --install terdut-server oci://git.ryuvia.com/niklas/terdut-server \
  --version 0.9.2 \
  --namespace terdut-server --create-namespace \
  --set networking.hostname=terdut.example.com

The chart expects a Gateway API Gateway named envoy-main in the envoy-gateway-system namespace to already exist — it renders an HTTPRoute against it rather than an Ingress. TLS is terminated at the gateway, so the server itself never sees a certificate.

Value	Default	Description
networking.hostname	terdut.example.com	Hostname the HTTPRoute serves
networking.listener	""	Gateway listener (sectionName) to bind to. Empty attaches to every matching listener, including plaintext HTTP — set it to the HTTPS listener's name to serve TLS only
networking.servicePort	8080	Port the route forwards to; keep in sync with service.port
bootstrap.enabled	true	Runs a post-install hook that creates the first user and stores its API key in the <release>-admin-key Secret. Already-bootstrapped servers are left alone
database.dsn	""	Required. Postgres DSN, with no password in it. The chart provisions no database
database.passwordSecret.name	""	Secret supplying PGPASSWORD. With the Zalando postgres operator, the Secret it generates for the role
database.passwordSecret.key	password	Key within that Secret

The API key travels in an Authorization: Bearer header, so set networking.listener whenever the hostname is reachable outside a trusted network.

The database

The chart provisions no database: it takes a DSN and expects a Postgres that already exists. In this cluster the wrapper chart declares an acid.zalan.do/v1 postgresql CR; anywhere else, any reachable Postgres 14+ will do.

The DSN carries no password. pgx falls back to libpq's environment variables for whatever the DSN leaves out, so the password arrives as PGPASSWORD from a Secret and never appears in values, in the rendered manifest or in kubectl describe pod. With the postgres operator that Secret is the one it generates for the role, so a rebuild mints a new password with nothing to keep in sync — the same wiring miniflux uses.

The server migrates its own schema on startup, so a new database only has to exist and be writable.

Backups

Postgres is backed up where it runs, not from here. The database pod carries a k8up k8up.io/backupcommand annotation that streams a pg_dump, the same way gitea and immich do in this cluster.

This used to be the app's problem: the SQLite database lived on a PVC beside the server, the image is FROM scratch with no interpreter to dump it, and WAL mode makes a file-level copy of the volume non-crash-consistent — so the chart shipped an idle python:*-alpine sidecar purely to give k8up somewhere to exec. The sidecar, the PVC and the backupSidecar values are all gone.

---

Configuration

Two kinds of setting, split by who changes them and how often.

Where the server is plugged in stays in the environment: the listen address, the database DSN, the ntfy URL and token, the public URL. They are needed before the database is open, and two of them are credentials.

How the server behaves lives in the database and is edited by an administrator in the web UI or through PUT /api/admin/settings, taking effect on the next sweep rather than at the next restart. The variables below marked seed are the value each of those starts from: written once, on first start, and never overwritten afterwards — a redeploy cannot put a chart's default back over an administrator's edit.

Variable	Default	Description
TERDUT_ADDR	:8080	TCP address to listen on
TERDUT_DB_DSN	—	Required. Postgres connection string, e.g. postgres://terdut:secret@localhost:5432/terdut?sslmode=require
TERDUT_ARCHIVE_AFTER	168h (7d)	seed. How long a resolved alert or incident stays in the default list before being auto-archived
TERDUT_STALE_AFTER	6h	seed. How long a firing alert may go without a refreshing webhook before it is treated as resolved — must exceed your Alertmanager repeat_interval
TERDUT_DEADMAN_MATCHERS	alertname=Watchdog	The default matchers a team starts with — switches are per team now, and this seeds teams that have no configuration of their own. ; separates matchers, , the label conditions within one, = is exact equality. Every matcher must name an alertname
TERDUT_DEADMAN_TIMEOUT	15m	How long a heartbeat may go unheard before its switch is declared dead — must be shorter than the repeat_interval of the route carrying it. 0 disables dead man's switch handling
TERDUT_DEADMAN_SEVERITY	critical	Severity a dead man's switch incident opens at
TERDUT_NTFY_URL	—	ntfy server to publish push notifications to. Empty disables notifications entirely
TERDUT_NTFY_TOKEN	—	Bearer token for an access-controlled ntfy
TERDUT_NTFY_FALLBACK_TOPIC	—	Topic used when nobody is on call
TERDUT_PUBLIC_URL	—	Base URL a phone uses to reach this server: the notification's link into the web UI, its Acknowledge button, and whether the session cookie is Secure
TERDUT_NOTIFY_REPEAT	15m	seed. How long an incident may sit unacknowledged before it is paged again. 0 notifies once and never repeats

Durations use Go syntax (30m, 12h, 168h). An unparseable value falls back to the default.

Note that TERDUT_STALE_AFTER and TERDUT_DEADMAN_TIMEOUT point in opposite directions. Staleness is a generous grace period around a repeat_interval you do not control; a dead man's switch is a deadline you set deliberately, and the heartbeat's route is configured to beat faster than it.

In the Helm chart the two sweeper durations are set via sweeper.staleAfter and sweeper.archiveAfter, dead man's switches via the deadman.* values, and notifications via the notify.* values.

---

Alertmanager configuration

Alerts arrive on a team's integration key, which says both that the sender may post and which team the alerts belong to. Mint one as an owner of the team:

curl -X POST https://terdut.example.com/api/teams/1/integrations \
  -H "Authorization: Bearer $TERDUT_API_KEY" \
  -H 'Content-Type: application/json' \
  -d '{"name":"prod alertmanager"}'

The response carries the key and the full URL once; only a SHA-256 hash is stored. Put it in your alertmanager.yml:

receivers:
  - name: terdut
    webhook_configs:
      - url: http://terdut-server:8080/api/integrations/<key>/alertmanager
        send_resolved: true

route:
  receiver: terdut

The whole URL is a credential, so treat it like one. Alertmanager 0.26 and later can read it from a file with url_file: instead, which keeps it out of your configuration repository:

      - url_file: /etc/alertmanager/secrets/terdut-webhook-url/url
        send_resolved: true

The webhook endpoint requires no authentication.

If you use the dead man's switch — and the default configuration does — give the heartbeat a route of its own, because the deadline is only as tight as the interval feeding it:

route:
  receiver: terdut
  repeat_interval: 4h
  routes:
    - matchers: [ 'alertname = "Watchdog"' ]
      receiver: terdut
      group_wait: 0s
      group_interval: 1m
      repeat_interval: 1m

That delivers a heartbeat every 2 minutes, not every minute. Alertmanager only reconsiders a group every group_interval, and at exactly one elapsed interval repeat_interval has not quite passed, so the send slips to the next tick — equal values give 2×. Two minutes against the 15 minute default is seven heartbeats per window, which is the point; use group_interval: 30s if you want the numbers to mean what they say.

kube-prometheus-stack users get the Watchdog alert (expr: vector(1)) for free; it just needs routing to terdut rather than to null.

---

Alerts and incidents

There are two objects, and the difference between them is the whole design.

An alert is Alertmanager's record. It has two states, firing and resolved, one row per fingerprint, and no human ever writes to it. The API exposes alerts read-only.

An incident is the work item. It goes triggered → acknowledged → resolved, carries an assignee, a snooze, notes and a timeline, and is the only thing people act on. Many alerts belong to one incident.

Correlation uses Alertmanager's groupKey

Alertmanager has already grouped alerts according to the group_by routing tree you configured, and it sends the resulting groupKey and groupLabels on every webhook. Incidents adopt that answer rather than re-grouping alerts a second time — if you want different correlation, change group_by in alertmanager.yml and terdut follows.

At most one incident is open per groupKey at a time. Alerts firing in a group that already has an open incident join it. The incident's severity is a high-water mark — the highest severity label any of its alerts has carried — so an incident that hit critical still reads as critical after the critical alert clears.

An incident opens only on a new occurrence

An incident opens when an alert transitions into firing: a fingerprint that was never seen, an alert with a newer startsAt, or a resolved alert that started again. The unchanged firing notifications Alertmanager re-sends every repeat_interval are none of those, and open nothing.

This is what makes closing an incident by hand mean something. Without the rule, POST /api/incidents/{id}/resolve would be undone by the next re-send of an alert that never stopped firing.

Leaving the open state

• Automatically, once every alert under the incident has stopped firing — whether by a resolved webhook or by the sweeper's stale-alert expiry. The incident gets "resolution_source": "alerts".
• By hand, via POST /api/incidents/{id}/resolve ("resolution_source": "manual"). This is terminal: a later occurrence in that group opens a new incident rather than reopening this one. If the alert underneath never stops firing, the incident stays closed — that is what resolving by hand asserts.
• On recovery, for a dead man's switch incident whose heartbeat started arriving again ("resolution_source": "recovered"). These incidents have no member alerts, so the automatic cascade above cannot reach them.

To quieten an incident you expect to come back, snooze it instead (POST /api/incidents/{id}/snooze). A snooze hides the incident from the default list without closing it, and expires by simply falling into the past.

On-call assignment

A new incident is assigned to whoever holds today's schedule entry at the moment it opens (GET /api/schedule/current). If nobody is scheduled it opens unassigned. Reassign with POST /api/incidents/{id}/assign.

One person holds a given day, so POST /api/schedule refuses a date somebody already has: taking a shift off the person expecting to be paged for it should not be something a plain call does by accident. Pass "replace": true to take them anyway. Either way the whole request is one transaction — a week where some days are free and some are taken moves as a unit, and a failure leaves the rota exactly as it was rather than with a hole in it.

Push notifications

With TERDUT_NTFY_URL set, an incident that opens is pushed to the on-call person's phone through ntfy. Everybody sets their own topic under Account in the web UI, where a Send a test push button proves it before an incident has to; PUT /api/users/{id}/notify is the same thing over the API, and an administrator may set somebody else's. A user with no topic falls back to TERDUT_NTFY_FALLBACK_TOPIC, as does an incident that opens with nobody on call. If neither yields a topic, nothing is queued.

The server is the install's one ntfy, from TERDUT_NTFY_URL, and is not something a user picks. Only the topic is per-person.

A topic is a shared secret with the ntfy server: anyone who knows it can both read the pages and publish to it, so an unguessable one is worth the trouble. That is also why the topic never appears in an incident's timeline, which every API key can read.

Three things get pushed:

• triggered — an incident opened. Priority follows severity (critical maps to ntfy's max priority, the one that overrides the phone's quiet settings).
• reminder — the incident is still triggered after TERDUT_NOTIFY_REPEAT. Repeats until somebody acts. Acknowledging, snoozing, resolving or archiving all stop it — snooze is the mute button.
• resolved — every alert under the incident stopped firing. Only sent to whoever was paged in the first place, and only for the automatic cascade: resolving by hand pushes nothing, since the person who did it already knows.

Notifications carry an Acknowledge button that acknowledges the incident without opening anything. It POSTs to /api/notify/ack/{token}, an unauthenticated route authorised by the 256-bit token in its path — minted fresh per notification, scoped to one incident and one action, and valid for 24 hours. A real API key is never put in a notification, because the message is stored on the ntfy server and cached on the device.

The token is not consumed by use. Acknowledging is idempotent, so a token stays valid for its full 24 hours and a second tap is a no-op that reports the incident's current state rather than an error — which is what you want when a tap is retried on a flaky mobile connection. What bounds it is scope, not a use count: one incident, one action, one day. Expired tokens are purged by the sweeper.

Two consequences worth planning for:

• /api/notify/ack/{token} must stay publicly reachable, or the button will not work when the responder is off your network.
• Notifications sent to the fallback topic carry no Acknowledge button. The topic is shared, and a button on it would let any subscriber acknowledge as somebody else.

Delivery is a queue, not an inline call: the webhook writes a row and a background notifier sends it within 30 seconds, retrying with exponential backoff up to 8 attempts. Nothing about ingestion blocks on ntfy being reachable.

Every delivery is recorded on the incident's timeline: a notified event once ntfy accepts the publish, and a notify_failed event when a notification exhausts its retries. Written from the result rather than at enqueue, so the timeline says what actually happened — and a page that never landed is visible instead of looking the same as one that did.

Escalation

Without a ladder, an unacknowledged incident re-pages the same topic every notify_repeat forever. That is a louder version of the same silence: if the person on call is asleep, out of signal, or has left, nothing else happens.

A team can configure an ordered ladder instead. Each level has a timeout and a set of targets, and a target is either a named person or whoever the team's rota says is on call today — the target that keeps working when the rota changes and nobody remembers to edit the policy.

level 1   5m    oncall            the rota gets first refusal
level 2   5m    user:bob          then a named second
                                  then repeat_count more rounds
                                  then the team's fallback topic, once

When a level's timeout passes with the incident still triggered, the next level is paged. Off the end of the ladder the whole thing runs again repeat_count times, and after that the team's fallback_topic is paged once as the end of the line. The incident stays open throughout: running out of people to wake is not the same as somebody answering.

Acknowledging or resolving stops it, which is the point — continuing to wake people after somebody has said "I have this" is how a tool teaches people to mute it. Snoozing pauses it: a deliberate "not now" holds the ladder where it is, and it resumes when the snooze runs out.

Every step is on the incident's timeline with the level and the names it woke, so somebody reading it afterwards can tell why their phone rang at 04:00. A level whose targets are all unreachable — no ntfy topic, a disabled account, an empty rota — is recorded as nobody reachable and the ladder moves on rather than stalling on a rung that cannot ring.

Reminders and escalation never both run. A team with a ladder gets escalation; a team without keeps the reminder behaviour exactly as it was. Two pages for one silence is the surest way to get a tool muted.

The ladder's fallback_topic is per team, unlike TERDUT_NTFY_FALLBACK_TOPIC, which is the install-wide topic used when an incident opens with nobody on call. They answer different questions: one is "nobody was scheduled", the other is "everybody scheduled has been tried".

Stale alert expiry

A resolved webhook is the only signal that an alert has stopped firing, so a notification that is dropped, silenced, or lost to a restart would otherwise pin that alert as firing forever. A background sweeper resolves firing alerts that Alertmanager has stopped refreshing, using either signal:

• the endsAt watermark on the last notification has passed, or
• no webhook has refreshed the alert within TERDUT_STALE_AFTER.

Alertmanager re-sends firing notifications every repeat_interval, which is what keeps a live alert fresh — so TERDUT_STALE_AFTER must be comfortably larger than your repeat_interval (default 4h), or live alerts will be resolved prematurely. Alerts resolved this way are marked "resolution_source": "expiry" to distinguish them from a real Alertmanager resolve ("alertmanager").

An expiry cascades: once it leaves an incident with nothing firing under it, the incident resolves too, in the same sweep.

Dead man's switch

Everything above assumes alerts arrive. If Prometheus stops evaluating, or Alertmanager cannot reach this server, nothing arrives — and silence looks exactly like everything being fine. A dead man's switch inverts the handling for one designated alert so that silence is the signal:

• receiving it opens no incident, and
• the absence of it does.

kube-prometheus-stack already ships the alert for this. Watchdog is expr: vector(1), so it fires permanently and is re-sent forever; it is worth nothing unless something downstream notices it stop. That is what TERDUT_DEADMAN_MATCHERS defaults to.

Switches belong to a team, which decides which of its own alerts are heartbeats and how long a silence has to last. An owner sets them through PUT /api/teams/{teamID}/deadman; a missed heartbeat opens an incident in the team whose integration received it.

The environment variables are the starting point, not the setting: at startup every team without a configuration of its own is given one from them, and an owner's later edit is never overwritten by a redeploy. A team created after that starts watching nothing until its owner says otherwise — inheriting an install-wide heartbeat would page a new team about a source it has never heard of.

A matcher is a set of exact label conditions, one of which must be the alertname, in the same format the environment variable uses:

alertname=Watchdog,cluster=prod; alertname=EdgeHeartbeat

The unit of monitoring is the fingerprint, not the alert name. Two clusters sending the same Watchdog are two independent switches, so a healthy one can never mask a dead one.

The lifecycle

A switch is dormant until its first heartbeat arrives. A configured matcher that has never been heard from opens nothing, so a fresh deploy or a restored database does not page. It also means a matcher that never matches anything is silently inert — check the startup log line, which lists the matchers that survived parsing.

Once armed, the sweeper declares it dead when either the heartbeat has not been refreshed within TERDUT_DEADMAN_TIMEOUT, or Alertmanager explicitly resolved it — the sender saying the heartbeat stopped needs no further waiting. That opens an incident at TERDUT_DEADMAN_SEVERITY, assigned and paged like any other, and marks the heartbeat alert "resolution_source": "deadman" so the alert list stops claiming a dead switch is firing.

It recovers when the heartbeat starts arriving again: the incident resolves with "resolution_source": "recovered" and the all-clear goes to whoever was paged.

Resolving the incident by hand sticks, the same way it does for an alert-backed one. While the switch stays silent nothing new opens — so a decommissioned source is a one-time page rather than a nag. The switch re-arms on the next heartbeat: come back and die again, and that is a new incident.

Two things to know

TERDUT_DEADMAN_TIMEOUT must be shorter than the repeat_interval of the route carrying the heartbeat, which is the exact opposite of TERDUT_STALE_AFTER. Inheriting a default repeat_interval of 4h gives you a switch that takes four hours to notice anything, so give the heartbeat its own route. Matched alerts are exempt from stale-alert expiry — a heartbeat answers to its own timeout and nothing else.

A dead man's switch incident has no member alerts: GET /api/incidents/{id}/alerts returns an empty list. There is no alert describing the problem, because the problem is that no alert arrived. What happened is on the timeline instead, as a deadman_silent event carrying the age of the last heartbeat, and the heartbeat's labels are on the incident's group_labels.

---

API reference

Authentication

All endpoints except /api/bootstrap, /api/integrations/{key}/alertmanager, /api/notify/ack/{token}, /api/login and /api/logout require either an API key:

Authorization: Bearer <api-key>

or the web UI's session cookie. A request that carries an Authorization header is judged on that header alone.

Two kinds of user exist. An administrator manages accounts: creating and deleting users, setting anybody's password, minting keys for anybody, and granting the flag itself. Everybody else works incidents — acknowledging, assigning, snoozing, resolving, noting — and manages their own account and nobody else's. An API key carries exactly the rights of the user it belongs to.

Getting an account. The first one comes from /api/bootstrap. After that it depends on signup_mode, an administrator setting:

• invite_only (the default) — a team owner mints a link with POST /api/teams/{teamID}/invites, and the person who opens it picks a username and password and lands in that team with the role the link carries. Links are single-use unless told otherwise, expire after seven days, and can be revoked before that.
• open — anybody who can reach the server can create an account, and must name a team, which they then own.

Invites are links, not email: this server has no SMTP, and adding it to send one message would be a subsystem to run, secure and monitor. Send the link however you already talk to the person.

A domain-restricted third mode was considered and dropped: with no email there is nothing to verify an address against, so it would only check the domain of a string somebody typed.

The first user, from /api/bootstrap, is an administrator. Users created afterwards are not, until an administrator says so. An install always keeps at least one: the last administrator can be neither deleted nor demoted, and nobody can delete or demote themselves.

Endpoints that require the flag answer 403 with {"error":"administrator access required"}.

Teams are the unit of tenancy, and are a separate axis from the administrator flag. A team owns its incidents, alerts, schedule and integrations, and a user sees exactly the teams they belong to. Within a team an owner configures it (schedule, integrations, membership) and a member works its incidents.

An administrator crosses that line in one direction only. They configure any team without being in it — every owner-only endpoint accepts the flag, because otherwise a team whose last owner left could never be repaired. They do not read any team: the queue, the alerts and the incidents are filtered by real membership, so an administrator sees a team's work only by joining it, which is a membership change and shows up as one. Administration is about accounts and the shape of a team, not about reading other people's incidents.

Anything belonging to a team you are not in answers 404, not 403: whether an incident exists is itself something only its team should learn.

Method	Path	Description
POST	/api/login	{"username","password"} → sets the session cookie, returns {user, has_password}. 429 after too many failures
POST	/api/logout	Ends the session and clears the cookie
GET	/api/me	The caller: {user, has_password}

Users

Method	Path	Description
admin marks an endpoint that requires the administrator flag; **self or
admin** marks one you may use on your own account and an administrator may use
on anybody's.

Method	Path	Who	Description
GET	/api/signup	—	Whether sign-up is open, and whether ?invite= is usable. No session needed: the caller has no account yet
POST	/api/signup	—	Create an account {"username","email","password","invite"?,"team_name"?} and sign in. 403 without a usable invite when the mode is invite-only
POST	/api/bootstrap	—	Create first user + API key {"username","email","password"?} (only works on empty DB). The user is an administrator
GET	/api/users	any	List users. Open to everybody: the queue's assignment control and the schedule both have to name people
GET	/api/users/{id}/teams	self or admin	The teams that user is in, each with their role. /api/teams is always about the caller; this one answers it about somebody else, for the admin page's per-user view. 404 for a user who does not exist, so "no teams" and "no such person" are distinguishable
POST	/api/users	admin	Create user {"username","email"}. Not an administrator
DELETE	/api/users/{id}	admin	Delete user (cascades to keys). 409 for yourself or the last administrator
PUT	/api/users/{id}/admin	admin	Grant or revoke the administrator flag {"is_admin"}. 409 for yourself or the last administrator
PUT	/api/users/{id}/disabled	admin	Take an account out of use, or put it back {"disabled"}. 409 for yourself or the last administrator
PUT	/api/users/{id}/notify	self or admin	Set push notification target {"ntfy_topic"} — empty string clears it
PUT	/api/users/{id}/password	self or admin	Set web UI password {"password","current_password"}. current_password is required only when changing your own existing password. Ends the user's other sessions
POST	/api/users/{id}/api-keys	self or admin	Issue API key {"name"} — key shown once
DELETE	/api/users/{id}/api-keys/{keyID}	self or admin	Revoke API key

Administration

Method	Path	Who	Description
GET	/api/admin/teams	admin	Every team on the server, with its member and open-incident counts. /api/teams answers "what am I in"; this answers "what is there"
GET	/api/admin/settings	admin	The editable settings with their bounds, plus the environment-configured ones, read-only. Never credentials
PUT	/api/admin/settings	admin	Change one or more {"key": seconds}, or {"signup_mode": "open"|"invite_only"}. 400 for an unknown key or a value outside its bounds

Alert ingestion

Alerts arrive on a team's integration key. The key is both the credential and the routing: it says that the sender may post, and which team the alerts belong to. Create one with POST /api/teams/{teamID}/integrations, which returns the key and the full URL once and stores only a SHA-256 hash.

Method	Path	Description
POST	/api/integrations/{key}/alertmanager	Alertmanager v4 webhook receiver for the key's team. 401 for an unknown key

This is the only way in. The pre-teams POST /api/alertmanager/webhook took no credential at all — anything able to reach the port could open an incident — and was removed in v0.13.0 once senders had moved onto keys.

Teams

owner below means an owner of that team or a system administrator, who passes every one of these without being a member — see Authentication. member means membership and nothing else: an administrator who is not in the team gets the same 404 as anybody else.

Method	Path	Who	Description
GET	/api/teams	any	The caller's own teams, each with their role
POST	/api/teams	any	Create a team {"name"}; the creator becomes its first owner
PUT	/api/teams/{teamID}	owner	Rename it {"name"}. 409 if the name is taken
DELETE	/api/teams/{teamID}	owner	Delete a team and everything under it. 409 while it has open incidents
GET	/api/teams/{teamID}/members	member	Who is in the team
POST	/api/teams/{teamID}/members	owner	Add a member, or change their role {"user_id","role"}
DELETE	/api/teams/{teamID}/members/{userID}	owner	Remove a member. 409 for the last owner
GET	/api/teams/{teamID}/integrations	member	List integrations. Never returns keys
POST	/api/teams/{teamID}/integrations	owner	Mint an integration {"name","kind"} — key and URL shown once
DELETE	/api/teams/{teamID}/integrations/{integrationID}	owner	Revoke an integration
GET	/api/teams/{teamID}/invites	owner	The team's invite links, with their uses and expiry. Never the tokens
POST	/api/teams/{teamID}/invites	owner	Mint one {"role","max_uses"} — the full URL is returned once
DELETE	/api/teams/{teamID}/invites/{inviteID}	owner	Revoke a link before it expires
GET	/api/teams/{teamID}/escalation	member	The team's escalation ladder {repeat_count, fallback_topic, levels[]}. Empty levels means the team has none
PUT	/api/teams/{teamID}/escalation	owner	Replace it wholesale. 400 for a level with no targets or no timeout — a rung that pages nobody is a silence with a number on it
GET	/api/teams/{teamID}/deadman	member	The team's dead man's switch configuration {matchers, timeout_seconds, severity}
PUT	/api/teams/{teamID}/deadman	owner	Replace it. 400 when no matcher names an alertname, because a switch that silently watches nothing is the failure this feature exists to prevent

Notifications

Method	Path	Description
POST	/api/notify/ack/{token}	Acknowledge an incident from a push notification's Acknowledge button. No auth: the token in the path is the credential — one incident, one action, 24 hours, idempotent. Must stay publicly reachable

Incidents

Method	Path	Description
GET	/api/incidents	List incidents. Filters: ?status=triggered|acknowledged|resolved, ?severity=, ?assigned_to=<user id>, ?archived=true, ?snoozed=true, ?from=YYYY-MM-DD, ?to=YYYY-MM-DD, ?sort=severity, ?limit= (default 50, max 500)
GET	/api/incidents/{id}	Get single incident, with its alerts inline
GET	/api/incidents/{id}/alerts	Alerts under this incident
GET	/api/incidents/{id}/timeline	Full event history, chronological
POST	/api/incidents/{id}/acknowledge	Acknowledge (stamps authed user + time)
DELETE	/api/incidents/{id}/acknowledge	Clear acknowledgement, back to triggered
POST	/api/incidents/{id}/resolve	Close by hand — terminal, see above
POST	/api/incidents/{id}/assign	Reassign {"user_id"}
POST	/api/incidents/{id}/snooze	Hide until {"until": RFC3339} or {"duration": "2h"}
DELETE	/api/incidents/{id}/snooze	Un-snooze
POST	/api/incidents/{id}/archive	Archive (hides from the default list)
DELETE	/api/incidents/{id}/archive	Un-archive
POST	/api/incidents/{id}/notes	Add a note {"content"}
DELETE	/api/incidents/{id}/notes/{eventID}	Delete own note

With no ?status= filter, GET /api/incidents returns open incidents only — the queue an on-call person wants. Currently snoozed and archived incidents are excluded unless asked for. Actions that only make sense on an open incident return 409 once it is resolved.

Notes are ordinary timeline events of type note; only they are deletable, and only by their author. The rest of the timeline is a record of what happened.

The incident object

Field	Type	Notes
id	integer	Server-assigned
group_key	string	Alertmanager's groupKey — opaque, treat as an identifier
title	string	Rendered from groupLabels
group_labels	object	String→string, as sent by Alertmanager
status	string	"triggered", "acknowledged" or "resolved"
severity	string	optional — high-water mark across the incident's alerts; never lowered
triggered_at	timestamp	When the incident opened
acknowledged_by_id / acknowledged_by / acknowledged_at		optional — user id, username, time
assigned_to_id / assigned_to		optional — user id, username
snoozed_until	timestamp	optional — a value in the past reads as not snoozed
resolved_at	timestamp	optional
resolution_source	string	optional — "alerts", "manual" or "recovered"
archived_at	timestamp	optional
alerts	array	Only on GET /api/incidents/{id}

Treat resolution_source as an open set, as with the alert field of the same name: degrade unknown values to "resolved, reason unknown".

The timeline event object

Field	Type	Notes
id	integer
incident_id	integer
type	string	See below — treat as an open set
user_id / username		optional — absent when the server acted rather than a person
alert_id	integer	optional — the alert an alert_added / alert_resolved event refers to
detail	string	optional — the note body, the snooze deadline, etc.
created_at	timestamp

Types written today: triggered, alert_added, alert_resolved, acknowledged, unacknowledged, assigned, snoozed, unsnoozed, resolved, note, notified, notify_failed, deadman_silent. On an assigned event user_id is the assignee, not the actor. New types may be added; render unknown ones generically rather than dropping them.

On notified and notify_failed, detail carries the notification kind (triggered | reminder | resolved), and on a failure the reason after it. user_id is who was paged — absent means the page went to the shared fallback topic and so belongs to nobody. The topic itself is never written to the timeline: it is a shared secret with the ntfy server, and every API key can read this.

Alerts

Alerts are read-only. Everything a person does happens on the incident.

Method	Path	Description
GET	/api/alerts	List alerts. Filters: ?status=firing|resolved, ?name=, ?incident_id=, ?archived=true, ?from=YYYY-MM-DD, ?to=YYYY-MM-DD, ?limit= (default 50, max 500)
GET	/api/alerts/{id}	Get single alert

Archived alerts are hidden from GET /api/alerts unless ?archived=true is passed; alert archiving is automatic housekeeping by the sweeper, not a user action. Resolved alerts carry resolution_source: "alertmanager" for a real resolved webhook, "expiry" when the sweeper inferred it (see Stale alert expiry), "deadman" for a heartbeat declared dead (see Dead man's switch).

The alert object

Returned by GET /api/alerts (as an array) and GET /api/alerts/{id}. Timestamps are RFC 3339 in UTC. Fields marked optional are omitted entirely when unset, so clients must treat them as nullable.

Field	Type	Notes
id	integer	Server-assigned; stable for the life of the row
fingerprint	string	Alertmanager's fingerprint — the upsert key
name	string	From the alertname label
status	string	"firing" or "resolved"
labels	object	String→string, as sent by Alertmanager
annotations	object	String→string, as sent by Alertmanager
starts_at	timestamp	When the alert instance began, per Prometheus
ends_at	timestamp	optional — absent while no end is known
generator_url	string	Link back to the originating Prometheus
received_at	timestamp	When the server last accepted a webhook for this alert — see below
incident_id	integer	optional — the most recent incident this alert belongs to
resolution_source	string	optional — "alertmanager", "expiry" or "deadman"
archived_at	timestamp	optional — set while archived

received_at is a liveness heartbeat

starts_at comes from Prometheus and never changes for the lifetime of an alert instance. It says when the problem began, not whether it is still happening — an alert that started twelve days ago looks identical whether Alertmanager refreshed it a minute ago or went silent a week ago.

received_at is the field that answers "is this still live". It is set to the server's clock on every accepted webhook for that fingerprint, including the unchanged firing notifications Alertmanager re-sends every repeat_interval. Clients may rely on this:

• A firing alert whose received_at is advancing is still being refreshed. Stale-dating it against repeat_interval is a valid liveness check, and it is what the built-in sweeper does (see Stale alert expiry).
• received_at tracks accepted payloads, not delivery attempts. A retry that describes an older instance than the stored one is discarded, and a discarded payload does not move received_at.
• It stops advancing once the alert resolves, because Alertmanager stops re-sending. On an alert resolved by the sweeper ("resolution_source": "expiry") it therefore marks the last time Alertmanager was actually heard from, which is earlier than ends_at.

GET /api/alerts is ordered by received_at descending — most recently refreshed first — and the ?from= / ?to= filters on both the alert and stats endpoints select on received_at, not starts_at.

resolution_source says how much to trust ends_at

An alert can leave the firing state two ways, and resolution_source records which happened. Clients may rely on this:

• Absent while firing. It is set only on resolve, and a re-fire under the same fingerprint clears it again, so its presence always agrees with "status": "resolved".

• "alertmanager" — a real resolved webhook arrived. ends_at is the end time Alertmanager reported. It is an observed value and can be displayed as fact.

• "expiry" — the sweeper inferred the resolve because Alertmanager stopped refreshing the alert (see Stale alert expiry). Nothing ever reported an end, so ends_at is approximate: it is either the stale endsAt watermark from the last notification, or — when that notification carried none — the time the sweep ran, which lags the last real contact by up to TERDUT_STALE_AFTER plus a sweep interval. Treat it as "no later than", not as when the problem stopped.

  On these alerts received_at is the more truthful signal: it marks the last time Alertmanager was actually heard from. Surfacing the distinction is worthwhile, since "expiry" can also mean the alert is still firing and the notification path broke.

• "deadman" — a heartbeat was declared dead (see Dead man's switch). Like "expiry", an inference from silence rather than an observed end, so ends_at is approximate — but a much tighter one, bounded by TERDUT_DEADMAN_TIMEOUT. It is also the one resolution a re-fire under the same starts_at can undo, since the switch coming back is exactly the evidence that the inference was wrong.

Treat the value as an open set and tolerate ones you do not recognise — new sources may be added, and unknown values should degrade to "resolved, reason unknown" rather than being rejected.

On-call schedule

Method	Path	Description
Each team keeps its own rota, so two teams can have two different people on call
on the same day. The person taking a shift has to be in the team — paging
somebody who cannot open the incident is worse than paging nobody.

Method	Path	Who	Description
POST	/api/teams/{teamID}/schedule	owner	Assign user to dates {"user_id", "dates":["YYYY-MM-DD",...], "replace"} — all-or-nothing
GET	/api/teams/{teamID}/schedule	member	List entries. Filters: ?from=YYYY-MM-DD, ?to=YYYY-MM-DD
DELETE	/api/teams/{teamID}/schedule/{id}	owner	Remove schedule entry
GET	/api/schedule/current	any	Who is on call today (UTC) in every team the caller is in — one entry per team, [] when nobody anywhere

Statistics

Every figure counts the caller's own teams only: a report that counted other teams' incidents would leak their volume, and their alert names through the top-alerts list, and would not be a number about the reader's work anyway.

All stat endpoints accept optional ?from=YYYY-MM-DD and ?to=YYYY-MM-DD, and exclude archived rows to match the default list views. Alert stats filter on received_at; incident stats filter on triggered_at.

Method	Path	Description
GET	/api/stats/incidents	{total, triggered, acknowledged, resolved, mtta_seconds, mttr_seconds}
GET	/api/stats/alerts	{total, firing, resolved} counts
GET	/api/stats/alerts/top	Most frequent alert names. ?limit= (default 10, max 100)
GET	/api/stats/alerts/by-hour	Count per hour-of-day (UTC), all 24 slots returned
GET	/api/stats/alerts/by-day	Count per day-of-week, all 7 slots with names returned

mtta_seconds (time to acknowledge) and mttr_seconds (time to resolve) are averages over incidents that have actually been acknowledged or resolved, and are null until there are any — null means "no data", not zero.

---

Upgrading to teams

Everything that existed before teams moves into one team called Default, and every existing user becomes an owner of it. The upgrade is a no-op for the people using it: the same queue, the same schedule, the same incidents, with a name on them.

What changes, and will need attention:

• Alert ingestion moved. Mint a key with POST /api/teams/{teamID}/integrations and point Alertmanager at the URL it returns. In v0.12.0 the old POST /api/alertmanager/webhook still worked, deprecated, routing everything to the oldest team; v0.13.0 removes it, so upgrade straight from v0.11.x to v0.13.0 only after the senders are moved.
• The schedule endpoints moved under /api/teams/{teamID}/schedule, and editing the rota is now an owner's job. GET /api/schedule/current stayed where it was but now returns an array — one entry per team with somebody on call — instead of a single object or a 404. This is a breaking API change for anything that reads it, terdut-tui included.
• Uniqueness is per team now. Two teams can legitimately see the same alert fingerprint, the same Alertmanager groupKey, and put somebody on call on the same date.

Dead man's switches moved too. TERDUT_DEADMAN_MATCHERS, _TIMEOUT and _SEVERITY are no longer the setting; they are the default each existing team is seeded with at startup, after which an owner edits them per team through PUT /api/teams/{teamID}/deadman and a redeploy never overwrites that.

Nothing else about an incident changes, and incidents never move between teams: an alert belongs to whichever team's key it arrived on.

Upgrading to roles

Before this release every authenticated caller could create and delete users, set anybody's password and mint anybody's API keys. That is now the administrator flag, and the migration makes every existing user an administrator — they already held those powers, so nobody's access changes on upgrade and demotion is a deliberate act afterwards. Promoting only the first user would have silently stripped the rest, and could leave an install whose only administrator is an account nobody has a password for.

Users created after the upgrade are not administrators. Hand the flag out with:

curl -X PUT https://terdut.example.com/api/users/7/admin \
  -H "Authorization: Bearer $TERDUT_API_KEY" \
  -H 'Content-Type: application/json' \
  -d '{"is_admin": true}'

Nothing in the API changed shape, so terdut-tui needs no new version — but a non-administrator now gets 403 where a 200 used to come back.

Upgrading from SQLite

Versions up to v0.10.2 stored everything in a SQLite file. From v0.11.1 the server needs TERDUT_DB_DSN and keeps nothing on disk.

The copy was done by scripts/sqlite-to-postgres.go, which was deleted in v0.13.0 along with the SQLite driver it was the last user of. It is still in the history — check out the v0.12.0 tag to get it:

git show v0.12.0:scripts/sqlite-to-postgres.go > sqlite-to-postgres.go

The cutover is ordered, and the server must not be running while the copy happens: stop the old version, let the new binary build the schema against an empty Postgres, run the script with -sqlite and -dsn, then start the new version for good. On Kubernetes step three runs as a Job with the same image against the PVC before it is removed.

The copy preserves every id, so incidents keep their numbers and the timeline, alert membership, outbox and ack tokens all still point where they did. It refuses a target that already has rows, so a second run cannot double-insert.

Upgrading to incidents

The incidents release moves the workflow off alerts, which is a breaking API change. These endpoints are gone:

Removed	Replacement
POST/DELETE /api/alerts/{id}/acknowledge	POST/DELETE /api/incidents/{id}/acknowledge
POST/DELETE /api/alerts/{id}/archive	POST/DELETE /api/incidents/{id}/archive (alert archiving is now sweeper-only)
GET/POST /api/alerts/{id}/comments	GET /api/incidents/{id}/timeline, POST /api/incidents/{id}/notes
DELETE /api/alerts/{id}/comments/{commentID}	DELETE /api/incidents/{id}/notes/{eventID}

The alert object also drops acknowledged_by_id, acknowledged_by and acknowledged_at, and gains incident_id.

Migration 008_incidents.sql runs automatically on start and preserves existing data: every alert gets a backfilled incident carrying its acknowledgement, and comments become timeline notes. Backfilled incidents have a group_key of backfill:<fingerprint> — there is no historical groupKey to correlate on, so they are one-per-alert rather than grouped.

Nothing about the two documented alert contracts changes: received_at is still advanced on every accepted webhook, and resolution_source still means what it did.

Upgrading to dead man's switches

Dead man's switch handling is on by default, watching alertname=Watchdog with a 15 minute timeout. If you already route Watchdog to this server, the behaviour of that alert changes on upgrade, in both directions:

• it stops opening incidents when it arrives, and
• it starts opening one when it stops arriving.

Check your repeat_interval before upgrading. The switch pages whenever a heartbeat has not been refreshed within TERDUT_DEADMAN_TIMEOUT, so a Watchdog route inheriting a 4h or 12h repeat_interval will page constantly against the 15 minute default. Either give the heartbeat its own fast route — the point of the feature — or set TERDUT_DEADMAN_TIMEOUT above your current repeat_interval until you have. TERDUT_DEADMAN_TIMEOUT=0 turns the whole thing off.

There is no migration and no schema change. An existing open incident from a Watchdog that arrived under the old behaviour is unaffected; resolve it by hand.

---

Development

make test-db         # start a local Postgres for the tests (podman or docker)
make test            # run all tests
go build ./...       # compile all packages
go run ./cmd/terdut  # run locally (needs TERDUT_DB_DSN)

The tests need a real Postgres, because the server does — there is no in-memory Postgres the way there was an in-memory SQLite. TERDUT_TEST_DSN says where it is, make test-db starts one on port 5433 and prints the DSN, and make test-db-stop removes it. Each test gets its own schema on that server, so tests cannot see each other's rows. An unset TERDUT_TEST_DSN fails the suite rather than skipping it: a run that quietly tests nothing is worse than one that does not run.

make fmt lint test helm-lint is the gate. It mirrors .gitea/workflows/ci.yaml step for step, so a green run here means a green pipeline — with one deliberate exception: make test adds -race, which CI does not. The sweeper, the notifier goroutine and the dead man's switch sweep all run concurrently against the same database, and a race between them would surface as a flaky incident in production rather than as a red build.

The web UI lives in internal/web/static/ as plain HTML, CSS and ES modules, embedded into the binary with go:embed. It has no build step and no npm, so editing a file and restarting the server is the whole loop.

Releasing

push or PR         → ci.yaml       gofmt, go vet, go test -race
                                   govulncheck, gitleaks
                                   helm lint + render
push tag vX.Y.Z    → release.yaml  the same gate, then publish:
                                     git.ryuvia.com/niklas/terdut-server:vX.Y.Z
                                     oci://git.ryuvia.com/niklas/terdut-server  X.Y.Z
                                   then trivy-scan the pushed image
PR to Ryuvia/charts →              bump the wrapper chart to X.Y.Z; on merge
                                   Flux reconciles and the release rolls out

Both artifacts go to the personal Gitea namespace rather than ryuvia, because Gitea scopes package visibility to the owner with no per-package override — so ryuvia/* is private because the org is. Publishing to niklas keeps them anonymously pullable, which is why no pull secret is needed in the cluster. Same reasoning, and the same choice, as riksdata and rd-web.

Saying "Release" runs all three rows: the release skill commits, pushes, tags, waits for the pipeline, and opens the Ryuvia/charts PR, stopping before the merge. See ~/.claude/skills/release/, or .release.conf here for this repo's part of it.

The chart is published only from the tag, by the chart job. There used to be a second publisher on every charts/** push to main, and the two raced for the same chart version with different answers — chart 0.9.0 went out reading appVersion: "latest" that way. One publisher, triggered by the tag (766f439). The cost is that a chart-only change has no version of its own and rides the next app tag.

Both workflows are thin drivers over the Makefile: ci.yaml runs make fmt lint test and make helm-lint, release.yaml adds make binaries, make push, make helm-package and make helm-push. That is deliberate — it is what makes a green local gate and a green pipeline the same code rather than two descriptions of it, and it is how riksdata and rd-web have always worked.

make push builds and pushes in one step, unlike those two, because the image is linux/amd64,linux/arm64 and buildx cannot load a multi-platform result into the local image store. make build stays single-platform and local-only. Both refuse VERSION=dev: publishing is one command, so it is also one command to run by accident. Publishing happens by pushing a tag.

Two things the release process needs to know about this repo:

• The image scan runs after publishing, like riksdata's and rd-web's: trivy cannot read a locally built image on this runner, so it pulls the pushed one. A red scan-image means do not bump the wrapper chart to that version — it cannot unpublish anything. The image is FROM scratch, so trivy sees exactly one target, the Go binary and its module graph.
• The wrapper chart's values.yaml has two tag: lines — the app image and the python backup sidecar — so chart-bump is given --image to say which one moves. The sidecar is on its way out with SQLite: once the wrapper chart drops it and declares a postgresql CR instead, there is one tag: line again, and --image becomes belt and braces.

The wrapper chart must have its own version: bumped in the same commit. Flux reconciles with reconcileStrategy: ChartVersion, so a chart whose version did not change produces no new artifact and the change is never deployed — with no error anywhere.