terdut-server/README.md

# Terminal Duty (terdut-server)

Incident management server for teams using Prometheus Alertmanager.

- Receives Alertmanager webhooks directly — no adapter needed
- Turns alerts into **incidents**, correlated by Alertmanager's own `groupKey`
- Incident workflow: acknowledge, assign, snooze, note, resolve, with a full timeline
- On-call schedule management, with new incidents auto-assigned to whoever is on call
- Alert and incident statistics, including MTTA and MTTR
- Web UI for phones and desktops, served by the same binary
- REST API with per-user API key authentication
- Single binary plus a Postgres — straightforward to self-host

---

## Quick start

**Prerequisites:** Go 1.21+

```bash
git clone https://git.ryuvia.com/niklas/terdut-server
cd terdut-server
go run ./cmd/terdut
```

The server starts on `:8080` with a `terdut.db` file in the working directory.

### Create the first user

```bash
curl -X POST http://localhost:8080/api/bootstrap \
  -H "Content-Type: application/json" \
  -d '{"username": "admin", "email": "admin@example.com", "password": "<at least 10 characters>"}'
```

Save the `api_key.key` value from the response — it is shown **once only**. The
`password` is optional and is what signs you in to the [web UI](#web-ui).

Use it as a bearer token for all subsequent requests:

```bash
export KEY=<your-key>
curl -H "Authorization: Bearer $KEY" http://localhost:8080/api/users
```

### Web UI

The server serves a web UI at `/`: the incident queue, each incident's alerts
and timeline with every action (acknowledge, assign, snooze, note, resolve,
archive), who is on call, the alert feed, and changing your own password. It is
built for a phone first. On a phone it has a bottom tab bar and a sticky action
bar, it follows the system's dark mode, and it can be added to the home screen.
From 900px wide it switches to a sidebar with the queue and the incident side by
side. Schedule editing, statistics and user management remain in
[terdut-tui](https://github.com/yeniklas/terdut-tui) for now.

You sign in with a username and password. Users have no password until one is
set, and a user without one can only use API keys:

```bash
# an admin sets someone's first password with their API key
curl -X PUT http://localhost:8080/api/users/2/password \
  -H "Authorization: Bearer $KEY" -H "Content-Type: application/json" \
  -d '{"password": "<at least 10 characters>"}'
```

After that, users change it themselves under *Account*. Changing your own
password requires the current one.

How a browser stays signed in:

- A successful login sets an `HttpOnly`, `SameSite=Lax` session cookie. It lasts
  30 days and slides forward while it is used, so an on-call phone stays signed
  in.
- The cookie is marked `Secure` when `TERDUT_PUBLIC_URL` starts with `https://`,
  so set it to the HTTPS address. TLS terminates at the gateway and the server
  itself only ever sees plain HTTP.
- Requests authenticated by the cookie are checked for cross-origin use (Go's
  `http.CrossOriginProtection`). That is the CSRF guard. Bearer-key clients are
  not affected.
- Setting a password signs that user out everywhere else.
- Ten failed logins for one username within 15 minutes lock that username for
  the rest of the window.

With `TERDUT_PUBLIC_URL` set, tapping a push notification opens the incident in
the web UI (`/incidents/{id}`).

### Docker

```bash
docker build -t terdut-server .
docker run -p 8080:8080 \
  -e TERDUT_DB_DSN='postgres://terdut:secret@host.docker.internal:5432/terdut?sslmode=disable' \
  terdut-server
```

The server creates its own schema on startup and needs a reachable Postgres; it stores nothing on
disk, so there is no volume to mount.

### Kubernetes

A Helm chart is published from this repository as an OCI artifact, versioned in lockstep
with the app — chart `x.y.z` is always app `vx.y.z`:

```bash
helm upgrade --install terdut-server oci://git.ryuvia.com/niklas/terdut-server \
  --version 0.9.2 \
  --namespace terdut-server --create-namespace \
  --set networking.hostname=terdut.example.com
```

The chart expects a [Gateway API](https://gateway-api.sigs.k8s.io/) Gateway named `envoy-main` in
the `envoy-gateway-system` namespace to already exist — it renders an `HTTPRoute` against it rather
than an `Ingress`. TLS is terminated at the gateway, so the server itself never sees a certificate.

| Value | Default | Description |
|---|---|---|
| `networking.hostname` | `terdut.example.com` | Hostname the `HTTPRoute` serves |
| `networking.listener` | `""` | Gateway listener (`sectionName`) to bind to. Empty attaches to every matching listener, **including plaintext HTTP** — set it to the HTTPS listener's name to serve TLS only |
| `networking.servicePort` | `8080` | Port the route forwards to; keep in sync with `service.port` |
| `bootstrap.enabled` | `true` | Runs a post-install hook that creates the first user and stores its API key in the `<release>-admin-key` Secret. Already-bootstrapped servers are left alone |
| `database.dsn` | `""` | **Required.** Postgres DSN, with no password in it. The chart provisions no database |
| `database.passwordSecret.name` | `""` | Secret supplying `PGPASSWORD`. With the Zalando postgres operator, the Secret it generates for the role |
| `database.passwordSecret.key` | `password` | Key within that Secret |

The API key travels in an `Authorization: Bearer` header, so set `networking.listener` whenever the
hostname is reachable outside a trusted network.

#### The database

The chart provisions no database: it takes a DSN and expects a Postgres that already exists. In this
cluster the wrapper chart declares an `acid.zalan.do/v1 postgresql` CR; anywhere else, any reachable
Postgres 14+ will do.

The DSN carries no password. pgx falls back to libpq's environment variables for whatever the DSN
leaves out, so the password arrives as `PGPASSWORD` from a Secret and never appears in values, in
the rendered manifest or in `kubectl describe pod`. With the postgres operator that Secret is the
one it generates for the role, so a rebuild mints a new password with nothing to keep in sync —
the same wiring miniflux uses.

The server migrates its own schema on startup, so a new database only has to exist and be writable.

#### Backups

Postgres is backed up where it runs, not from here. The database pod carries a
[k8up](https://k8up.io/) `k8up.io/backupcommand` annotation that streams a `pg_dump`, the same way
gitea and immich do in this cluster.

This used to be the app's problem: the SQLite database lived on a PVC beside the server, the image
is `FROM scratch` with no interpreter to dump it, and WAL mode makes a file-level copy of the volume
non-crash-consistent — so the chart shipped an idle `python:*-alpine` sidecar purely to give k8up
somewhere to exec. The sidecar, the PVC and the `backupSidecar` values are all gone.

---

## Configuration

Two kinds of setting, split by who changes them and how often.

**Where the server is plugged in** stays in the environment: the listen address,
the database DSN, the ntfy URL and token, the public URL. They are needed before
the database is open, and two of them are credentials.

**How the server behaves** lives in the database and is edited by an
administrator in the web UI or through `PUT /api/admin/settings`, taking effect
on the next sweep rather than at the next restart. The variables below marked
**seed** are the value each of those starts from: written once, on first start,
and never overwritten afterwards — a redeploy cannot put a chart's default back
over an administrator's edit.

| Variable | Default | Description |
|---|---|---|
| `TERDUT_ADDR` | `:8080` | TCP address to listen on |
| `TERDUT_DB_DSN` | — | **Required.** Postgres connection string, e.g. `postgres://terdut:secret@localhost:5432/terdut?sslmode=require` |
| `TERDUT_ARCHIVE_AFTER` | `168h` (7d) | **seed.** How long a resolved alert or incident stays in the default list before being auto-archived |
| `TERDUT_STALE_AFTER` | `6h` | **seed.** How long a firing alert may go without a refreshing webhook before it is treated as resolved — **must exceed your Alertmanager `repeat_interval`** |
| `TERDUT_DEADMAN_MATCHERS` | `alertname=Watchdog` | The **default** matchers a team starts with — switches are per team now, and this seeds teams that have no configuration of their own. `;` separates matchers, `,` the label conditions within one, `=` is exact equality. Every matcher must name an `alertname` |
| `TERDUT_DEADMAN_TIMEOUT` | `15m` | How long a heartbeat may go unheard before its switch is declared dead — **must be shorter than the `repeat_interval` of the route carrying it**. `0` disables dead man's switch handling |
| `TERDUT_DEADMAN_SEVERITY` | `critical` | Severity a dead man's switch incident opens at |
| `TERDUT_NTFY_URL` | — | ntfy server to publish push notifications to. Empty disables notifications entirely |
| `TERDUT_NTFY_TOKEN` | — | Bearer token for an access-controlled ntfy |
| `TERDUT_NTFY_FALLBACK_TOPIC` | — | Topic used when nobody is on call |
| `TERDUT_PUBLIC_URL` | — | Base URL a phone uses to reach this server: the notification's link into the web UI, its Acknowledge button, and whether the session cookie is `Secure` |
| `TERDUT_NOTIFY_REPEAT` | `15m` | **seed.** How long an incident may sit unacknowledged before it is paged again. `0` notifies once and never repeats |

Durations use Go syntax (`30m`, `12h`, `168h`). An unparseable value falls back to the default.

Note that `TERDUT_STALE_AFTER` and `TERDUT_DEADMAN_TIMEOUT` point in opposite directions. Staleness
is a generous grace period around a `repeat_interval` you do not control; a dead man's switch is a
deadline you set deliberately, and the heartbeat's route is configured to beat faster than it.

In the Helm chart the two sweeper durations are set via `sweeper.staleAfter` and `sweeper.archiveAfter`, dead man's switches via the `deadman.*` values, and notifications via the `notify.*` values.

---

## Alertmanager configuration

Alerts arrive on a team's **integration key**, which says both that the sender
may post and which team the alerts belong to. Mint one as an owner of the team:

```bash
curl -X POST https://terdut.example.com/api/teams/1/integrations \
  -H "Authorization: Bearer $TERDUT_API_KEY" \
  -H 'Content-Type: application/json' \
  -d '{"name":"prod alertmanager"}'
```

The response carries the key and the full URL **once**; only a SHA-256 hash is
stored. Put it in your `alertmanager.yml`:

```yaml
receivers:
  - name: terdut
    webhook_configs:
      - url: http://terdut-server:8080/api/integrations/<key>/alertmanager
        send_resolved: true

route:
  receiver: terdut
```

The whole URL is a credential, so treat it like one. Alertmanager 0.26 and
later can read it from a file with `url_file:` instead, which keeps it out of
your configuration repository:

```yaml
      - url_file: /etc/alertmanager/secrets/terdut-webhook-url/url
        send_resolved: true
```

The webhook endpoint requires no authentication.

If you use the [dead man's switch](#dead-mans-switch) — and the default configuration does — give
the heartbeat a route of its own, because the deadline is only as tight as the interval feeding it:

```yaml
route:
  receiver: terdut
  repeat_interval: 4h
  routes:
    - matchers: [ 'alertname = "Watchdog"' ]
      receiver: terdut
      group_wait: 0s
      group_interval: 1m
      repeat_interval: 1m
```

That delivers a heartbeat every **2 minutes**, not every minute. Alertmanager only reconsiders a
group every `group_interval`, and at exactly one elapsed interval `repeat_interval` has not *quite*
passed, so the send slips to the next tick — equal values give 2×. Two minutes against the 15 minute
default is seven heartbeats per window, which is the point; use `group_interval: 30s` if you want
the numbers to mean what they say.

kube-prometheus-stack users get the `Watchdog` alert (`expr: vector(1)`) for free; it just needs
routing to terdut rather than to `null`.

---

## Alerts and incidents

There are two objects, and the difference between them is the whole design.

**An alert is Alertmanager's record.** It has two states, `firing` and
`resolved`, one row per fingerprint, and no human ever writes to it. The API
exposes alerts read-only.

**An incident is the work item.** It goes `triggered → acknowledged → resolved`,
carries an assignee, a snooze, notes and a timeline, and is the only thing people
act on. Many alerts belong to one incident.

### Correlation uses Alertmanager's `groupKey`

Alertmanager has already grouped alerts according to the `group_by` routing tree
you configured, and it sends the resulting `groupKey` and `groupLabels` on every
webhook. Incidents adopt that answer rather than re-grouping alerts a second
time — if you want different correlation, change `group_by` in
`alertmanager.yml` and terdut follows.

At most one incident is open per `groupKey` at a time. Alerts firing in a group
that already has an open incident join it. The incident's `severity` is a
high-water mark — the highest `severity` label any of its alerts has carried — so
an incident that hit `critical` still reads as critical after the critical alert
clears.

### An incident opens only on a new occurrence

An incident opens when an alert **transitions into firing**: a fingerprint that
was never seen, an alert with a newer `startsAt`, or a resolved alert that
started again. The unchanged firing notifications Alertmanager re-sends every
`repeat_interval` are none of those, and open nothing.

This is what makes closing an incident by hand mean something. Without the rule,
`POST /api/incidents/{id}/resolve` would be undone by the next re-send of an
alert that never stopped firing.

### Leaving the open state

- **Automatically**, once every alert under the incident has stopped firing —
  whether by a resolved webhook or by the sweeper's
  [stale-alert expiry](#stale-alert-expiry). The incident gets
  `"resolution_source": "alerts"`.
- **By hand**, via `POST /api/incidents/{id}/resolve`
  (`"resolution_source": "manual"`). This is **terminal**: a later occurrence in
  that group opens a *new* incident rather than reopening this one. If the alert
  underneath never stops firing, the incident stays closed — that is what
  resolving by hand asserts.
- **On recovery**, for a [dead man's switch](#dead-mans-switch) incident whose
  heartbeat started arriving again (`"resolution_source": "recovered"`). These
  incidents have no member alerts, so the automatic cascade above cannot reach
  them.

To quieten an incident you expect to come back, snooze it instead
(`POST /api/incidents/{id}/snooze`). A snooze hides the incident from the default
list without closing it, and expires by simply falling into the past.

### On-call assignment

A new incident is assigned to whoever holds today's schedule entry at the moment
it opens (`GET /api/schedule/current`). If nobody is scheduled it opens
unassigned. Reassign with `POST /api/incidents/{id}/assign`.

One person holds a given day, so `POST /api/schedule` refuses a date somebody
already has: taking a shift off the person expecting to be paged for it should
not be something a plain call does by accident. Pass `"replace": true` to take
them anyway. Either way the whole request is one transaction — a week where some
days are free and some are taken moves as a unit, and a failure leaves the rota
exactly as it was rather than with a hole in it.

### Push notifications

With `TERDUT_NTFY_URL` set, an incident that opens is pushed to the on-call
person's phone through [ntfy](https://ntfy.sh). Set each user's topic with
`PUT /api/users/{id}/notify`; a user with no topic falls back to
`TERDUT_NTFY_FALLBACK_TOPIC`, as does an incident that opens with nobody on call.
If neither yields a topic, nothing is queued.

Three things get pushed:

- **triggered** — an incident opened. Priority follows severity (`critical` maps
  to ntfy's max priority, the one that overrides the phone's quiet settings).
- **reminder** — the incident is still `triggered` after `TERDUT_NOTIFY_REPEAT`.
  Repeats until somebody acts. Acknowledging, snoozing, resolving or archiving
  all stop it — snooze is the mute button.
- **resolved** — every alert under the incident stopped firing. Only sent to
  whoever was paged in the first place, and only for the automatic cascade:
  resolving by hand pushes nothing, since the person who did it already knows.

Notifications carry an **Acknowledge** button that acknowledges the incident
without opening anything. It POSTs to `/api/notify/ack/{token}`, an
unauthenticated route authorised by the 256-bit token in its path — minted fresh
per notification, scoped to one incident and one action, and valid for 24 hours.
A real API key is never put in a notification, because the message is stored on
the ntfy server and cached on the device.

The token is **not** consumed by use. Acknowledging is idempotent, so a token
stays valid for its full 24 hours and a second tap is a no-op that reports the
incident's current state rather than an error — which is what you want when a
tap is retried on a flaky mobile connection. What bounds it is scope, not a use
count: one incident, one action, one day. Expired tokens are purged by the
sweeper.

Two consequences worth planning for:

- `/api/notify/ack/{token}` **must stay publicly reachable**, or the button will
  not work when the responder is off your network.
- Notifications sent to the fallback topic carry **no** Acknowledge button. The
  topic is shared, and a button on it would let any subscriber acknowledge as
  somebody else.

Delivery is a queue, not an inline call: the webhook writes a row and a
background notifier sends it within 30 seconds, retrying with exponential
backoff up to 8 attempts. Nothing about ingestion blocks on ntfy being reachable.

Every delivery is recorded on the incident's timeline: a `notified` event once
ntfy accepts the publish, and a `notify_failed` event when a notification
exhausts its retries. Written from the result rather than at enqueue, so the
timeline says what actually happened — and a page that never landed is visible
instead of looking the same as one that did.

### Stale alert expiry

A resolved webhook is the only signal that an alert has stopped firing, so a
notification that is dropped, silenced, or lost to a restart would otherwise pin
that alert as firing forever. A background sweeper resolves firing alerts that
Alertmanager has stopped refreshing, using either signal:

- the `endsAt` watermark on the last notification has passed, or
- no webhook has refreshed the alert within `TERDUT_STALE_AFTER`.

Alertmanager re-sends firing notifications every `repeat_interval`, which is what
keeps a live alert fresh — so `TERDUT_STALE_AFTER` must be comfortably larger
than your `repeat_interval` (default 4h), or live alerts will be resolved
prematurely. Alerts resolved this way are marked `"resolution_source": "expiry"`
to distinguish them from a real Alertmanager resolve (`"alertmanager"`).

An expiry cascades: once it leaves an incident with nothing firing under it, the
incident resolves too, in the same sweep.

### Dead man's switch

Everything above assumes alerts arrive. If Prometheus stops evaluating, or
Alertmanager cannot reach this server, nothing arrives — and silence looks
exactly like everything being fine. A dead man's switch inverts the handling for
one designated alert so that silence is the signal:

- **receiving** it opens no incident, and
- the **absence** of it does.

kube-prometheus-stack already ships the alert for this. `Watchdog` is
`expr: vector(1)`, so it fires permanently and is re-sent forever; it is worth
nothing unless something downstream notices it stop. That is what
`TERDUT_DEADMAN_MATCHERS` defaults to.

**Switches belong to a team**, which decides which of its own alerts are
heartbeats and how long a silence has to last. An owner sets them through
`PUT /api/teams/{teamID}/deadman`; a missed heartbeat opens an incident in the
team whose integration received it.

The environment variables are the starting point, not the setting: at startup
every team **without** a configuration of its own is given one from them, and an
owner's later edit is never overwritten by a redeploy. A team created after
that starts watching nothing until its owner says otherwise — inheriting an
install-wide heartbeat would page a new team about a source it has never heard
of.

A matcher is a set of exact label conditions, one of which must be the
`alertname`, in the same format the environment variable uses:

```
alertname=Watchdog,cluster=prod; alertname=EdgeHeartbeat
```

**The unit of monitoring is the fingerprint, not the alert name.** Two clusters
sending the same `Watchdog` are two independent switches, so a healthy one can
never mask a dead one.

#### The lifecycle

A switch is **dormant** until its first heartbeat arrives. A configured matcher
that has never been heard from opens nothing, so a fresh deploy or a restored
database does not page. It also means a matcher that never matches anything is
silently inert — check the startup log line, which lists the matchers that
survived parsing.

Once armed, the sweeper declares it **dead** when either the heartbeat has not
been refreshed within `TERDUT_DEADMAN_TIMEOUT`, or Alertmanager explicitly
resolved it — the sender saying the heartbeat stopped needs no further waiting.
That opens an incident at `TERDUT_DEADMAN_SEVERITY`, assigned and paged like any
other, and marks the heartbeat alert `"resolution_source": "deadman"` so the
alert list stops claiming a dead switch is firing.

It **recovers** when the heartbeat starts arriving again: the incident resolves
with `"resolution_source": "recovered"` and the all-clear goes to whoever was
paged.

Resolving the incident by hand sticks, the same way it does for an alert-backed
one. While the switch stays silent nothing new opens — so a decommissioned
source is a one-time page rather than a nag. The switch **re-arms** on the next
heartbeat: come back and die again, and that is a new incident.

#### Two things to know

`TERDUT_DEADMAN_TIMEOUT` must be **shorter** than the `repeat_interval` of the
route carrying the heartbeat, which is the exact opposite of
`TERDUT_STALE_AFTER`. Inheriting a default `repeat_interval` of 4h gives you a
switch that takes four hours to notice anything, so give the heartbeat
[its own route](#alertmanager-configuration). Matched alerts are exempt from
stale-alert expiry — a heartbeat answers to its own timeout and nothing else.

A dead man's switch incident has **no member alerts**:
`GET /api/incidents/{id}/alerts` returns an empty list. There is no alert
describing the problem, because the problem is that no alert arrived. What
happened is on the timeline instead, as a `deadman_silent` event carrying the age
of the last heartbeat, and the heartbeat's labels are on the incident's
`group_labels`.

---

## API reference

### Authentication

All endpoints except `/api/bootstrap`, `/api/integrations/{key}/alertmanager`,
`/api/notify/ack/{token}`, `/api/login` and `/api/logout` require either an API key:

```
Authorization: Bearer <api-key>
```

or the web UI's session cookie. A request that carries an `Authorization` header
is judged on that header alone.

Two kinds of user exist. An **administrator** manages accounts: creating and
deleting users, setting anybody's password, minting keys for anybody, and
granting the flag itself. Everybody else works incidents — acknowledging,
assigning, snoozing, resolving, noting — and manages their own account and
nobody else's. An API key carries exactly the rights of the user it belongs to.

The first user, from `/api/bootstrap`, is an administrator. Users created
afterwards are not, until an administrator says so. An install always keeps at
least one: the last administrator can be neither deleted nor demoted, and
nobody can delete or demote themselves.

Endpoints that require the flag answer `403` with
`{"error":"administrator access required"}`.

**Teams** are the unit of tenancy, and are a separate axis from the administrator
flag. A team owns its incidents, alerts, schedule and integrations, and a user
sees exactly the teams they belong to — an administrator is not implicitly in
every team, because administration is about accounts, not about reading other
people's incidents. Within a team an **owner** configures it (schedule,
integrations, membership) and a **member** works its incidents.

Anything belonging to a team you are not in answers `404`, not `403`: whether an
incident exists is itself something only its team should learn.

| Method | Path | Description |
|---|---|---|
| `POST` | `/api/login` | `{"username","password"}` → sets the session cookie, returns `{user, has_password}`. `429` after too many failures |
| `POST` | `/api/logout` | Ends the session and clears the cookie |
| `GET` | `/api/me` | The caller: `{user, has_password}` |

### Users

| Method | Path | Description |
|---|---|---|
**admin** marks an endpoint that requires the administrator flag; **self or
admin** marks one you may use on your own account and an administrator may use
on anybody's.

| Method | Path | Who | Description |
|---|---|---|---|
| `POST` | `/api/bootstrap` | — | Create first user + API key `{"username","email","password"?}` (only works on empty DB). The user is an administrator |
| `GET` | `/api/users` | any | List users. Open to everybody: the queue's assignment control and the schedule both have to name people |
| `POST` | `/api/users` | **admin** | Create user `{"username","email"}`. Not an administrator |
| `DELETE` | `/api/users/{id}` | **admin** | Delete user (cascades to keys). `409` for yourself or the last administrator |
| `PUT` | `/api/users/{id}/admin` | **admin** | Grant or revoke the administrator flag `{"is_admin"}`. `409` for yourself or the last administrator |
| `PUT` | `/api/users/{id}/disabled` | **admin** | Take an account out of use, or put it back `{"disabled"}`. `409` for yourself or the last administrator |
| `PUT` | `/api/users/{id}/notify` | self or admin | Set push notification target `{"ntfy_topic"}` — empty string clears it |
| `PUT` | `/api/users/{id}/password` | self or admin | Set web UI password `{"password","current_password"}`. `current_password` is required only when changing your own existing password. Ends the user's other sessions |
| `POST` | `/api/users/{id}/api-keys` | self or admin | Issue API key `{"name"}` — key shown once |
| `DELETE` | `/api/users/{id}/api-keys/{keyID}` | self or admin | Revoke API key |

### Administration

| Method | Path | Who | Description |
|---|---|---|---|
| `GET` | `/api/admin/teams` | **admin** | Every team on the server, with its member and open-incident counts. `/api/teams` answers "what am I in"; this answers "what is there" |
| `GET` | `/api/admin/settings` | **admin** | The editable settings with their bounds, plus the environment-configured ones, read-only. Never credentials |
| `PUT` | `/api/admin/settings` | **admin** | Change one or more `{"key": seconds}`. `400` for an unknown key or a value outside its bounds |

### Alert ingestion

Alerts arrive on a team's integration key. The key is both the credential and the
routing: it says that the sender may post, and which team the alerts belong to.
Create one with `POST /api/teams/{teamID}/integrations`, which returns the key
and the full URL once and stores only a SHA-256 hash.

| Method | Path | Description |
|---|---|---|
| `POST` | `/api/integrations/{key}/alertmanager` | Alertmanager v4 webhook receiver for the key's team. `401` for an unknown key |

This is the only way in. The pre-teams `POST /api/alertmanager/webhook` took no
credential at all — anything able to reach the port could open an incident —
and was removed in v0.13.0 once senders had moved onto keys.

### Teams

| Method | Path | Who | Description |
|---|---|---|---|
| `GET` | `/api/teams` | any | The caller's own teams, each with their role |
| `POST` | `/api/teams` | any | Create a team `{"name"}`; the creator becomes its first owner |
| `PUT` | `/api/teams/{teamID}` | **owner** | Rename it `{"name"}`. `409` if the name is taken |
| `DELETE` | `/api/teams/{teamID}` | **owner** | Delete a team and everything under it. `409` while it has open incidents |
| `GET` | `/api/teams/{teamID}/members` | member | Who is in the team |
| `POST` | `/api/teams/{teamID}/members` | **owner** | Add a member, or change their role `{"user_id","role"}` |
| `DELETE` | `/api/teams/{teamID}/members/{userID}` | **owner** | Remove a member. `409` for the last owner |
| `GET` | `/api/teams/{teamID}/integrations` | member | List integrations. Never returns keys |
| `POST` | `/api/teams/{teamID}/integrations` | **owner** | Mint an integration `{"name","kind"}` — key and URL shown once |
| `DELETE` | `/api/teams/{teamID}/integrations/{integrationID}` | **owner** | Revoke an integration |
| `GET` | `/api/teams/{teamID}/deadman` | member | The team's [dead man's switch](#dead-mans-switch) configuration `{matchers, timeout_seconds, severity}` |
| `PUT` | `/api/teams/{teamID}/deadman` | **owner** | Replace it. `400` when no matcher names an `alertname`, because a switch that silently watches nothing is the failure this feature exists to prevent |

### Notifications

| Method | Path | Description |
|---|---|---|
| `POST` | `/api/notify/ack/{token}` | Acknowledge an incident from a push notification's Acknowledge button. No auth: the token in the path is the credential — one incident, one action, 24 hours, idempotent. Must stay publicly reachable |

### Incidents

| Method | Path | Description |
|---|---|---|
| `GET` | `/api/incidents` | List incidents. Filters: `?status=triggered\|acknowledged\|resolved`, `?severity=`, `?assigned_to=<user id>`, `?archived=true`, `?snoozed=true`, `?from=YYYY-MM-DD`, `?to=YYYY-MM-DD`, `?sort=severity`, `?limit=` (default 50, max 500) |
| `GET` | `/api/incidents/{id}` | Get single incident, with its alerts inline |
| `GET` | `/api/incidents/{id}/alerts` | Alerts under this incident |
| `GET` | `/api/incidents/{id}/timeline` | Full event history, chronological |
| `POST` | `/api/incidents/{id}/acknowledge` | Acknowledge (stamps authed user + time) |
| `DELETE` | `/api/incidents/{id}/acknowledge` | Clear acknowledgement, back to `triggered` |
| `POST` | `/api/incidents/{id}/resolve` | Close by hand — **terminal**, see above |
| `POST` | `/api/incidents/{id}/assign` | Reassign `{"user_id"}` |
| `POST` | `/api/incidents/{id}/snooze` | Hide until `{"until": RFC3339}` or `{"duration": "2h"}` |
| `DELETE` | `/api/incidents/{id}/snooze` | Un-snooze |
| `POST` | `/api/incidents/{id}/archive` | Archive (hides from the default list) |
| `DELETE` | `/api/incidents/{id}/archive` | Un-archive |
| `POST` | `/api/incidents/{id}/notes` | Add a note `{"content"}` |
| `DELETE` | `/api/incidents/{id}/notes/{eventID}` | Delete own note |

With no `?status=` filter, `GET /api/incidents` returns **open** incidents only —
the queue an on-call person wants. Currently snoozed and archived incidents are
excluded unless asked for. Actions that only make sense on an open incident
return `409` once it is resolved.

Notes are ordinary timeline events of type `note`; only they are deletable, and
only by their author. The rest of the timeline is a record of what happened.

#### The incident object

| Field | Type | Notes |
|---|---|---|
| `id` | integer | Server-assigned |
| `group_key` | string | Alertmanager's `groupKey` — opaque, treat as an identifier |
| `title` | string | Rendered from `groupLabels` |
| `group_labels` | object | String→string, as sent by Alertmanager |
| `status` | string | `"triggered"`, `"acknowledged"` or `"resolved"` |
| `severity` | string | *optional* — high-water mark across the incident's alerts; never lowered |
| `triggered_at` | timestamp | When the incident opened |
| `acknowledged_by_id` / `acknowledged_by` / `acknowledged_at` | | *optional* — user id, username, time |
| `assigned_to_id` / `assigned_to` | | *optional* — user id, username |
| `snoozed_until` | timestamp | *optional* — a value in the past reads as not snoozed |
| `resolved_at` | timestamp | *optional* |
| `resolution_source` | string | *optional* — `"alerts"`, `"manual"` or `"recovered"` |
| `archived_at` | timestamp | *optional* |
| `alerts` | array | Only on `GET /api/incidents/{id}` |

Treat `resolution_source` as an open set, as with the alert field of the same
name: degrade unknown values to "resolved, reason unknown".

#### The timeline event object

| Field | Type | Notes |
|---|---|---|
| `id` | integer | |
| `incident_id` | integer | |
| `type` | string | See below — treat as an open set |
| `user_id` / `username` | | *optional* — absent when the server acted rather than a person |
| `alert_id` | integer | *optional* — the alert an `alert_added` / `alert_resolved` event refers to |
| `detail` | string | *optional* — the note body, the snooze deadline, etc. |
| `created_at` | timestamp | |

Types written today: `triggered`, `alert_added`, `alert_resolved`,
`acknowledged`, `unacknowledged`, `assigned`, `snoozed`, `unsnoozed`, `resolved`,
`note`, `notified`, `notify_failed`, `deadman_silent`. On an `assigned` event
`user_id` is the **assignee**, not the actor. New types may be added; render
unknown ones generically rather than dropping them.

On `notified` and `notify_failed`, `detail` carries the notification kind
(`triggered` | `reminder` | `resolved`), and on a failure the reason after it.
`user_id` is who was paged — absent means the page went to the shared fallback
topic and so belongs to nobody. The topic itself is never written to the
timeline: it is a shared secret with the ntfy server, and every API key can read
this.

### Alerts

Alerts are read-only. Everything a person does happens on the incident.

| Method | Path | Description |
|---|---|---|
| `GET` | `/api/alerts` | List alerts. Filters: `?status=firing\|resolved`, `?name=`, `?incident_id=`, `?archived=true`, `?from=YYYY-MM-DD`, `?to=YYYY-MM-DD`, `?limit=` (default 50, max 500) |
| `GET` | `/api/alerts/{id}` | Get single alert |

Archived alerts are hidden from `GET /api/alerts` unless `?archived=true` is
passed; alert archiving is automatic housekeeping by the sweeper, not a user
action. Resolved alerts carry `resolution_source`: `"alertmanager"` for a real
resolved webhook, `"expiry"` when the sweeper inferred it (see
[Stale alert expiry](#stale-alert-expiry)), `"deadman"` for a heartbeat declared
dead (see [Dead man's switch](#dead-mans-switch)).

#### The alert object

Returned by `GET /api/alerts` (as an array) and `GET /api/alerts/{id}`.
Timestamps are RFC 3339 in UTC. Fields marked *optional* are omitted entirely
when unset, so clients must treat them as nullable.

| Field | Type | Notes |
|---|---|---|
| `id` | integer | Server-assigned; stable for the life of the row |
| `fingerprint` | string | Alertmanager's fingerprint — the upsert key |
| `name` | string | From the `alertname` label |
| `status` | string | `"firing"` or `"resolved"` |
| `labels` | object | String→string, as sent by Alertmanager |
| `annotations` | object | String→string, as sent by Alertmanager |
| `starts_at` | timestamp | When the alert instance began, **per Prometheus** |
| `ends_at` | timestamp | *optional* — absent while no end is known |
| `generator_url` | string | Link back to the originating Prometheus |
| `received_at` | timestamp | When the server last accepted a webhook for this alert — see below |
| `incident_id` | integer | *optional* — the most recent incident this alert belongs to |
| `resolution_source` | string | *optional* — `"alertmanager"`, `"expiry"` or `"deadman"` |
| `archived_at` | timestamp | *optional* — set while archived |

##### `received_at` is a liveness heartbeat

`starts_at` comes from Prometheus and **never changes** for the lifetime of an
alert instance. It says when the problem began, not whether it is still
happening — an alert that started twelve days ago looks identical whether
Alertmanager refreshed it a minute ago or went silent a week ago.

`received_at` is the field that answers "is this still live". It is set to the
server's clock on **every accepted webhook** for that fingerprint, including the
unchanged firing notifications Alertmanager re-sends every `repeat_interval`.
Clients may rely on this:

- **A firing alert whose `received_at` is advancing is still being refreshed.**
  Stale-dating it against `repeat_interval` is a valid liveness check, and it is
  what the built-in sweeper does (see
  [Stale alert expiry](#stale-alert-expiry)).
- **`received_at` tracks accepted payloads, not delivery attempts.** A retry
  that describes an older instance than the stored one is discarded, and a
  discarded payload does not move `received_at`.
- **It stops advancing once the alert resolves,** because Alertmanager stops
  re-sending. On an alert resolved by the sweeper
  (`"resolution_source": "expiry"`) it therefore marks the last time
  Alertmanager was actually heard from, which is earlier than `ends_at`.

`GET /api/alerts` is ordered by `received_at` descending — most recently
refreshed first — and the `?from=` / `?to=` filters on both the alert and stats
endpoints select on `received_at`, not `starts_at`.

##### `resolution_source` says how much to trust `ends_at`

An alert can leave the firing state two ways, and `resolution_source` records
which happened. Clients may rely on this:

- **Absent while firing.** It is set only on resolve, and a re-fire under the
  same fingerprint clears it again, so its presence always agrees with
  `"status": "resolved"`.
- **`"alertmanager"` — a real resolved webhook arrived.** `ends_at` is the end
  time Alertmanager reported. It is an observed value and can be displayed as
  fact.
- **`"expiry"` — the sweeper inferred the resolve** because Alertmanager stopped
  refreshing the alert (see [Stale alert expiry](#stale-alert-expiry)). Nothing
  ever reported an end, so **`ends_at` is approximate**: it is either the stale
  `endsAt` watermark from the last notification, or — when that notification
  carried none — the time the sweep ran, which lags the last real contact by up
  to `TERDUT_STALE_AFTER` plus a sweep interval. Treat it as "no later than",
  not as when the problem stopped.

  On these alerts `received_at` is the more truthful signal: it marks the last
  time Alertmanager was actually heard from. Surfacing the distinction is
  worthwhile, since `"expiry"` can also mean the alert is still firing and the
  notification path broke.

- **`"deadman"` — a heartbeat was declared dead** (see
  [Dead man's switch](#dead-mans-switch)). Like `"expiry"`, an inference from
  silence rather than an observed end, so `ends_at` is approximate — but a much
  tighter one, bounded by `TERDUT_DEADMAN_TIMEOUT`. It is also the one resolution
  a re-fire under the same `starts_at` can undo, since the switch coming back is
  exactly the evidence that the inference was wrong.

Treat the value as an open set and tolerate ones you do not recognise — new
sources may be added, and unknown values should degrade to "resolved, reason
unknown" rather than being rejected.

### On-call schedule

| Method | Path | Description |
|---|---|---|
Each team keeps its own rota, so two teams can have two different people on call
on the same day. The person taking a shift has to be in the team — paging
somebody who cannot open the incident is worse than paging nobody.

| Method | Path | Who | Description |
|---|---|---|---|
| `POST` | `/api/teams/{teamID}/schedule` | **owner** | Assign user to dates `{"user_id", "dates":["YYYY-MM-DD",...], "replace"}` — all-or-nothing |
| `GET` | `/api/teams/{teamID}/schedule` | member | List entries. Filters: `?from=YYYY-MM-DD`, `?to=YYYY-MM-DD` |
| `DELETE` | `/api/teams/{teamID}/schedule/{id}` | **owner** | Remove schedule entry |
| `GET` | `/api/schedule/current` | any | Who is on call today (UTC) in **every** team the caller is in — one entry per team, `[]` when nobody anywhere |

### Statistics

Every figure counts the caller's own teams only: a report that counted other
teams' incidents would leak their volume, and their alert names through the
top-alerts list, and would not be a number about the reader's work anyway.

All stat endpoints accept optional `?from=YYYY-MM-DD` and `?to=YYYY-MM-DD`, and exclude archived rows to match the default list views. Alert stats filter on `received_at`; incident stats filter on `triggered_at`.

| Method | Path | Description |
|---|---|---|
| `GET` | `/api/stats/incidents` | `{total, triggered, acknowledged, resolved, mtta_seconds, mttr_seconds}` |
| `GET` | `/api/stats/alerts` | `{total, firing, resolved}` counts |
| `GET` | `/api/stats/alerts/top` | Most frequent alert names. `?limit=` (default 10, max 100) |
| `GET` | `/api/stats/alerts/by-hour` | Count per hour-of-day (UTC), all 24 slots returned |
| `GET` | `/api/stats/alerts/by-day` | Count per day-of-week, all 7 slots with names returned |

`mtta_seconds` (time to acknowledge) and `mttr_seconds` (time to resolve) are
averages over incidents that have actually been acknowledged or resolved, and are
**null** until there are any — null means "no data", not zero.

---

## Upgrading to teams

Everything that existed before teams moves into one team called **Default**, and
every existing user becomes an owner of it. The upgrade is a no-op for the
people using it: the same queue, the same schedule, the same incidents, with a
name on them.

What changes, and will need attention:

- **Alert ingestion moved.** Mint a key with
  `POST /api/teams/{teamID}/integrations` and point Alertmanager at the URL it
  returns. In v0.12.0 the old `POST /api/alertmanager/webhook` still worked,
  deprecated, routing everything to the oldest team; **v0.13.0 removes it**, so
  upgrade straight from v0.11.x to v0.13.0 only after the senders are moved.
- **The schedule endpoints moved** under `/api/teams/{teamID}/schedule`, and
  editing the rota is now an owner's job. `GET /api/schedule/current` stayed
  where it was but now returns an **array** — one entry per team with somebody
  on call — instead of a single object or a 404. This is a breaking API change
  for anything that reads it, terdut-tui included.
- **Uniqueness is per team now.** Two teams can legitimately see the same alert
  fingerprint, the same Alertmanager groupKey, and put somebody on call on the
  same date.

**Dead man's switches moved too.** `TERDUT_DEADMAN_MATCHERS`, `_TIMEOUT` and
`_SEVERITY` are no longer the setting; they are the default each existing team
is seeded with at startup, after which an owner edits them per team through
`PUT /api/teams/{teamID}/deadman` and a redeploy never overwrites that.

Nothing else about an incident changes, and incidents never move between teams:
an alert belongs to whichever team's key it arrived on.

## Upgrading to roles

Before this release every authenticated caller could create and delete users,
set anybody's password and mint anybody's API keys. That is now the
administrator flag, and the migration **makes every existing user an
administrator** — they already held those powers, so nobody's access changes on
upgrade and demotion is a deliberate act afterwards. Promoting only the first
user would have silently stripped the rest, and could leave an install whose
only administrator is an account nobody has a password for.

Users created after the upgrade are not administrators. Hand the flag out with:

```bash
curl -X PUT https://terdut.example.com/api/users/7/admin \
  -H "Authorization: Bearer $TERDUT_API_KEY" \
  -H 'Content-Type: application/json' \
  -d '{"is_admin": true}'
```

Nothing in the API changed shape, so terdut-tui needs no new version — but a
non-administrator now gets `403` where a `200` used to come back.

## Upgrading from SQLite

Versions up to v0.10.2 stored everything in a SQLite file. From v0.11.1 the server needs
`TERDUT_DB_DSN` and keeps nothing on disk.

The copy was done by `scripts/sqlite-to-postgres.go`, which **was deleted in v0.13.0** along
with the SQLite driver it was the last user of. It is still in the history — check out the
`v0.12.0` tag to get it:

```bash
git show v0.12.0:scripts/sqlite-to-postgres.go > sqlite-to-postgres.go
```

The cutover is ordered, and the server must not be running while the copy happens: stop the
old version, let the new binary build the schema against an empty Postgres, run the script
with `-sqlite` and `-dsn`, then start the new version for good. On Kubernetes step three runs
as a Job with the same image against the PVC before it is removed.

The copy preserves every id, so incidents keep their numbers and the timeline, alert
membership, outbox and ack tokens all still point where they did. It refuses a target that
already has rows, so a second run cannot double-insert.

## Upgrading to incidents

The incidents release moves the workflow off alerts, which is a **breaking API
change**. These endpoints are gone:

| Removed | Replacement |
|---|---|
| `POST`/`DELETE` `/api/alerts/{id}/acknowledge` | `POST`/`DELETE` `/api/incidents/{id}/acknowledge` |
| `POST`/`DELETE` `/api/alerts/{id}/archive` | `POST`/`DELETE` `/api/incidents/{id}/archive` (alert archiving is now sweeper-only) |
| `GET`/`POST` `/api/alerts/{id}/comments` | `GET /api/incidents/{id}/timeline`, `POST /api/incidents/{id}/notes` |
| `DELETE /api/alerts/{id}/comments/{commentID}` | `DELETE /api/incidents/{id}/notes/{eventID}` |

The alert object also drops `acknowledged_by_id`, `acknowledged_by` and
`acknowledged_at`, and gains `incident_id`.

Migration `008_incidents.sql` runs automatically on start and preserves existing
data: every alert gets a backfilled incident carrying its acknowledgement, and
comments become timeline notes. Backfilled incidents have a `group_key` of
`backfill:<fingerprint>` — there is no historical `groupKey` to correlate on, so
they are one-per-alert rather than grouped.

Nothing about the two documented alert contracts changes: `received_at` is still
advanced on every accepted webhook, and `resolution_source` still means what it
did.

## Upgrading to dead man's switches

Dead man's switch handling is **on by default**, watching `alertname=Watchdog`
with a 15 minute timeout. If you already route `Watchdog` to this server, the
behaviour of that alert changes on upgrade, in both directions:

- it stops opening incidents when it arrives, and
- it starts opening one when it stops arriving.

**Check your `repeat_interval` before upgrading.** The switch pages whenever a
heartbeat has not been refreshed within `TERDUT_DEADMAN_TIMEOUT`, so a `Watchdog`
route inheriting a 4h or 12h `repeat_interval` will page constantly against the
15 minute default. Either give the heartbeat
[its own fast route](#alertmanager-configuration) — the point of the feature — or
set `TERDUT_DEADMAN_TIMEOUT` above your current `repeat_interval` until you have.
`TERDUT_DEADMAN_TIMEOUT=0` turns the whole thing off.

There is no migration and no schema change. An existing open incident from a
`Watchdog` that arrived under the old behaviour is unaffected; resolve it by hand.

---

## Development

```bash
make test-db         # start a local Postgres for the tests (podman or docker)
make test            # run all tests
go build ./...       # compile all packages
go run ./cmd/terdut  # run locally (needs TERDUT_DB_DSN)
```

The tests need a real Postgres, because the server does — there is no in-memory Postgres the
way there was an in-memory SQLite. `TERDUT_TEST_DSN` says where it is, `make test-db` starts
one on port 5433 and prints the DSN, and `make test-db-stop` removes it. Each test gets its
own schema on that server, so tests cannot see each other's rows. An unset `TERDUT_TEST_DSN`
fails the suite rather than skipping it: a run that quietly tests nothing is worse than one
that does not run.

`make fmt lint test helm-lint` is the gate. It mirrors `.gitea/workflows/ci.yaml` step for
step, so a green run here means a green pipeline — with one deliberate exception: `make test`
adds `-race`, which CI does not. The sweeper, the notifier goroutine and the dead man's switch
sweep all run concurrently against the same database, and a race between them would surface as
a flaky incident in production rather than as a red build.

The web UI lives in `internal/web/static/` as plain HTML, CSS and ES modules,
embedded into the binary with `go:embed`. It has no build step and no npm, so
editing a file and restarting the server is the whole loop.

## Releasing

```
push or PR         → ci.yaml       gofmt, go vet, go test -race
                                   govulncheck, gitleaks
                                   helm lint + render
push tag vX.Y.Z    → release.yaml  the same gate, then publish:
                                     git.ryuvia.com/niklas/terdut-server:vX.Y.Z
                                     oci://git.ryuvia.com/niklas/terdut-server  X.Y.Z
                                   then trivy-scan the pushed image
PR to Ryuvia/charts →              bump the wrapper chart to X.Y.Z; on merge
                                   Flux reconciles and the release rolls out
```

Both artifacts go to the **personal** Gitea namespace rather than `ryuvia`, because Gitea
scopes package visibility to the owner with no per-package override — so `ryuvia/*` is private
because the org is. Publishing to `niklas` keeps them anonymously pullable, which is why no
pull secret is needed in the cluster. Same reasoning, and the same choice, as riksdata and
rd-web.

Saying **"Release"** runs all three rows: the `release` skill commits, pushes, tags, waits for
the pipeline, and opens the `Ryuvia/charts` PR, stopping before the merge. See
`~/.claude/skills/release/`, or `.release.conf` here for this repo's part of it.

The chart is published **only** from the tag, by the `chart` job. There used to be a second
publisher on every `charts/**` push to main, and the two raced for the same chart version with
different answers — chart 0.9.0 went out reading `appVersion: "latest"` that way. One
publisher, triggered by the tag (`766f439`). The cost is that a chart-only change has no
version of its own and rides the next app tag.

Both workflows are thin drivers over the Makefile: `ci.yaml` runs `make fmt lint test` and
`make helm-lint`, `release.yaml` adds `make binaries`, `make push`, `make helm-package` and
`make helm-push`. That is deliberate — it is what makes a green local gate and a green
pipeline the same code rather than two descriptions of it, and it is how riksdata and rd-web
have always worked.

`make push` builds and pushes in one step, unlike those two, because the image is
`linux/amd64,linux/arm64` and buildx cannot load a multi-platform result into the local image
store. `make build` stays single-platform and local-only. Both refuse `VERSION=dev`:
publishing is one command, so it is also one command to run by accident. Publishing happens
by pushing a tag.

Two things the release process needs to know about this repo:

- **The image scan runs after publishing**, like riksdata's and rd-web's: trivy cannot read
  a locally built image on this runner, so it pulls the pushed one. A red `scan-image` means
  do not bump the wrapper chart to that version — it cannot unpublish anything. The image is
  `FROM scratch`, so trivy sees exactly one target, the Go binary and its module graph.
- **The wrapper chart's `values.yaml` has two `tag:` lines** — the app image and the python
  backup sidecar — so `chart-bump` is given `--image` to say which one moves. The sidecar is
  on its way out with SQLite: once the wrapper chart drops it and declares a `postgresql` CR
  instead, there is one `tag:` line again, and `--image` becomes belt and braces.

The wrapper chart must have **its own `version:` bumped in the same commit**. Flux reconciles
with `reconcileStrategy: ChartVersion`, so a chart whose version did not change produces no
new artifact and the change is never deployed — with no error anywhere.