diff --git a/CLAUDE.md b/CLAUDE.md index 278c78c..ef79927 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -12,7 +12,7 @@ Preconditions and the plan, without side effects: ``` Config is `.release.conf` here plus `make release-vars`. The process itself lives in -`~/.claude/skills/release/`; why it is shaped this way is in README.md §Releasing. +`~/.claude/skills/release/`; why it is shaped this way is in docs/development.md (Releasing). Three things about this repo specifically: diff --git a/README.md b/README.md index da37499..fc57704 100644 --- a/README.md +++ b/README.md @@ -1,31 +1,72 @@ # Terminal Duty (terdut-server) -Incident management server for teams using Prometheus Alertmanager. +Incident management for teams that already run Prometheus Alertmanager. Point Alertmanager at it, +and alerts become incidents that get assigned to whoever is on call, paged, escalated when nobody +answers, and tracked to resolution. One binary, one Postgres. -- Receives Alertmanager webhooks directly — no adapter needed -- Turns alerts into **incidents**, correlated by Alertmanager's own `groupKey` -- Incident workflow: acknowledge, assign, snooze, note, resolve, with a full timeline -- On-call schedule management, with new incidents auto-assigned to whoever is on call -- Alert and incident statistics, including MTTA and MTTR -- Web UI for phones and desktops, served by the same binary -- REST API with per-user API key authentication -- Single binary plus a Postgres — straightforward to self-host +![The incident queue with an incident open](./docs/images/queue-light.png) ---- +## Highlights + +- **Alertmanager-native.** Receives Alertmanager webhooks directly, with no adapter, and groups + alerts into incidents by Alertmanager's own `groupKey`. An incident opens on a new occurrence, + not on every re-send. [Details](./docs/incidents.md) +- **A real incident workflow.** Acknowledge, assign, snooze, add notes, resolve and archive, with a + full timeline of who did what and when. Several clusters can feed one team without + cross-talk. [Details](./docs/incidents.md) +- **On-call rota.** Each team keeps its own rota, and new incidents go to whoever is on call. + [Web UI](./docs/web-ui.md) +- **Pages that escalate.** Push notifications through [ntfy](https://ntfy.sh), with an + Acknowledge button right in the notification, and escalation ladders that move on to the next + level when nobody answers. [Notifications](./docs/notifications.md) · + [Escalation](./docs/escalation.md) +- **Notices when the alerts stop.** Dead man's switches turn the absence of a heartbeat such as + `Watchdog` into an incident. [Details](./docs/dead-mans-switch.md) +- **Teams.** Every team owns its queue, rota, escalation, alert sources and switches; people see + only the teams they belong to. +- **Stats.** Incident counts, mean time to acknowledge and resolve, and alert frequency by name, + hour and day. [Web UI](./docs/web-ui.md) +- **Single sign-on.** OpenID Connect with group-to-team and administrator mapping, and a device + flow so a terminal client can sign in through the browser. [Details](./docs/single-sign-on.md) +- **Built for the phone first.** The web UI is served by the same binary, follows the system's + dark mode, and can be added to the home screen. +- **An API for everything.** A REST API with API keys and service accounts for automation. + [Reference](./docs/api.md) +- **Easy to run.** A scratch container image, a Helm chart, and + [terdut-operator](https://git.ryuvia.com/niklas/terdut-operator) if you want teams and + escalation as Kubernetes objects. [Deployment](./docs/deployment.md) + +## Screenshots + +| | | +|---|---| +| ![An incident with a note on its timeline](./docs/images/incident-note.png) | ![The queue in dark mode](./docs/images/queue-dark.png) | +| **Work an incident**: acknowledge, assign, snooze, note, resolve | **Dark mode**, following the system | +| ![The on-call page](./docs/images/oncall.png) | ![An escalation ladder](./docs/images/team-escalation.png) | +| **On call**: who holds the pager now, and the week ahead | **Escalation**: who is paged next, and when | +| ![Statistics](./docs/images/stats.png) | ![The alert feed](./docs/images/alerts.png) | +| **Stats**: MTTA, MTTR and what fires most | **Alerts**: the raw feed behind the incidents | + +On a phone the queue and the incident page are the same interface, with a sticky action bar: + +

+ The queue on a phone + An incident on a phone +

## Quick start -**Prerequisites:** Go 1.21+ +You need Go 1.25+ and a Postgres 14+ the server can reach. ```bash git clone https://git.ryuvia.com/niklas/terdut-server cd terdut-server +export TERDUT_DB_DSN='postgres://terdut:secret@localhost:5432/terdut?sslmode=disable' go run ./cmd/terdut ``` -The server starts on `:8080` with a `terdut.db` file in the working directory. - -### Create the first user +The server creates its schema on startup and listens on `:8080`. Create the first user, an +administrator, while no user exists yet: ```bash curl -X POST http://localhost:8080/api/bootstrap \ @@ -33,1334 +74,36 @@ curl -X POST http://localhost:8080/api/bootstrap \ -d '{"username": "admin", "email": "admin@example.com", "password": ""}' ``` -Save the `api_key.key` value from the response — it is shown **once only**. The -`password` is optional and is what signs you in to the [web UI](#web-ui). - -Use it as a bearer token for all subsequent requests: +Sign in at with that username and password. The response also carries an +API key, shown **once**, for scripts: ```bash -export KEY= curl -H "Authorization: Bearer $KEY" http://localhost:8080/api/users ``` -### Web UI +Then create a team, add an alert source to get a webhook URL, and point Alertmanager's webhook +receiver at it: see [Alertmanager configuration](./docs/alertmanager.md). -The server serves a web UI at `/`: the incident queue, each incident's alerts -and timeline with every action (acknowledge, assign, snooze, note, resolve, -archive), who is on call, the alert feed, and an *Account* tab for your own -password and the ntfy topic your pages go to. It is built for a phone first. On a phone -it navigates through a hamburger menu and has a sticky action bar, it follows the -system's dark mode, and it can be added to the home screen. From 900px wide it switches -to a sidebar with the queue and the incident side by side. The Stats page shows -incident counts, MTTA and MTTR, and alert frequency by name, hour and day over a -chosen range. +## Documentation -You sign in with a username and password. Users have no password until one is -set, and a user without one can only use API keys: +The [documentation index](./docs/README.md) lists everything. The main pages: -```bash -# an admin sets someone's first password with their API key -curl -X PUT http://localhost:8080/api/users/2/password \ - -H "Authorization: Bearer $KEY" -H "Content-Type: application/json" \ - -d '{"password": ""}' -``` - -After that, users change it themselves under *Account*. Changing your own -password requires the current one. - -How a browser stays signed in: - -- A successful login sets an `HttpOnly`, `SameSite=Lax` session cookie. It lasts - 30 days and slides forward while it is used, so an on-call phone stays signed - in. -- The cookie is marked `Secure` when `TERDUT_PUBLIC_URL` starts with `https://`, - so set it to the HTTPS address. TLS terminates at the gateway and the server - itself only ever sees plain HTTP. -- Requests authenticated by the cookie are checked for cross-origin use (Go's - `http.CrossOriginProtection`). That is the CSRF guard. Bearer-key clients are - not affected. -- Setting a password signs that user out everywhere else. -- Ten failed logins for one username within 15 minutes lock that username for - the rest of the window. - -With `TERDUT_PUBLIC_URL` set, tapping a push notification opens the incident in -the web UI (`/incidents/{id}`). - -A **Team** tab holds everything a team owns, in five sub-sections with a URL -each and a strip across the top to move between them: the on-call rota -(`/team/rota`), the membership (`/team/members`), the escalation ladder -(`/team/escalation`), the alert sources with their keys (`/team/sources`) and -the dead man's switches (`/team/deadman`). `/team` itself is an overview — who -is on call today, how many members and owners, how many ladder levels, how many -keys and how many switches — so a page fetches only what it shows. An owner -edits it; a member sees the same pages read-only, because the server refuses -their writes anyway. Somebody in more than one team picks between them above -the strip, since the choice changes the subject of all five. - -The rota is a month at a time, one coloured initial per day with a legend -underneath, and it says how many days are left uncovered — the question a rota -is read for is who holds which stretch, and a run of one colour answers it -where a list of dates does not. An owner taps a day to hand it to somebody or -empty it, and fills a whole shift from the range form folded in below. - -The **Admin** tab appears only for a system administrator, and holds what -belongs to the whole server rather than to one team. It has three sub-sections, -each with a URL of its own and a strip across the top to move between them: -every team (`/admin/teams`), every user (`/admin/users`), and the settings that -used to be environment variables (`/admin/settings`). `/admin` itself is an -overview — how many of each, and what each section is for. Adding somebody is -minting them an invite link into a team, rather than creating a bare account: -the person who accepts it picks their own password, so one never passes through -an administrator, and the link carries the team, so they land somewhere with a -queue in it. That happens on the team's own page, since an invite is a fact -about a team; the user list points there rather than asking which team beside a -form. - -A name in the team list opens **that team's page**, at `/admin/teams/{id}`: when it -was created, how many are in it and how much is open, a field to rename it, the -members with their roles, the invites into it, and deletion. The member list is the -one thing there that needed a new endpoint — `GET /api/teams/{id}/members` is -member-only and answers `404` to an administrator who is not in the team, which is -the rule and not an oversight, so the page reads `GET /api/admin/teams/{id}` instead. -An administrator still sees none of that team's incidents, alerts or rota. - -A name in the user list opens **that person's page**, at `/admin/users/{id}`: their -email and when they joined, where their notifications go, whether they are an -administrator, whether the account is disabled, the teams they are in with their -role in each, a password field for a first or forgotten one, and deletion. It is -the one place membership is edited from the person's side — the Team tab answers -"who is in this team", and answering "which teams is this person in" there means -visiting each team in turn. - -### Single sign-on (OIDC) - -terdut can sign people in through any OpenID Connect provider; the examples use -[Authentik](https://goauthentik.io/). Groups at the provider decide who may sign -in, which teams they belong to and whether they administer the install, much as -Grafana's OAuth role and org mapping does. Password login keeps working alongside -it unless you turn it off. - -**At the provider**, create an OAuth2/OpenID provider and an application for it: -a *confidential* client, redirect URI `/api/oidc/callback`, and -the `openid`, `profile` and `email` scopes. The issuer is the application's, e.g. -`https://auth.example.com/application/o/terdut/`. Then set: - -```sh -TERDUT_PUBLIC_URL=https://terdut.example.com -TERDUT_OIDC_ISSUER=https://auth.example.com/application/o/terdut/ -TERDUT_OIDC_CLIENT_ID=terdut -TERDUT_OIDC_CLIENT_SECRET=... -TERDUT_OIDC_ALLOWED_GROUPS=terdut-users,terdut-admins -TERDUT_OIDC_ADMIN_GROUP=terdut-admins -``` - -Which team a group grants is not server-wide config: each team names its own -group(s), set by that team's own owner (or an administrator) from its Members -tab, or `PUT /api/teams/{teamID}/oidc-groups {"member_group":"sre","owner_group":"sre-leads"}`. -A team must already exist before a group can grant access to it — the sync -never creates one. - -The web UI's sign-in page shows a "Sign in with " button (a plain link to -`/api/oidc/login`) above the password form, or instead of it when -`TERDUT_PASSWORD_LOGIN=false`; it asks `GET /api/auth/config` what the server offers -(`password_login`, `oidc.enabled`, `oidc.name`). A refused sign-in comes back to that -page with the reason spelled out. Access the groups grant is badged **SSO** on the -Team, Admin and per-user pages, with its edit and remove controls disabled, and the -Account page does not offer to set a password nobody could use. - -**What a sign-in does** - -1. *Who.* The provider's `(issuer, subject)` is the identity. The first time, a - user is found by email — only when the provider marks it verified, or - `TERDUT_OIDC_TRUST_EMAIL` is set — or created with no password. A username taken - by somebody else gets a numeric suffix (`alice-2`). Username and email follow the - provider at each sign-in. Authentik reports `email_verified` as false unless - configured otherwise, so linking existing users usually needs - `TERDUT_OIDC_TRUST_EMAIL=true`. -2. *Whether.* With `TERDUT_OIDC_ALLOWED_GROUPS` set, somebody in none of them is - refused and nothing is created. -3. *What.* The administrator flag follows `TERDUT_OIDC_ADMIN_GROUP`. Team roles - follow each team's own `oidc_member_group`/`oidc_owner_group`; where both of a - team's groups match, the owner group wins. - -**Managed access.** What the sync grants is marked as managed by single sign-on, -and only that is ever changed by it. It is added at sign-in, and removed at the -next sign-in after the group is gone, even if that leaves a team without an owner -(an administrator can always repair a team) — the provider is the source of truth -for what it grants, so the last-owner and last-administrator guards do not apply. -Memberships and administrators added by hand are left alone; the exception is a -hand-added member whose team's own group grants a *higher* role, who is raised and -from then on managed. Editing managed access by hand (`POST` or `DELETE` on a -team's members, revoking an SSO-granted administrator) is refused with `409`, since -the next sign-in would undo it. - -> **Upgrading past migration 013: reconfigure every team's groups.** -> `TERDUT_OIDC_GROUP_MAPPINGS` is gone, and the sync no longer creates a team by -> name. Group-to-team-role mapping is now each team's own setting — an owner sets -> it from the Members tab, or `PUT /api/teams/{teamID}/oidc-groups`. Until a team's -> owner does that, an OIDC-sourced membership in it is dropped at that user's next -> SSO sign-in, the same as any other loss of group access. Set every team's groups -> before affected users next sign in, to avoid a visible gap in access. - -**How fast changes arrive.** Groups are read only at sign-in. A session made by an -SSO sign-in has a hard ceiling (`TERDUT_OIDC_SESSION_MAX_AGE`, default 12h) that -sliding never extends, so a change at the provider reaches terdut within that time. -Password sessions are unaffected. - -> **API keys are not revoked when somebody is removed at the provider.** terdut -> holds no refresh token and never asks the provider again, so a person removed -> from every allowed group loses their sessions within `TERDUT_OIDC_SESSION_MAX_AGE` -> and cannot sign in again, but keeps any API key they made (the TUI and scripts use -> them) until an administrator disables the user in terdut. - -**Signing in from a terminal.** A client with no browser of its own, such as the -TUI over SSH, signs in with a device code, run by terdut itself so the terminal -never talks to the provider: - -1. The terminal calls `POST /api/oidc/device` and shows the person a link - (`/device?code=XXXX-XXXX`) and the code. -2. On any device the person opens the link, signs in (by the provider or by - password, whatever the login page offers), sees the code and the account, and - presses **Approve**. Only a browser session can approve; an API key cannot. -3. The terminal polls `POST /api/oidc/device/token` every 5 seconds and is given the - ordinary `terdut_session` cookie once. A person who signs in through the provider - gets the same `TERDUT_OIDC_SESSION_MAX_AGE` ceiling on the terminal's session as - on their browser's. - -A login expires after 10 minutes. `GET /api/auth/config` reports `device_login`. - -**If the provider is down**, terdut still starts (discovery is fetched on first -use) and password login is the way in. With `TERDUT_PASSWORD_LOGIN=false` that way -is closed: set it back to `true`. The first administrator comes from the bootstrap -endpoint, and stays a manual administrator that no group can revoke; on an SSO-only -install set `bootstrap.enabled: false` in the chart if you don't want that account, -or keep it and never give it a password. - -### Docker - -```bash -docker build -t terdut-server . -docker run -p 8080:8080 \ - -e TERDUT_DB_DSN='postgres://terdut:secret@host.docker.internal:5432/terdut?sslmode=disable' \ - terdut-server -``` - -The server creates its own schema on startup and needs a reachable Postgres; it stores nothing on -disk, so there is no volume to mount. - -### Kubernetes - -A Helm chart is published from this repository as an OCI artifact, versioned in lockstep -with the app — chart `x.y.z` is always app `vx.y.z`: - -```bash -helm upgrade --install terdut-server oci://git.ryuvia.com/niklas/terdut-server \ - --version 0.9.2 \ - --namespace terdut-server --create-namespace \ - --set networking.hostname=terdut.example.com -``` - -The chart expects a [Gateway API](https://gateway-api.sigs.k8s.io/) Gateway named `envoy-main` in -the `envoy-gateway-system` namespace to already exist — it renders an `HTTPRoute` against it rather -than an `Ingress`. TLS is terminated at the gateway, so the server itself never sees a certificate. - -| Value | Default | Description | -|---|---|---| -| `networking.hostname` | `terdut.example.com` | Hostname the `HTTPRoute` serves | -| `networking.listener` | `""` | Gateway listener (`sectionName`) to bind to. Empty attaches to every matching listener, **including plaintext HTTP** — set it to the HTTPS listener's name to serve TLS only | -| `networking.servicePort` | `8080` | Port the route forwards to; keep in sync with `service.port` | -| `bootstrap.enabled` | `true` | Runs a post-install hook that creates the first user and stores its API key in the `-admin-key` Secret. Already-bootstrapped servers are left alone | -| `database.dsn` | `""` | **Required.** Postgres DSN, with no password in it. The chart provisions no database | -| `database.passwordSecret.name` | `""` | Secret supplying `PGPASSWORD`. With the Zalando postgres operator, the Secret it generates for the role | -| `database.passwordSecret.key` | `password` | Key within that Secret | - -The API key travels in an `Authorization: Bearer` header, so set `networking.listener` whenever the -hostname is reachable outside a trusted network. - -#### The database - -The chart provisions no database: it takes a DSN and expects a Postgres that already exists. In this -cluster the wrapper chart declares an `acid.zalan.do/v1 postgresql` CR; anywhere else, any reachable -Postgres 14+ will do. - -The DSN carries no password. pgx falls back to libpq's environment variables for whatever the DSN -leaves out, so the password arrives as `PGPASSWORD` from a Secret and never appears in values, in -the rendered manifest or in `kubectl describe pod`. With the postgres operator that Secret is the -one it generates for the role, so a rebuild mints a new password with nothing to keep in sync — -the same wiring miniflux uses. - -The server migrates its own schema on startup, so a new database only has to exist and be writable. - -#### Backups - -Postgres is backed up where it runs, not from here. The database pod carries a -[k8up](https://k8up.io/) `k8up.io/backupcommand` annotation that streams a `pg_dump`, the same way -gitea and immich do in this cluster. - -This used to be the app's problem: the SQLite database lived on a PVC beside the server, the image -is `FROM scratch` with no interpreter to dump it, and WAL mode makes a file-level copy of the volume -non-crash-consistent — so the chart shipped an idle `python:*-alpine` sidecar purely to give k8up -somewhere to exec. The sidecar, the PVC and the `backupSidecar` values are all gone. - ---- - -## Configuration - -Two kinds of setting, split by who changes them and how often. - -**Where the server is plugged in** stays in the environment: the listen address, -the database DSN, the ntfy URL and token, the public URL. They are needed before -the database is open, and two of them are credentials. - -**How the server behaves** lives in the database and is edited by an -administrator in the web UI or through `PUT /api/admin/settings`, taking effect -on the next sweep rather than at the next restart. The variables below marked -**seed** are the value each of those starts from: written once, on first start, -and never overwritten afterwards — a redeploy cannot put a chart's default back -over an administrator's edit. - -| Variable | Default | Description | -|---|---|---| -| `TERDUT_ADDR` | `:8080` | TCP address to listen on | -| `TERDUT_DB_DSN` | — | **Required.** Postgres connection string, e.g. `postgres://terdut:secret@localhost:5432/terdut?sslmode=require` | -| `TERDUT_ARCHIVE_AFTER` | `168h` (7d) | **seed.** How long a resolved alert or incident stays in the default list before being auto-archived | -| `TERDUT_STALE_AFTER` | `6h` | **seed.** How long a firing alert may go without a refreshing webhook before it is treated as resolved — **must exceed your Alertmanager `repeat_interval`** | -| `TERDUT_DEADMAN_MATCHERS` | `alertname=Watchdog` | The **default** matchers a team starts with — switches are per team now, and this seeds teams that have no configuration of their own. `;` separates matchers, `,` the label conditions within one, `=` is exact equality. Every matcher must name an `alertname` | -| `TERDUT_DEADMAN_TIMEOUT` | `15m` | How long a heartbeat may go unheard before its switch is declared dead — **must be shorter than the `repeat_interval` of the route carrying it**. `0` disables dead man's switch handling | -| `TERDUT_DEADMAN_SEVERITY` | `critical` | Severity a dead man's switch incident opens at | -| `TERDUT_NTFY_URL` | — | ntfy server to publish push notifications to. Empty disables notifications entirely | -| `TERDUT_NTFY_TOKEN` | — | Bearer token for an access-controlled ntfy | -| `TERDUT_NTFY_FALLBACK_TOPIC` | — | Topic used when nobody is on call | -| `TERDUT_PUBLIC_URL` | — | Base URL a phone uses to reach this server: the notification's link into the web UI, its Acknowledge button, and whether the session cookie is `Secure` | -| `TERDUT_NOTIFY_REPEAT` | `15m` | **seed.** How long an incident may sit unacknowledged before it is paged again. `0` notifies once and never repeats | -| `TERDUT_PASSWORD_LOGIN` | `true` | `false` refuses password login and password sign-up (`403`), leaving single sign-on the only way in. Refused at startup unless SSO is configured | -| `TERDUT_OPERATOR_MODE` | `false` | Declares this install gitops-managed: a session's or a user's own API key's writes to teams, escalation policies, dead man's switches and integrations are refused (`403 reason:"operator_managed"`); a [service account](#service-accounts)'s are not. Team membership and the schedule stay editable regardless | -| `TERDUT_OIDC_ISSUER` | — | Turns single sign-on on. The provider's issuer URL; discovery is read from `/.well-known/openid-configuration`. See [Single sign-on](#single-sign-on-oidc) | -| `TERDUT_OIDC_CLIENT_ID` / `TERDUT_OIDC_CLIENT_SECRET` | — | **Required with an issuer.** The confidential client registered at the provider. Keep the secret in a Secret, not in values | -| `TERDUT_OIDC_NAME` | `SSO` | What the sign-in button calls the provider | -| `TERDUT_OIDC_SCOPES` | `openid profile email` | Scopes requested, comma or space separated. Authentik puts `groups` behind `profile` | -| `TERDUT_OIDC_USERNAME_CLAIM` / `_EMAIL_CLAIM` / `_GROUPS_CLAIM` | `preferred_username` / `email` / `groups` | ID token claims read for the username, email and groups | -| `TERDUT_OIDC_TRUST_EMAIL` | `false` | Link a first sign-in to an existing local user by email even if the provider does not mark the address verified | -| `TERDUT_OIDC_ALLOWED_GROUPS` | — | Comma-separated. Only people in one of these may sign in. Empty admits everybody the provider authenticates | -| `TERDUT_OIDC_ADMIN_GROUP` | — | Members are system administrators | -| `TERDUT_OIDC_SESSION_MAX_AGE` | `12h` | Hard ceiling on a session made by an SSO sign-in | - -Durations use Go syntax (`30m`, `12h`, `168h`). An unparseable value falls back to the default. - -Note that `TERDUT_STALE_AFTER` and `TERDUT_DEADMAN_TIMEOUT` point in opposite directions. Staleness -is a generous grace period around a `repeat_interval` you do not control; a dead man's switch is a -deadline you set deliberately, and the heartbeat's route is configured to beat faster than it. - -In the Helm chart the two sweeper durations are set via `sweeper.staleAfter` and `sweeper.archiveAfter`, dead man's switches via the `deadman.*` values, notifications via the `notify.*` values, single sign-on via `oidc.*` and `passwordLogin`, and operator mode via `operatorMode`. - ---- - -## Alertmanager configuration - -Alerts arrive on a team's **integration key**, which says both that the sender -may post and which team the alerts belong to. Mint one as an owner of the team: - -```bash -curl -X POST https://terdut.example.com/api/teams/1/integrations \ - -H "Authorization: Bearer $TERDUT_API_KEY" \ - -H 'Content-Type: application/json' \ - -d '{"name":"prod alertmanager"}' -``` - -The response carries the key and the full URL **once**; only a SHA-256 hash is -stored. Put it in your `alertmanager.yml`: - -```yaml -receivers: - - name: terdut - webhook_configs: - - url: http://terdut-server:8080/api/integrations//alertmanager - send_resolved: true - -route: - receiver: terdut -``` - -The whole URL is a credential, so treat it like one. Alertmanager 0.26 and -later can read it from a file with `url_file:` instead, which keeps it out of -your configuration repository: - -```yaml - - url_file: /etc/alertmanager/secrets/terdut-webhook-url/url - send_resolved: true -``` - -The webhook endpoint requires no authentication. - -If you use the [dead man's switch](#dead-mans-switch) — and the default configuration does — give -the heartbeat a route of its own, because the deadline is only as tight as the interval feeding it: - -```yaml -route: - receiver: terdut - repeat_interval: 4h - routes: - - matchers: [ 'alertname = "Watchdog"' ] - receiver: terdut - group_wait: 0s - group_interval: 1m - repeat_interval: 1m -``` - -That delivers a heartbeat every **2 minutes**, not every minute. Alertmanager only reconsiders a -group every `group_interval`, and at exactly one elapsed interval `repeat_interval` has not *quite* -passed, so the send slips to the next tick — equal values give 2×. Two minutes against the 15 minute -default is seven heartbeats per window, which is the point; use `group_interval: 30s` if you want -the numbers to mean what they say. - -kube-prometheus-stack users get the `Watchdog` alert (`expr: vector(1)`) for free; it just needs -routing to terdut rather than to `null`. - ---- - -## Alerts and incidents - -There are two objects, and the difference between them is the whole design. - -**An alert is Alertmanager's record.** It has two states, `firing` and -`resolved`, one row per fingerprint, and no human ever writes to it. The API -exposes alerts read-only. - -**An incident is the work item.** It goes `triggered → acknowledged → resolved`, -carries an assignee, a snooze, notes and a timeline, and is the only thing people -act on. Many alerts belong to one incident. - -### Correlation uses Alertmanager's `groupKey` - -Alertmanager has already grouped alerts according to the `group_by` routing tree -you configured, and it sends the resulting `groupKey` and `groupLabels` on every -webhook. Incidents adopt that answer rather than re-grouping alerts a second -time — if you want different correlation, change `group_by` in -`alertmanager.yml` and terdut follows. - -At most one incident is open per `groupKey` at a time. Alerts firing in a group -that already has an open incident join it. The incident's `severity` is a -high-water mark — the highest `severity` label any of its alerts has carried — so -an incident that hit `critical` still reads as critical after the critical alert -clears. - -### Several clusters, one team - -A team with one Alertmanager per Kubernetes cluster, each posting to its own -source, needs two settings or the clusters run together. - -1. Give every alert a `cluster` label at the source. In Prometheus that is - `externalLabels: {cluster: prod-eu}` (kube-prometheus-stack: - `prometheus.prometheusSpec.externalLabels`). -2. Add `cluster` to `group_by` in `alertmanager.yml`. - -The second one is the one that matters. Incidents are matched on the team and -Alertmanager's `groupKey`, and the `groupKey` does not include external labels: -without `cluster` in `group_by`, the same alert in two clusters has the same -key and joins one incident. With it, each cluster gets its own, `cluster` is in -the incident's `group_labels`, and the web UI shows it as a coloured chip on the -queue, the incident and the alert list, instead of leaving it in the title. -An alert that is not grouped by `cluster` still shows the chip on the alert -list, which reads the label from the alert itself. - -The queue has a cluster dropdown once there are two or more values to choose -between. It filters on the incident's `cluster` group label -(`GET /api/incidents?cluster=...`), so it only sees incidents grouped by it. - -### An incident opens only on a new occurrence - -An incident opens when an alert **transitions into firing**: a fingerprint that -was never seen, an alert with a newer `startsAt`, or a resolved alert that -started again. The unchanged firing notifications Alertmanager re-sends every -`repeat_interval` are none of those, and open nothing. - -This is what makes closing an incident by hand mean something. Without the rule, -`POST /api/incidents/{id}/resolve` would be undone by the next re-send of an -alert that never stopped firing. - -### Leaving the open state - -- **Automatically**, once every alert under the incident has stopped firing — - whether by a resolved webhook or by the sweeper's - [stale-alert expiry](#stale-alert-expiry). The incident gets - `"resolution_source": "alerts"`. -- **By hand**, via `POST /api/incidents/{id}/resolve` - (`"resolution_source": "manual"`). This is **terminal**: a later occurrence in - that group opens a *new* incident rather than reopening this one. If the alert - underneath never stops firing, the incident stays closed — that is what - resolving by hand asserts. -- **On recovery**, for a [dead man's switch](#dead-mans-switch) incident whose - heartbeat started arriving again (`"resolution_source": "recovered"`). These - incidents have no member alerts, so the automatic cascade above cannot reach - them. - -To quieten an incident you expect to come back, snooze it instead -(`POST /api/incidents/{id}/snooze`). A snooze hides the incident from the default -list without closing it, and expires by simply falling into the past. - -### On-call assignment - -A new incident is assigned to whoever holds today's schedule entry at the moment -it opens (`GET /api/schedule/current`). If nobody is scheduled it opens -unassigned. Reassign with `POST /api/incidents/{id}/assign`. - -One person holds a given day, so `POST /api/schedule` refuses a date somebody -already has: taking a shift off the person expecting to be paged for it should -not be something a plain call does by accident. Pass `"replace": true` to take -them anyway. Either way the whole request is one transaction — a week where some -days are free and some are taken moves as a unit, and a failure leaves the rota -exactly as it was rather than with a hole in it. - -### Push notifications - -With `TERDUT_NTFY_URL` set, an incident that opens is pushed to the on-call -person's phone through [ntfy](https://ntfy.sh). Everybody sets their own topic -under *Account* in the web UI, where a **Send a test push** button proves it -before an incident has to; `PUT /api/users/{id}/notify` is the same thing over -the API, and an administrator may set somebody else's. A user with no topic -falls back to `TERDUT_NTFY_FALLBACK_TOPIC`, as does an incident that opens with -nobody on call. If neither yields a topic, nothing is queued. - -The **server** is the install's one ntfy, from `TERDUT_NTFY_URL`, and is not -something a user picks. Only the topic is per-person. - -A topic is a shared secret with the ntfy server: anyone who knows it can both -read the pages and publish to it, so an unguessable one is worth the trouble. -That is also why the topic never appears in an incident's timeline, which every -API key can read. - -Three things get pushed: - -- **triggered** — an incident opened. Priority follows severity (`critical` maps - to ntfy's max priority, the one that overrides the phone's quiet settings). -- **reminder** — the incident is still `triggered` after `TERDUT_NOTIFY_REPEAT`. - Repeats until somebody acts. Acknowledging, snoozing, resolving or archiving - all stop it — snooze is the mute button. -- **resolved** — every alert under the incident stopped firing. Only sent to - whoever was paged in the first place, and only for the automatic cascade: - resolving by hand pushes nothing, since the person who did it already knows. - -Notifications carry an **Acknowledge** button that acknowledges the incident -without opening anything. It POSTs to `/api/notify/ack/{token}`, an -unauthenticated route authorised by the 256-bit token in its path — minted fresh -per notification, scoped to one incident and one action, and valid for 24 hours. -A real API key is never put in a notification, because the message is stored on -the ntfy server and cached on the device. - -The token is **not** consumed by use. Acknowledging is idempotent, so a token -stays valid for its full 24 hours and a second tap is a no-op that reports the -incident's current state rather than an error — which is what you want when a -tap is retried on a flaky mobile connection. What bounds it is scope, not a use -count: one incident, one action, one day. Expired tokens are purged by the -sweeper. - -Two consequences worth planning for: - -- `/api/notify/ack/{token}` **must stay publicly reachable**, or the button will - not work when the responder is off your network. -- Notifications sent to the fallback topic carry **no** Acknowledge button. The - topic is shared, and a button on it would let any subscriber acknowledge as - somebody else. - -Delivery is a queue, not an inline call: the webhook writes a row and a -background notifier sends it within 30 seconds, retrying with exponential -backoff up to 8 attempts. Nothing about ingestion blocks on ntfy being reachable. - -Every delivery is recorded on the incident's timeline: a `notified` event once -ntfy accepts the publish, and a `notify_failed` event when a notification -exhausts its retries. Written from the result rather than at enqueue, so the -timeline says what actually happened — and a page that never landed is visible -instead of looking the same as one that did. - -### Escalation - -Without a ladder, an unacknowledged incident re-pages the same topic every -`notify_repeat` forever. That is a louder version of the same silence: if the -person on call is asleep, out of signal, or has left, nothing else happens. - -A team can configure an ordered ladder instead. Each level has a timeout and a -set of targets, and a target is either a named person or **whoever the team's -rota says is on call today** — the target that keeps working when the rota -changes and nobody remembers to edit the policy. - -``` -level 1 5m oncall the rota gets first refusal -level 2 5m user:bob then a named second - then repeat_count more rounds - then the team's fallback topic, once -``` - -When a level's timeout passes with the incident still `triggered`, the next -level is paged. Off the end of the ladder the whole thing runs again -`repeat_count` times, and after that the team's `fallback_topic` is paged once -as the end of the line. The incident stays open throughout: running out of -people to wake is not the same as somebody answering. - -**Acknowledging or resolving stops it**, which is the point — continuing to wake -people after somebody has said "I have this" is how a tool teaches people to -mute it. **Snoozing pauses it**: a deliberate "not now" holds the ladder where -it is, and it resumes when the snooze runs out. - -Every step is on the incident's timeline with the level and the names it woke, -so somebody reading it afterwards can tell why their phone rang at 04:00. A -level whose targets are all unreachable — no ntfy topic, a disabled account, an -empty rota — is recorded as `nobody reachable` and the ladder moves on rather -than stalling on a rung that cannot ring. - -**Reminders and escalation never both run.** A team with a ladder gets -escalation; a team without keeps the reminder behaviour exactly as it was. Two -pages for one silence is the surest way to get a tool muted. - -The ladder's `fallback_topic` is per team, unlike `TERDUT_NTFY_FALLBACK_TOPIC`, -which is the install-wide topic used when an incident opens with nobody on call. -They answer different questions: one is "nobody was scheduled", the other is -"everybody scheduled has been tried". - -### Stale alert expiry - -A resolved webhook is the only signal that an alert has stopped firing, so a -notification that is dropped, silenced, or lost to a restart would otherwise pin -that alert as firing forever. A background sweeper resolves firing alerts that -Alertmanager has stopped refreshing, using either signal: - -- the `endsAt` watermark on the last notification has passed, or -- no webhook has refreshed the alert within `TERDUT_STALE_AFTER`. - -Alertmanager re-sends firing notifications every `repeat_interval`, which is what -keeps a live alert fresh — so `TERDUT_STALE_AFTER` must be comfortably larger -than your `repeat_interval` (default 4h), or live alerts will be resolved -prematurely. Alerts resolved this way are marked `"resolution_source": "expiry"` -to distinguish them from a real Alertmanager resolve (`"alertmanager"`). - -An expiry cascades: once it leaves an incident with nothing firing under it, the -incident resolves too, in the same sweep. - -### Dead man's switch - -Everything above assumes alerts arrive. If Prometheus stops evaluating, or -Alertmanager cannot reach this server, nothing arrives — and silence looks -exactly like everything being fine. A dead man's switch inverts the handling for -one designated alert so that silence is the signal: - -- **receiving** it opens no incident, and -- the **absence** of it does. - -kube-prometheus-stack already ships the alert for this. `Watchdog` is -`expr: vector(1)`, so it fires permanently and is re-sent forever; it is worth -nothing unless something downstream notices it stop. That is what -`TERDUT_DEADMAN_MATCHERS` defaults to. - -**Switches belong to a team**, which decides which of its own alerts are -heartbeats and how long a silence has to last. Each **switch** is a row of its -own — a name, one matcher, a timeout and a severity — so switches in one team -can have different deadlines. An owner adds and removes them on **Team → -Switches**, which lists each with a status (**healthy**, **dead**, or -**dormant** until its first heartbeat), when it was last heard from, and when it -last opened an incident; a matcher that several clusters satisfy is broken down -per cluster. The API is `POST`/`DELETE /api/teams/{teamID}/deadman/switches`. A -missed heartbeat opens an incident in the team whose integration received it. -Removing a switch stops the watching; an incident it already opened stays open -until somebody resolves it. - -The environment variables are the starting point, not the setting: the **first** -time the server starts, every team is given a switch per default matcher from -them, once. After that a team's switches are its own — an owner's edit or -deletion is never put back by a redeploy. A team created later starts watching -nothing until its owner says otherwise — inheriting an install-wide heartbeat -would page a new team about a source it has never heard of. - -A matcher is a set of exact label conditions, one of which must be the -`alertname`, in the format the environment variable uses (one matcher per switch; the -variable takes several, separated by `;`): - -``` -alertname=Watchdog,cluster=prod; alertname=EdgeHeartbeat -``` - -**The unit of monitoring is the fingerprint, not the alert name.** Two clusters -sending the same `Watchdog` are two independent switches, so a healthy one can -never mask a dead one. - -#### The lifecycle - -A switch is **dormant** until its first heartbeat arrives. A configured matcher -that has never been heard from opens nothing, so a fresh deploy or a restored -database does not page. It also means a matcher that never matches anything is -silently inert — check the startup log line, which lists the matchers that -survived parsing. - -Once armed, the sweeper declares it **dead** when either the heartbeat has not -been refreshed within `TERDUT_DEADMAN_TIMEOUT`, or Alertmanager explicitly -resolved it — the sender saying the heartbeat stopped needs no further waiting. -That opens an incident at `TERDUT_DEADMAN_SEVERITY`, assigned and paged like any -other, and marks the heartbeat alert `"resolution_source": "deadman"` so the -alert list stops claiming a dead switch is firing. - -It **recovers** when the heartbeat starts arriving again: the incident resolves -with `"resolution_source": "recovered"` and the all-clear goes to whoever was -paged. - -Resolving the incident by hand sticks, the same way it does for an alert-backed -one. While the switch stays silent nothing new opens — so a decommissioned -source is a one-time page rather than a nag. The switch **re-arms** on the next -heartbeat: come back and die again, and that is a new incident. - -#### Two things to know - -`TERDUT_DEADMAN_TIMEOUT` must be **shorter** than the `repeat_interval` of the -route carrying the heartbeat, which is the exact opposite of -`TERDUT_STALE_AFTER`. Inheriting a default `repeat_interval` of 4h gives you a -switch that takes four hours to notice anything, so give the heartbeat -[its own route](#alertmanager-configuration). Matched alerts are exempt from -stale-alert expiry — a heartbeat answers to its own timeout and nothing else. - -A dead man's switch incident has **no member alerts**: -`GET /api/incidents/{id}/alerts` returns an empty list. There is no alert -describing the problem, because the problem is that no alert arrived. What -happened is on the timeline instead, as a `deadman_silent` event carrying the age -of the last heartbeat, and the heartbeat's labels are on the incident's -`group_labels`. - ---- - -## API reference - -### Authentication - -All endpoints except `/api/bootstrap`, `/api/integrations/{key}/alertmanager`, -`/api/notify/ack/{token}`, `/api/login`, `/api/logout`, `/api/auth/config`, -`/api/version`, `/api/oidc/login`, `/api/oidc/callback`, `/api/oidc/device` and -`/api/oidc/device/token` require either an API key: - -``` -Authorization: Bearer -``` - -or the web UI's session cookie. A request that carries an `Authorization` header -is judged on that header alone. - -Two kinds of user exist. An **administrator** manages accounts: creating and -deleting users, setting anybody's password, minting keys for anybody, and -granting the flag itself. Everybody else works incidents — acknowledging, -assigning, snoozing, resolving, noting — and manages their own account and -nobody else's. An API key carries exactly the rights of the user it belongs to. - -A third principal, the **service account**, exists for automation (a -Kubernetes operator, most likely) that needs to manage teams, escalation -policies, dead man's switches and integrations without impersonating a human. -It is not a user — it never signs in, never appears in a team's member list, -and never holds the administrator flag — and its key is prefixed `tdsa_` so it -reads as one at a glance in a log line. See [Service accounts](#service-accounts). - -**Getting an account.** The first one comes from `/api/bootstrap`. After that -it depends on `signup_mode`, an administrator setting: - -- `invite_only` (the default) — a team owner mints a link with - `POST /api/teams/{teamID}/invites`, and the person who opens it picks a - username and password and lands in that team with the role the link carries. - Links are single-use unless told otherwise, expire after seven days, and can - be revoked before that. -- `open` — anybody who can reach the server can create an account, and must - name a team, which they then own. - -Invites are **links, not email**: this server has no SMTP, and adding it to send -one message would be a subsystem to run, secure and monitor. Send the link -however you already talk to the person. - -A domain-restricted third mode was considered and dropped: with no email there -is nothing to verify an address against, so it would only check the domain of a -string somebody typed. - -The first user, from `/api/bootstrap`, is an administrator. Users created -afterwards are not, until an administrator says so. An install always keeps at -least one: the last administrator can be neither deleted nor demoted, and -nobody can delete or demote themselves. - -Endpoints that require the flag answer `403` with -`{"error":"administrator access required"}`. - -**Teams** are the unit of tenancy, and are a separate axis from the administrator -flag. A team owns its incidents, alerts, schedule and integrations, and a user -sees exactly the teams they belong to. Within a team an **owner** configures it -(schedule, integrations, membership) and a **member** works its incidents. - -An administrator crosses that line in one direction only. They **configure any -team** without being in it — every owner-only endpoint accepts the flag, because -otherwise a team whose last owner left could never be repaired. They do **not -read any team**: the queue, the alerts and the incidents are filtered by real -membership, so an administrator sees a team's work only by joining it, which is -a membership change and shows up as one. Administration is about accounts and -the shape of a team, not about reading other people's incidents. - -Anything belonging to a team you are not in answers `404`, not `403`: whether an -incident exists is itself something only its team should learn. - -**Operator mode** (`TERDUT_OPERATOR_MODE`, see [Configuration](#configuration)) -declares this install gitops-managed. When it is on, a session or a user's own -API key gets `403 {"error": "...", "reason": "operator_managed"}` on every -write this README marks **owner**-gated under Teams below (creating, renaming -or deleting a team; its OIDC group binding; its escalation ladder; its dead -man's switches; its integrations) — a service account's writes are unaffected. -Team membership and invites are deliberately excluded: they are never -gitops-managed, in operator mode or out of it. `GET /api/auth/config` reports -`operator_mode` so a client can grey those sections out before a write is ever -attempted. - -| Method | Path | Description | -|---|---|---| -| `GET` | `/api/auth/config` | How to sign in: `{"password_login", "oidc": {"enabled","name"}, "device_login", "operator_mode"}`. No session needed | -| `GET` | `/api/version` | `{"version"}` — this build's version string. No session needed, the same as `/healthz` | -| `POST` | `/api/login` | `{"username","password"}` → sets the session cookie, returns `{user, has_password}`. `429` after too many failures; `403` when `TERDUT_PASSWORD_LOGIN=false` | -| `GET` | `/api/oidc/login` | Starts a single sign-on sign-in: redirects the browser to the provider. `?next=/path` is where to land afterwards; only a path on this server is honoured. Only exists when SSO is configured | -| `POST` | `/api/oidc/device` | Starts a device login: returns `{device_code, user_code, verification_url, interval, expires_in}`. Only exists when SSO is configured | -| `POST` | `/api/oidc/device/token` | `{"device_code"}` → `202 {"status":"pending"}`, then `200` with the session cookie once approved (once only). `410` with `{"error":"expired"}` or `{"error":"denied"}`; `429 {"error":"slow_down"}` if polled faster than `interval` | -| `POST` | `/api/oidc/device/approve` | **session** — `{"user_code"}`. Approves a pending device login as the caller. `403` for an API key; `404` for an unknown, expired or already decided code | -| `POST` | `/api/oidc/device/deny` | **session** — `{"user_code"}`. Refuses it | -| `GET` | `/api/oidc/callback` | Where the provider sends the browser back. Sets the session cookie and redirects to `/`, or to `/?sso_error=` — one of `denied`, `expired`, `failed`, `unavailable`, `not_allowed`, `no_email`, `email_conflict`, `disabled`, `not_bootstrapped` (no user exists on this install yet — sign in again once something has called `/api/bootstrap`) | -| `POST` | `/api/logout` | Ends the session and clears the cookie | -| `GET` | `/api/me` | The caller: `{user, has_password}` | - -### Users - -| Method | Path | Description | -|---|---|---| -**admin** marks an endpoint that requires the administrator flag; **self or -admin** marks one you may use on your own account and an administrator may use -on anybody's. - -| Method | Path | Who | Description | -|---|---|---|---| -| `GET` | `/api/signup` | — | Whether sign-up is open, and whether `?invite=` is usable. No session needed: the caller has no account yet | -| `POST` | `/api/signup` | — | Create an account `{"username","email","password","invite"?,"team_name"?}` and sign in. `403` without a usable invite when the mode is invite-only | -| `POST` | `/api/bootstrap` | — | Create first user + API key `{"username","email","password"?}` (only works on empty DB). The user is an administrator | -| `GET` | `/api/users` | any | List users. Open to everybody: the queue's assignment control and the schedule both have to name people | -| `GET` | `/api/users/{id}/teams` | self or admin | The teams that user is in, each with their role. `/api/teams` is always about the caller; this one answers it about somebody else, for the admin page's per-user view. `404` for a user who does not exist, so "no teams" and "no such person" are distinguishable | -| `POST` | `/api/users` | **admin** | Create user `{"username","email"}`. Not an administrator | -| `DELETE` | `/api/users/{id}` | **admin** | Delete user (cascades to keys). `409` for yourself or the last administrator | -| `PUT` | `/api/users/{id}/admin` | **admin** | Grant or revoke the administrator flag `{"is_admin"}`. `409` for yourself, the last administrator, or an administrator granted by single sign-on | -| `PUT` | `/api/users/{id}/disabled` | **admin** | Take an account out of use, or put it back `{"disabled"}`. `409` for yourself or the last administrator | -| `PUT` | `/api/users/{id}/notify` | self or admin | Set push notification target `{"ntfy_topic"}` — empty string clears it | -| `PUT` | `/api/users/{id}/password` | self or admin | Set web UI password `{"password","current_password"}`. `current_password` is required only when changing your own existing password. Ends the user's other sessions | -| `POST` | `/api/users/{id}/api-keys` | self or admin | Issue API key `{"name"}` — key shown once | -| `DELETE` | `/api/users/{id}/api-keys/{keyID}` | self or admin | Revoke API key | - -### Administration - -| Method | Path | Who | Description | -|---|---|---|---| -| `GET` | `/api/admin/teams` | **admin** | Every team on the server, with its member and open-incident counts. `/api/teams` answers "what am I in"; this answers "what is there" | -| `GET` | `/api/admin/teams/{teamID}` | **admin** | One team and who is in it: `{"team", "members"}`. `404` for a team that does not exist. `GET /api/teams/{teamID}/members` is **member**-only and still `404`s an administrator from outside the team — reading a team's shape and reading its work are different questions, so they are different endpoints | -| `GET` | `/api/admin/settings` | **admin** | The editable settings with their bounds, plus the environment-configured ones, read-only. Never credentials | -| `PUT` | `/api/admin/settings` | **admin** | Change one or more `{"key": seconds}`, or `{"signup_mode": "open"\|"invite_only"}`. `400` for an unknown key or a value outside its bounds | - -### Service accounts - -A service account is a scoped, non-human credential for automation — not a -`users` row, so it never signs in, is never a team member, and never carries -the administrator flag. Two scopes: - -- **instance** — the same reach system administration has over teams: create - one, and mint a **team**-scoped account against any of them. There is no - cap on how many instance-scoped accounts exist, but ordinarily there is one, - belonging to whatever is provisioning this install end to end. -- **team** — owner-equivalent for that one team, and nothing else: every - **owner**-gated endpoint under [Teams](#teams), membership and invites - included. Nothing narrower is enforced server-side; what actually keeps - membership out of automation's hands is that no operator built against this - scope should ever call those two endpoints — see - [operator mode](#authentication) and `SERVICE-ACCOUNTS.md`'s note on this. - -A key is shown once, at creation or rotation, and only its hash is stored — -the same handling as a user's API key. Losing it means minting a new one; -there is no way to recover a raw key from the server. - -| Method | Path | Who | Description | -|---|---|---|---| -| `GET` | `/api/service-accounts` | **admin** | Every service account. Pass `?name=` instead to look one up by its exact name — open to **any** authenticated caller (human or service account), since it returns no key material and is how an account finds its own id | -| `POST` | `/api/service-accounts` | owner\* | Create one and mint its first key `{"name","scope","team_id"?}` (`team_id` required for `scope:"team"`, absent for `scope:"instance"`). Returns `{"service_account", "key"}` — `key.key` shown once | -| `POST` | `/api/service-accounts/{id}/keys` | owner\* | Mint an additional key `{"name"}` — rotation without recreating the account. Shown once | -| `DELETE` | `/api/service-accounts/{id}/keys/{keyID}` | owner\* | Revoke one key | - -\* For an **instance**-scoped account: a system administrator only. For a -**team**-scoped account: a system administrator, that team's own human owner, -an instance-scoped service account (minting a narrower credential for a team -it just created), or — for the two key endpoints only — the account rotating -or revoking its own key, which is not a privilege escalation, the same -reasoning a user's own API keys rest on. - -### Alert ingestion - -Alerts arrive on a team's integration key. The key is both the credential and the -routing: it says that the sender may post, and which team the alerts belong to. -Create one with `POST /api/teams/{teamID}/integrations`, which returns the key -and the full URL once and stores only a SHA-256 hash. - -| Method | Path | Description | -|---|---|---| -| `POST` | `/api/integrations/{key}/alertmanager` | Alertmanager v4 webhook receiver for the key's team. `401` for an unknown key | - -This is the only way in. The pre-teams `POST /api/alertmanager/webhook` took no -credential at all — anything able to reach the port could open an incident — -and was removed in v0.13.0 once senders had moved onto keys. - -### Teams - -**owner** below means an owner of that team, a system administrator (who -passes every one of these without being a member), or that team's own -team-scoped [service account](#service-accounts) — including membership and -invites, technically, though no automation this scope was designed for -(a Kubernetes operator's CRDs, see `SERVICE-ACCOUNTS.md`) ever models team -membership or would call those two. See [Authentication](#authentication). -**member** means membership and nothing else: an administrator who is not in -the team gets the same `404` as anybody else. - -| Method | Path | Who | Description | -|---|---|---|---| -| `GET` | `/api/teams` | any | The caller's own teams, each with their role | -| `POST` | `/api/teams` | any | Create a team `{"name"}`; a human creator becomes its first owner. An instance-scoped [service account](#service-accounts) may also create one, and it gets no owner at all — expected for a team an operator is about to hand a team-scoped credential to, not an orphaned team a human made | -| `PUT` | `/api/teams/{teamID}` | **owner** | Rename it `{"name"}`. `409` if the name is taken | -| `DELETE` | `/api/teams/{teamID}` | **owner** | Delete a team and everything under it. `409` while it has open incidents | -| `GET` | `/api/teams/{teamID}/members` | member | Who is in the team, with `status` (`oncall` if the rota has them today, `unpageable` when a page to them would go nowhere — even if they are on call — else `reachable`), `on_call`, `next_shift` (first rota day after today), `pageable` and `problem` (`has no ntfy topic` / `account is disabled`; never the topic itself) and `last_active_at` (their newest session or API-key use). Every member sees the same list | -| `POST` | `/api/teams/{teamID}/members` | **owner** | Add a member, or change their role `{"user_id","role"}`. `409` when it would demote the last owner, or the membership is managed by single sign-on | -| `DELETE` | `/api/teams/{teamID}/members/{userID}` | **owner** | Remove a member. `409` for the last owner, or a membership managed by single sign-on | -| `GET` | `/api/teams/{teamID}/oidc-groups` | member | Which groups control this team's membership: `{"member_group","owner_group"}`. An empty string means no group grants that role here | -| `PUT` | `/api/teams/{teamID}/oidc-groups` | **owner** | Set them. An empty string clears a binding | -| `GET` | `/api/teams/{teamID}/integrations` | member | List integrations. Never returns keys. Each carries `status` (`active` if its key posted within 24h, `quiet` if it has but not lately, `never`), `last_used_at` (last webhook, usable or not), `last_alert_at` (when an alert last arrived on it) and `alerts_24h` (distinct alerts it refreshed in the last day). Alerts delivered before the source was recorded (migration 010) have none, so the last two fill in as Alertmanager re-sends them | -| `PATCH` | `/api/teams/{teamID}/integrations/{integrationID}` | **owner** | Rename `{"name"}`. The key does not change | -| `POST` | `/api/teams/{teamID}/integrations` | **owner** | Mint an integration `{"name","kind"}` — key and URL shown once | -| `DELETE` | `/api/teams/{teamID}/integrations/{integrationID}` | **owner** | Revoke an integration. Alerts it delivered stay, unattributed | -| `GET` | `/api/teams/{teamID}/invites` | **owner** | The team's invite links, with their uses and expiry. Never the tokens | -| `POST` | `/api/teams/{teamID}/invites` | **owner** | Mint one `{"role","max_uses"}` — the full URL is returned once | -| `DELETE` | `/api/teams/{teamID}/invites/{inviteID}` | **owner** | Revoke a link before it expires | -| `GET` | `/api/teams/{teamID}/escalation` | member | The team's [escalation ladder](#escalation) `{repeat_count, fallback_topic, levels[], last_escalated_at?, last_escalated_incident_id?}`. Empty levels means the team has none. Each level also carries `status` (`ready`, `escalating` when an unanswered incident has climbed to it, `unreachable` when nobody on it could be woken), `waiting` (ids of the open incidents on it) and, per target, `username` (who it means today — the person on call, for a rota target), `reachable` and `problem`. The extra fields are output only; `PUT` takes the plain shape | -| `PUT` | `/api/teams/{teamID}/escalation` | **owner** | Replace it wholesale. `400` for a level with no targets or no timeout — a rung that pages nobody is a silence with a number on it | -| `GET` | `/api/teams/{teamID}/deadman/switches` | member | The team's [dead man's switches](#dead-mans-switch), each `{id, name, matcher, timeout_seconds, severity, status, last_heartbeat_at, last_triggered_at, open_incident_id, sources[]}`. `status` is `healthy`, `dead` or `dormant`; `sources` has one entry per heartbeat fingerprint. Empty when the team watches nothing | -| `POST` | `/api/teams/{teamID}/deadman/switches` | **owner** | Add one: `{name?, matcher, timeout_seconds, severity?}`. `400` when the matcher names no `alertname` or holds several, or the timeout is not positive — a switch that silently watches nothing is the failure this feature exists to prevent | -| `PUT` | `/api/teams/{teamID}/deadman/switches/{switchID}` | **owner** | Replace one in place, same body and validation as create. Its id is unchanged — for an automated caller reconciling a spec change, unlike delete-and-recreate | -| `DELETE` | `/api/teams/{teamID}/deadman/switches/{switchID}` | **owner** | Stop watching. An incident it opened stays open. `404` for a switch of another team | - -### Notifications - -| Method | Path | Description | -|---|---|---| -| `POST` | `/api/notify/ack/{token}` | Acknowledge an incident from a push notification's Acknowledge button. No auth: the token in the path is the credential — one incident, one action, 24 hours, idempotent. Must stay publicly reachable | - -### Incidents - -| Method | Path | Description | -|---|---|---| -| `GET` | `/api/incidents` | List incidents. Filters: `?status=triggered\|acknowledged\|resolved`, `?severity=`, `?assigned_to=`, `?archived=true`, `?snoozed=true`, `?from=YYYY-MM-DD`, `?to=YYYY-MM-DD`, `?sort=severity`, `?cluster=`, `?limit=` (default 50, max 500) | -| `GET` | `/api/incidents/clusters` | The distinct `cluster` values on the caller's incidents from the last 90 days, sorted (`?team_id=` narrows it). An empty array when nothing carries the label | -| `GET` | `/api/incidents/{id}` | Get single incident, with its alerts inline | -| `GET` | `/api/incidents/{id}/alerts` | Alerts under this incident | -| `GET` | `/api/incidents/{id}/timeline` | Full event history, chronological | -| `POST` | `/api/incidents/{id}/acknowledge` | Acknowledge (stamps authed user + time) | -| `DELETE` | `/api/incidents/{id}/acknowledge` | Clear acknowledgement, back to `triggered` | -| `POST` | `/api/incidents/{id}/resolve` | Close by hand — **terminal**, see above | -| `POST` | `/api/incidents/{id}/assign` | Reassign `{"user_id"}` | -| `POST` | `/api/incidents/{id}/snooze` | Hide until `{"until": RFC3339}` or `{"duration": "2h"}` | -| `DELETE` | `/api/incidents/{id}/snooze` | Un-snooze | -| `POST` | `/api/incidents/{id}/archive` | Archive (hides from the default list) | -| `DELETE` | `/api/incidents/{id}/archive` | Un-archive | -| `POST` | `/api/incidents/{id}/notes` | Add a note `{"content"}` | -| `DELETE` | `/api/incidents/{id}/notes/{eventID}` | Delete own note | - -With no `?status=` filter, `GET /api/incidents` returns **open** incidents only — -the queue an on-call person wants. Currently snoozed and archived incidents are -excluded unless asked for. Actions that only make sense on an open incident -return `409` once it is resolved. - -Notes are ordinary timeline events of type `note`; only they are deletable, and -only by their author. The rest of the timeline is a record of what happened. - -#### The incident object - -| Field | Type | Notes | -|---|---|---| -| `id` | integer | Server-assigned | -| `group_key` | string | Alertmanager's `groupKey` — opaque, treat as an identifier | -| `title` | string | Rendered from `groupLabels` | -| `group_labels` | object | String→string, as sent by Alertmanager | -| `status` | string | `"triggered"`, `"acknowledged"` or `"resolved"` | -| `severity` | string | *optional* — high-water mark across the incident's alerts; never lowered | -| `triggered_at` | timestamp | When the incident opened | -| `acknowledged_by_id` / `acknowledged_by` / `acknowledged_at` | | *optional* — user id, username, time | -| `assigned_to_id` / `assigned_to` | | *optional* — user id, username | -| `snoozed_until` | timestamp | *optional* — a value in the past reads as not snoozed | -| `resolved_at` | timestamp | *optional* | -| `resolution_source` | string | *optional* — `"alerts"`, `"manual"` or `"recovered"` | -| `archived_at` | timestamp | *optional* | -| `alerts` | array | Only on `GET /api/incidents/{id}` | - -Treat `resolution_source` as an open set, as with the alert field of the same -name: degrade unknown values to "resolved, reason unknown". - -#### The timeline event object - -| Field | Type | Notes | -|---|---|---| -| `id` | integer | | -| `incident_id` | integer | | -| `type` | string | See below — treat as an open set | -| `user_id` / `username` | | *optional* — absent when the server acted rather than a person | -| `alert_id` | integer | *optional* — the alert an `alert_added` / `alert_resolved` event refers to | -| `detail` | string | *optional* — the note body, the snooze deadline, etc. | -| `created_at` | timestamp | | - -Types written today: `triggered`, `alert_added`, `alert_resolved`, -`acknowledged`, `unacknowledged`, `assigned`, `archived`, `unarchived`, `snoozed`, -`unsnoozed`, `resolved`, `note`, `notified`, `notify_failed`, `deadman_silent`. On an -`assigned` event `user_id` is the **assignee**, not the actor; the actor is in -`actor_user_id`/`actor_username` or `actor_service_account_id`/`actor_service_account_name` -(absent on assignments made before they were recorded). New types may be added; render -unknown ones generically rather than dropping them. - -On `notified` and `notify_failed`, `detail` carries the notification kind -(`triggered` | `reminder` | `resolved`), and on a failure the reason after it. -`user_id` is who was paged — absent means the page went to the shared fallback -topic and so belongs to nobody. The topic itself is never written to the -timeline: it is a shared secret with the ntfy server, and every API key can read -this. - -### Alerts - -Alerts are read-only. Everything a person does happens on the incident. - -| Method | Path | Description | -|---|---|---| -| `GET` | `/api/alerts` | List alerts. Filters: `?status=firing\|resolved`, `?name=`, `?incident_id=`, `?archived=true`, `?from=YYYY-MM-DD`, `?to=YYYY-MM-DD`, `?limit=` (default 50, max 500) | -| `GET` | `/api/alerts/{id}` | Get single alert | - -Archived alerts are hidden from `GET /api/alerts` unless `?archived=true` is -passed; alert archiving is automatic housekeeping by the sweeper, not a user -action. Resolved alerts carry `resolution_source`: `"alertmanager"` for a real -resolved webhook, `"expiry"` when the sweeper inferred it (see -[Stale alert expiry](#stale-alert-expiry)), `"deadman"` for a heartbeat declared -dead (see [Dead man's switch](#dead-mans-switch)). - -#### The alert object - -Returned by `GET /api/alerts` (as an array) and `GET /api/alerts/{id}`. -Timestamps are RFC 3339 in UTC. Fields marked *optional* are omitted entirely -when unset, so clients must treat them as nullable. - -| Field | Type | Notes | -|---|---|---| -| `id` | integer | Server-assigned; stable for the life of the row | -| `fingerprint` | string | Alertmanager's fingerprint — the upsert key | -| `name` | string | From the `alertname` label | -| `status` | string | `"firing"` or `"resolved"` | -| `labels` | object | String→string, as sent by Alertmanager | -| `annotations` | object | String→string, as sent by Alertmanager | -| `starts_at` | timestamp | When the alert instance began, **per Prometheus** | -| `ends_at` | timestamp | *optional* — absent while no end is known | -| `generator_url` | string | Link back to the originating Prometheus | -| `received_at` | timestamp | When the server last accepted a webhook for this alert — see below | -| `incident_id` | integer | *optional* — the most recent incident this alert belongs to | -| `resolution_source` | string | *optional* — `"alertmanager"`, `"expiry"` or `"deadman"` | -| `archived_at` | timestamp | *optional* — set while archived | - -##### `received_at` is a liveness heartbeat - -`starts_at` comes from Prometheus and **never changes** for the lifetime of an -alert instance. It says when the problem began, not whether it is still -happening — an alert that started twelve days ago looks identical whether -Alertmanager refreshed it a minute ago or went silent a week ago. - -`received_at` is the field that answers "is this still live". It is set to the -server's clock on **every accepted webhook** for that fingerprint, including the -unchanged firing notifications Alertmanager re-sends every `repeat_interval`. -Clients may rely on this: - -- **A firing alert whose `received_at` is advancing is still being refreshed.** - Stale-dating it against `repeat_interval` is a valid liveness check, and it is - what the built-in sweeper does (see - [Stale alert expiry](#stale-alert-expiry)). -- **`received_at` tracks accepted payloads, not delivery attempts.** A retry - that describes an older instance than the stored one is discarded, and a - discarded payload does not move `received_at`. -- **It stops advancing once the alert resolves,** because Alertmanager stops - re-sending. On an alert resolved by the sweeper - (`"resolution_source": "expiry"`) it therefore marks the last time - Alertmanager was actually heard from, which is earlier than `ends_at`. - -`GET /api/alerts` is ordered by `received_at` descending — most recently -refreshed first — and the `?from=` / `?to=` filters on both the alert and stats -endpoints select on `received_at`, not `starts_at`. - -##### `resolution_source` says how much to trust `ends_at` - -An alert can leave the firing state two ways, and `resolution_source` records -which happened. Clients may rely on this: - -- **Absent while firing.** It is set only on resolve, and a re-fire under the - same fingerprint clears it again, so its presence always agrees with - `"status": "resolved"`. -- **`"alertmanager"` — a real resolved webhook arrived.** `ends_at` is the end - time Alertmanager reported. It is an observed value and can be displayed as - fact. -- **`"expiry"` — the sweeper inferred the resolve** because Alertmanager stopped - refreshing the alert (see [Stale alert expiry](#stale-alert-expiry)). Nothing - ever reported an end, so **`ends_at` is approximate**: it is either the stale - `endsAt` watermark from the last notification, or — when that notification - carried none — the time the sweep ran, which lags the last real contact by up - to `TERDUT_STALE_AFTER` plus a sweep interval. Treat it as "no later than", - not as when the problem stopped. - - On these alerts `received_at` is the more truthful signal: it marks the last - time Alertmanager was actually heard from. Surfacing the distinction is - worthwhile, since `"expiry"` can also mean the alert is still firing and the - notification path broke. - -- **`"deadman"` — a heartbeat was declared dead** (see - [Dead man's switch](#dead-mans-switch)). Like `"expiry"`, an inference from - silence rather than an observed end, so `ends_at` is approximate — but a much - tighter one, bounded by `TERDUT_DEADMAN_TIMEOUT`. It is also the one resolution - a re-fire under the same `starts_at` can undo, since the switch coming back is - exactly the evidence that the inference was wrong. - -Treat the value as an open set and tolerate ones you do not recognise — new -sources may be added, and unknown values should degrade to "resolved, reason -unknown" rather than being rejected. - -### On-call schedule - -| Method | Path | Description | -|---|---|---| -Each team keeps its own rota, so two teams can have two different people on call -on the same day. The person taking a shift has to be in the team — paging -somebody who cannot open the incident is worse than paging nobody. - -| Method | Path | Who | Description | -|---|---|---|---| -| `POST` | `/api/teams/{teamID}/schedule` | **owner** | Assign user to dates `{"user_id", "dates":["YYYY-MM-DD",...], "replace"}` — all-or-nothing | -| `GET` | `/api/teams/{teamID}/schedule` | member | List entries. Filters: `?from=YYYY-MM-DD`, `?to=YYYY-MM-DD` | -| `DELETE` | `/api/teams/{teamID}/schedule/{id}` | **owner** | Remove schedule entry | -| `GET` | `/api/schedule/current` | any | Who is on call today (UTC) in **every** team the caller is in — one entry per team, `[]` when nobody anywhere | - -### Statistics - -Every figure counts the caller's own teams only: a report that counted other -teams' incidents would leak their volume, and their alert names through the -top-alerts list, and would not be a number about the reader's work anyway. - -All stat endpoints accept optional `?from=YYYY-MM-DD` and `?to=YYYY-MM-DD`, and exclude archived rows to match the default list views. Alert stats filter on `received_at`; incident stats filter on `triggered_at`. - -| Method | Path | Description | -|---|---|---| -| `GET` | `/api/stats/incidents` | `{total, triggered, acknowledged, resolved, mtta_seconds, mttr_seconds}` | -| `GET` | `/api/stats/alerts` | `{total, firing, resolved}` counts | -| `GET` | `/api/stats/alerts/top` | Most frequent alert names. `?limit=` (default 10, max 100) | -| `GET` | `/api/stats/alerts/by-hour` | Count per hour-of-day (UTC), all 24 slots returned | -| `GET` | `/api/stats/alerts/by-day` | Count per day-of-week, all 7 slots with names returned | - -`mtta_seconds` (time to acknowledge) and `mttr_seconds` (time to resolve) are -averages over incidents that have actually been acknowledged or resolved, and are -**null** until there are any — null means "no data", not zero. - ---- - -## Upgrading to teams - -Everything that existed before teams moves into one team called **Default**, and -every existing user becomes an owner of it. The upgrade is a no-op for the -people using it: the same queue, the same schedule, the same incidents, with a -name on them. - -What changes, and will need attention: - -- **Alert ingestion moved.** Mint a key with - `POST /api/teams/{teamID}/integrations` and point Alertmanager at the URL it - returns. In v0.12.0 the old `POST /api/alertmanager/webhook` still worked, - deprecated, routing everything to the oldest team; **v0.13.0 removes it**, so - upgrade straight from v0.11.x to v0.13.0 only after the senders are moved. -- **The schedule endpoints moved** under `/api/teams/{teamID}/schedule`, and - editing the rota is now an owner's job. `GET /api/schedule/current` stayed - where it was but now returns an **array** — one entry per team with somebody - on call — instead of a single object or a 404. This is a breaking API change - for anything that reads it, terdut-tui included. -- **Uniqueness is per team now.** Two teams can legitimately see the same alert - fingerprint, the same Alertmanager groupKey, and put somebody on call on the - same date. - -**Dead man's switches moved too.** `TERDUT_DEADMAN_MATCHERS`, `_TIMEOUT` and -`_SEVERITY` are no longer the setting; they are the default each existing team -is seeded with at startup, after which an owner manages them per team through -`/api/teams/{teamID}/deadman/switches` and a redeploy never overwrites that. - -Nothing else about an incident changes, and incidents never move between teams: -an alert belongs to whichever team's key it arrived on. - -## Upgrading to roles - -Before this release every authenticated caller could create and delete users, -set anybody's password and mint anybody's API keys. That is now the -administrator flag, and the migration **makes every existing user an -administrator** — they already held those powers, so nobody's access changes on -upgrade and demotion is a deliberate act afterwards. Promoting only the first -user would have silently stripped the rest, and could leave an install whose -only administrator is an account nobody has a password for. - -Users created after the upgrade are not administrators. Hand the flag out with: - -```bash -curl -X PUT https://terdut.example.com/api/users/7/admin \ - -H "Authorization: Bearer $TERDUT_API_KEY" \ - -H 'Content-Type: application/json' \ - -d '{"is_admin": true}' -``` - -Nothing in the API changed shape, so terdut-tui needs no new version — but a -non-administrator now gets `403` where a `200` used to come back. - -## Upgrading from SQLite - -Versions up to v0.10.2 stored everything in a SQLite file. From v0.11.1 the server needs -`TERDUT_DB_DSN` and keeps nothing on disk. - -The copy was done by `scripts/sqlite-to-postgres.go`, which **was deleted in v0.13.0** along -with the SQLite driver it was the last user of. It is still in the history — check out the -`v0.12.0` tag to get it: - -```bash -git show v0.12.0:scripts/sqlite-to-postgres.go > sqlite-to-postgres.go -``` - -The cutover is ordered, and the server must not be running while the copy happens: stop the -old version, let the new binary build the schema against an empty Postgres, run the script -with `-sqlite` and `-dsn`, then start the new version for good. On Kubernetes step three runs -as a Job with the same image against the PVC before it is removed. - -The copy preserves every id, so incidents keep their numbers and the timeline, alert -membership, outbox and ack tokens all still point where they did. It refuses a target that -already has rows, so a second run cannot double-insert. - -## Upgrading to incidents - -The incidents release moves the workflow off alerts, which is a **breaking API -change**. These endpoints are gone: - -| Removed | Replacement | +| | | |---|---| -| `POST`/`DELETE` `/api/alerts/{id}/acknowledge` | `POST`/`DELETE` `/api/incidents/{id}/acknowledge` | -| `POST`/`DELETE` `/api/alerts/{id}/archive` | `POST`/`DELETE` `/api/incidents/{id}/archive` (alert archiving is now sweeper-only) | -| `GET`/`POST` `/api/alerts/{id}/comments` | `GET /api/incidents/{id}/timeline`, `POST /api/incidents/{id}/notes` | -| `DELETE /api/alerts/{id}/comments/{commentID}` | `DELETE /api/incidents/{id}/notes/{eventID}` | +| [Deployment](./docs/deployment.md) | Docker, Helm chart, database, backups, the operator | +| [Configuration](./docs/configuration.md) | Environment variables and settings | +| [Alertmanager configuration](./docs/alertmanager.md) | Routes, keys, webhooks | +| [Alerts and incidents](./docs/incidents.md) | Correlation, lifecycle, on-call assignment | +| [Single sign-on](./docs/single-sign-on.md) | OIDC and the terminal device flow | +| [API reference](./docs/api.md) | Every endpoint | +| [Development and releasing](./docs/development.md) | Tests, CI gate, release pipeline | -The alert object also drops `acknowledged_by_id`, `acknowledged_by` and -`acknowledged_at`, and gains `incident_id`. +## Related -Migration `008_incidents.sql` runs automatically on start and preserves existing -data: every alert gets a backfilled incident carrying its acknowledgement, and -comments become timeline notes. Backfilled incidents have a `group_key` of -`backfill:` — there is no historical `groupKey` to correlate on, so -they are one-per-alert rather than grouped. +- [terdut-tui](https://git.ryuvia.com/niklas/terdut-tui): a terminal client for the same server. +- [terdut-operator](https://git.ryuvia.com/niklas/terdut-operator): a Kubernetes operator that + runs the server and manages teams, escalation, switches and alert sources as objects. -Nothing about the two documented alert contracts changes: `received_at` is still -advanced on every accepted webhook, and `resolution_source` still means what it -did. +## License -## Upgrading to dead man's switches - -Dead man's switch handling is **on by default**, watching `alertname=Watchdog` -with a 15 minute timeout. If you already route `Watchdog` to this server, the -behaviour of that alert changes on upgrade, in both directions: - -- it stops opening incidents when it arrives, and -- it starts opening one when it stops arriving. - -**Check your `repeat_interval` before upgrading.** The switch pages whenever a -heartbeat has not been refreshed within `TERDUT_DEADMAN_TIMEOUT`, so a `Watchdog` -route inheriting a 4h or 12h `repeat_interval` will page constantly against the -15 minute default. Either give the heartbeat -[its own fast route](#alertmanager-configuration) — the point of the feature — or -set `TERDUT_DEADMAN_TIMEOUT` above your current `repeat_interval` until you have. -`TERDUT_DEADMAN_TIMEOUT=0` turns the whole thing off. - -There is no migration and no schema change. An existing open incident from a -`Watchdog` that arrived under the old behaviour is unaffected; resolve it by hand. - ---- - -## Development - -```bash -make test-db # start a local Postgres for the tests (podman or docker) -make test # run all tests -go build ./... # compile all packages -go run ./cmd/terdut # run locally (needs TERDUT_DB_DSN) -``` - -The tests need a real Postgres, because the server does — there is no in-memory Postgres the -way there was an in-memory SQLite. `TERDUT_TEST_DSN` says where it is, `make test-db` starts -one on port 5433 and prints the DSN, and `make test-db-stop` removes it. Each test gets its -own schema on that server, so tests cannot see each other's rows. An unset `TERDUT_TEST_DSN` -fails the suite rather than skipping it: a run that quietly tests nothing is worse than one -that does not run. - -`make fmt lint test helm-lint` is the gate. It mirrors `.gitea/workflows/ci.yaml` step for -step, so a green run here means a green pipeline — with one deliberate exception: `make test` -adds `-race`, which CI does not. The sweeper, the notifier goroutine and the dead man's switch -sweep all run concurrently against the same database, and a race between them would surface as -a flaky incident in production rather than as a red build. - -The web UI lives in `internal/web/static/` as plain HTML, CSS and ES modules, -embedded into the binary with `go:embed`. It has no build step and no npm, so -editing a file and restarting the server is the whole loop. - -## Releasing - -``` -push or PR → ci.yaml gofmt, go vet, go test -race - govulncheck, gitleaks - helm lint + render -push tag vX.Y.Z → release.yaml the same gate, then publish: - git.ryuvia.com/niklas/terdut-server:vX.Y.Z - oci://git.ryuvia.com/niklas/terdut-server X.Y.Z - then trivy-scan the pushed image -PR to Ryuvia/charts → bump the wrapper chart to X.Y.Z; on merge - Flux reconciles and the release rolls out -``` - -Both artifacts go to the **personal** Gitea namespace rather than `ryuvia`, because Gitea -scopes package visibility to the owner with no per-package override — so `ryuvia/*` is private -because the org is. Publishing to `niklas` keeps them anonymously pullable, which is why no -pull secret is needed in the cluster. Same reasoning, and the same choice, as riksdata and -rd-web. - -Saying **"Release"** runs all three rows: the `release` skill commits, pushes, tags, waits for -the pipeline, and opens the `Ryuvia/charts` PR, stopping before the merge. See -`~/.claude/skills/release/`, or `.release.conf` here for this repo's part of it. - -The chart is published **only** from the tag, by the `chart` job. There used to be a second -publisher on every `charts/**` push to main, and the two raced for the same chart version with -different answers — chart 0.9.0 went out reading `appVersion: "latest"` that way. One -publisher, triggered by the tag (`766f439`). The cost is that a chart-only change has no -version of its own and rides the next app tag. - -Both workflows are thin drivers over the Makefile: `ci.yaml` runs `make fmt lint test` and -`make helm-lint`, `release.yaml` adds `make binaries`, `make push`, `make helm-package` and -`make helm-push`. That is deliberate — it is what makes a green local gate and a green -pipeline the same code rather than two descriptions of it, and it is how riksdata and rd-web -have always worked. - -`make push` builds and pushes in one step, unlike those two, because the image is -`linux/amd64,linux/arm64` and buildx cannot load a multi-platform result into the local image -store. `make build` stays single-platform and local-only. Both refuse `VERSION=dev`: -publishing is one command, so it is also one command to run by accident. Publishing happens -by pushing a tag. - -Two things the release process needs to know about this repo: - -- **The image scan runs after publishing**, like riksdata's and rd-web's: trivy cannot read - a locally built image on this runner, so it pulls the pushed one. A red `scan-image` means - do not bump the wrapper chart to that version — it cannot unpublish anything. The image is - `FROM scratch`, so trivy sees exactly one target, the Go binary and its module graph. -- **The wrapper chart's `values.yaml` has two `tag:` lines** — the app image and the python - backup sidecar — so `chart-bump` is given `--image` to say which one moves. The sidecar is - on its way out with SQLite: once the wrapper chart drops it and declares a `postgresql` CR - instead, there is one `tag:` line again, and `--image` becomes belt and braces. - -The wrapper chart must have **its own `version:` bumped in the same commit**. Flux reconciles -with `reconcileStrategy: ChartVersion`, so a chart whose version did not change produces no -new artifact and the change is never deployed — with no error anywhere. +See [LICENSE](./LICENSE). diff --git a/SERVICE-ACCOUNTS.md b/SERVICE-ACCOUNTS.md index 2d264c0..ff768a9 100644 --- a/SERVICE-ACCOUNTS.md +++ b/SERVICE-ACCOUNTS.md @@ -1,263 +1,60 @@ -# Service accounts: a scoped, non-human credential type +# Service accounts -This is a design note for a feature, not an implementation plan — it exists to -propose the shape before writing code. It's raised directly by `terdut-operator` -(a separate repo, no shared code — see its `DESIGN.md` §6, §9, §13), which needs -a credential for unattended, repeatable API access and currently has no good one -available. Anything automating terdut-server long-term (this operator, CI, future -integrations) hits the same gap, so this is written as a general primitive, not -operator-specific. +A non-human credential for automation (terdut-operator, CI, scripts). It is not a +`users` row: no password, no `is_admin`, no OIDC identity, so it can never be +pulled into login or group sync, and it is never mistaken for a person in an +audit trail. The bearer token has the same shape as an API key (SHA-256 hash +stored, raw value shown once), prefixed `tdsa_`. -## The problem +## Scopes -terdut-server has two credential types today, and neither fits "an unattended -process that manages teams/schedules/policies on someone's behalf": +- **instance** — acts as owner of every team's *configuration* (rename, OIDC + groups, escalation, dead man's switches, integrations, members, delete) and may + create teams. It is not a member of any team, so it reads no incidents or + queue. It is never an administrator: user management and + `/api/admin/settings` stay human-only. +- **team** — acts as owner of exactly one team, through a single synthetic + membership. It may also mint another service account for its own team. -- **User API keys** (`api_keys`, `internal/api/users.go`) are always tied to a - real `users` row and carry that user's full rights — every team they're a - member of, their admin flag if set. There's no `kind`/`service` marker - distinguishing "a human's personal automation key" from "a login session," and - no way to mint one scoped to less than the full user. -- **Integration keys** (`integrations`, `internal/api/*teams*.go`) are team-scoped, - but narrowly: they authenticate exactly one inbound Alertmanager webhook call - (`POST /api/integrations/{key}/alertmanager`) and nothing else. They're not a - general management-API credential and shouldn't become one — overloading a - narrow, one-way ingestion credential with broad read/write access would weaken - the one property that makes it safe to embed in an Alertmanager config today. +An account has many keys, so rotating is "mint a new key, revoke the old one" +without losing the account's identity or history. -The result: any automation that needs to create teams, set escalation policies, -manage dead-man switches, or rotate integration keys has to hold a real human -admin's or team owner's API key. That key is exactly as powerful as that person -logging in — full team access, and full instance access if they're an admin. -`terdut-operator`'s design ran directly into this (its DESIGN.md §6): its -described bootstrap/rotation flow assumed a repeatable, identity-scoped way to -get a credential, and `/api/bootstrap`'s actual behavior (single-shot per -install, gated on `COUNT(*) FROM users`, confirmed via `internal/api/users.go` -and `charts/terdut-server/templates/bootstrap-job.yaml`) doesn't provide one — -it mints exactly one founding admin, once, ever. +## Endpoints -## Goals +- `POST /api/service-accounts` `{name, scope, team_id}` — returns the account and + its first key. An instance-scoped account is granted by a human administrator; + a team-scoped one by an administrator, that team's owner, or an instance-scoped + account. +- `GET /api/service-accounts?name=` — look one up by name. +- `POST /api/service-accounts/{id}/keys`, `DELETE .../keys/{keyID}` — mint or + revoke a key. An instance-scoped account may manage any team-scoped account's + keys, and any account may manage its own. -- A credential type that isn't a human: doesn't touch OIDC group sync, login, - session, or the `is_admin`/account-management semantics that come with a real - `users` row. -- Two scopes matching the two shapes automation actually needs: instance-wide - (create/list teams — what a server-owning controller needs) and team-scoped - (manage one team's escalation policy, dead-man switches, integrations, - schedule, OIDC group bindings — what a per-team controller or integration - needs). -- Repeatable issuance and rotation — unlike `/api/bootstrap`, callable more than - once, by anything that already holds admin rights, without destroying and - recreating state to get a fresh credential. -- Visibly distinct from a human in every place identity shows up (audit trails, - timeline entries, UI attribution) — a service account acting on a team should - never be indistinguishable from a person. +## Seeding the operator's account -## Non-goals +`TERDUT_OPERATOR_KEY` (at least 32 characters) creates the instance-scoped account +`terdut-operator` if missing and replaces its `seed` key with this value at every +start (`internal/api/operator_key.go`). The deployer generates the key and +nothing has to call `/api/bootstrap` for it; rotating is a restart with a new +value. With `TERDUT_OPERATOR_MODE` on, configuration writes by humans are refused +and a service account of either scope passes. -- Not a general OAuth2/OIDC client-credentials flow — this is a bearer-token - primitive matching the shape `api_keys` already uses (SHA-256 hash stored, - raw key shown once at creation), not a new auth protocol. -- Not replacing integration keys — those stay as the narrow, one-way webhook - credential they are today. -- Not modeling per-endpoint or per-verb permissions within a scope — `instance` - and `team` are the only two scopes for now; finer-grained scoping is future - work if a real need shows up. +## How it is enforced -## Proposed shape +Every request resolves to one `Caller` (`internal/api/caller.go`): a human +(session or API key) or a service account. -### Schema +- `Caller.IsAdmin()` is true only for a human administrator. `AdminOnly` and + `requireSelfOrAdmin` key on it alone; do not widen them — each time a gap came up + the fix was a narrower purpose-built capability instead. +- `Caller.IsInstanceServiceAccount()` is true only for an instance-scoped account, + never for a human. `requireTeamOwner` and `callerOwnsTeam` admit it for any team. +- `Caller.Role(teamID)`/`TeamIDs()` are a human's memberships or a team-scoped + account's single owner membership; instance scope has none. +- `Caller.AsHuman()` is what a handler must call when it needs a real `user_id`; + handlers meant for people answer 403 to a service account instead of writing a + zero id. -```sql -CREATE TABLE service_accounts ( - id BIGSERIAL PRIMARY KEY, - name TEXT NOT NULL UNIQUE, -- e.g. "terdut-operator" - scope TEXT NOT NULL CHECK (scope IN ('instance', 'team')), - team_id BIGINT REFERENCES teams(id) ON DELETE CASCADE, - -- team_id required iff scope = 'team'; NULL iff scope = 'instance' - created_by BIGINT REFERENCES users(id), - created_at TIMESTAMPTZ NOT NULL DEFAULT now() -); - -CREATE TABLE service_account_keys ( - id BIGSERIAL PRIMARY KEY, - service_account_id BIGINT NOT NULL REFERENCES service_accounts(id) ON DELETE CASCADE, - key_hash TEXT NOT NULL UNIQUE, - name TEXT NOT NULL, -- e.g. "initial", "2026-Q4-rotation" - created_at TIMESTAMPTZ NOT NULL DEFAULT now(), - last_used_at TIMESTAMPTZ -); -``` - -Deliberately not a `users` row: no `password_hash`, no `is_admin`, no -`user_identities` linkage, so it's structurally impossible for a service account -to be pulled into OIDC group sync or password login. Multiple keys per account -(mirroring `api_keys`' existing one-user-many-keys shape) so rotation is "mint a -new key, revoke the old one," not "recreate the account." - -### Endpoints - -- `POST /api/service-accounts` — instance-scope/admin-only. Body: - `{"name": ..., "scope": "instance"|"team", "teamID": ... }` (teamID required - iff scope=team, and caller must be that team's owner or a system admin). - Returns the account plus its first raw key (shown once, same pattern as - `POST /api/users/{id}/api-keys`). Safe to call again with the same `name` — - see "idempotent lookup" below — unlike `/api/bootstrap`, which is inherently - one-shot by design (it's answering "does any user exist yet," a question with - no analogue once one already does). -- `POST /api/service-accounts/{id}/keys` — mint an additional key on an existing - account (self-service-equivalent: instance admin for `instance` scope, team - owner or system admin for `team` scope). Enables rotation without recreating - the account or losing its identity/audit history. -- `DELETE /api/service-accounts/{id}/keys/{keyID}` — revoke one key, mirroring - `DELETE /api/users/{id}/api-keys/{keyID}`. -- `GET /api/service-accounts?name=` — look up an existing account by name. - This is what turns "I tried to create my account and got a conflict" into a - normal flow instead of an error: a controller that expects to have already - registered itself calls this first, and only falls through to `POST` if - nothing comes back. - -### Auth middleware - -**Revised** (this section originally described an aspiration that didn't -match what shipped — `TEAM-LOOKUP.md` already caught one instance of that, -and a fuller audit found three more; this is the corrected, as-built -description, not the original proposal). - -`internal/api/middleware.go`'s dual resolution (`Authorization: Bearer` → -`apiKeyUser()`, or session cookie → `sessionUser()`) and the service-account -path (`serviceAccountFor()`) both resolve into one `Caller` type -(`internal/api/caller.go`), not two parallel, un-unified context -representations the way an earlier version of this server kept them. Every -authorization predicate reads `Caller`'s methods: - -- `Caller.IsAdmin()` — true **only** for a human system administrator, never - for a service account of either scope, under any circumstance. `AdminOnly` - and `requireSelfOrAdmin` key on this alone — user management - (`POST /api/users`, `PUT /api/users/{id}/admin`, etc.) and - `GET/PUT /api/admin/settings` stay human-only, forever. The original text - here claimed an instance-scoped service account satisfies `AdminOnly` "for - team-creation/listing purposes" — that was never true of the shipped code - (`TEAM-LOOKUP.md` caught the listing half; the creation half was always a - separate, bespoke check in `handleCreateTeam`, not `AdminOnly` itself) and - is not being made true now. Don't widen `AdminOnly`: every time this has - come up, the fix has been a narrower, purpose-built capability instead - (`?name=` lookups for teams and service accounts; now - `terdut-operator`'s own invite-minting feature for the one real gap this - boundary left — how a human ever gets a first login on a no-OIDC, - operator-managed install. See the bottom of "What this unblocks.") -- `Caller.IsInstanceServiceAccount()` — true only for an instance-scoped - service account, never for a human (including a human admin). - `handleCreateTeam` uses exactly this: a human creates a team by being a - human (and becomes its owner); an instance-scoped service account creates - one with no human owner at all. The two paths are not interchangeable, so - this predicate deliberately does not also admit a human admin. -- `Caller.Role(teamID)`/`TeamIDs()` — a human's real `team_members` rows, or - a team-scoped service account's single synthetic owner membership - (`serveAsServiceAccount`). This is what makes `requireTeamMember`/ - `requireTeamOwner` treat a team-scoped service account as owner-equivalent - for that one team, with no separate branch needed in either function. -- `Caller.ServiceAccountID()` — used by `OperatorModeBlock` ("any service - account passes") and by `callerMayManageServiceAccount`'s self-rotation - check. -- `Caller.AsHuman()` — the accessor every handler that needs a real - `user_id` to act on behalf of must call and check, instead of reading a - user off context unconditionally. Before the `Caller` type existed, four - handlers did the latter and silently misbehaved for a service-account - caller: `handleMe` and `handleTestNotification` 500'd (a zero-value user id - that matches no row), `handleDismissOnboarding` silently no-op'd (`UPDATE - ... WHERE id = 0` affects nothing, still returns 204), and - `handleCreateInvite` wrote that same zero value into `invites.created_by` - — a real foreign-key violation, not just a wrong answer, since that column - is nullable but was never passed as `nil`. All four now call `AsHuman()` - and return an explicit 403 ("this endpoint is for human accounts only") - or, for the invite case, leave `created_by` `NULL` the same way - `handleCreateServiceAccount` already did for the analogous situation. - -**Team scope is owner-equivalent for every `requireTeamOwner` endpoint, -membership and invites included — by design, not by an unclosed gap.** An -earlier version of this document flagged this as "acknowledged rather than -closed," kept in check only by the social convention that nobody *builds* -automation against those two routes. That convention is retired: -`terdut-operator`'s `TerdutTeam` controller now mints and revokes its own -team's invite link through exactly this capability (its existing -team-scoped credential, `POST`/`DELETE /api/teams/{teamID}/invites`), which -is the real fix for the human-onboarding gap below — not a narrower -carve-out of this capability. `service_accounts_test.go`'s -`TestServiceAccount_TeamScopeManagesItsOwnInvites` pins it. - -**A team-scoped account can also mint another service account scoped to its -own team** (`handleCreateServiceAccount`'s `callerOwnsTeam` branch, which a -team-scoped caller already satisfies for its own team via the synthetic -membership above). Kept, not restricted, for the same reason: a team-scoped -credential is that team's owner's reach, full stop — carving this one -capability out while leaving membership/invites alone would be an arbitrary -asymmetry. Pinned by -`TestServiceAccount_TeamScopeCanMintAnotherAccountForItsOwnTeam`. - -**`callerMayManageServiceAccount` gained the one load-bearing fix this -redesign exists for:** an instance-scoped service account may manage -(mint/revoke a key on) *any* team-scoped account, not only one admin, that -team's human owner, or the account itself. `handleCreateServiceAccount` -already let an instance-scoped caller *create* a team-scoped account for -any team; this closes the gap where adopting or rotating one it didn't just -create in the same call — exactly `terdut-operator`'s documented -adopt-on-409 crash-window recovery (its own `DESIGN.md` §5) — 403'd forever -instead of succeeding (`terdut-operator#3`). Pinned by -`TestServiceAccount_InstanceScopeAdoptsAnExistingTeamScopedAccountsKey`. - -Anywhere identity is recorded for a human (incident timeline -`acknowledged_by`/`assigned_to`, audit-relevant fields), a service-account -caller is still coerced into a bare `user_id` of `0` today — `Caller`'s new -`Identity()` accessor exists for exactly this follow-up, but wiring it in -needs a schema migration (an actor-attribution column distinct from -`user_id`) and is deliberately out of scope here. Tracked separately, not by -this document. - -## What this unblocks - -Directly resolves `terdut-operator` DESIGN.md §6's two broken assumptions: -1. **Bootstrap becomes single-purpose again.** `/api/bootstrap` mints exactly - the founding human admin, once. The operator's actual first-reconcile flow: - call `/api/bootstrap` only on a genuinely empty install; otherwise (or - immediately after, if it won the bootstrap race) call - `GET /api/service-accounts?name=terdut-operator`, and `POST` one if it - doesn't exist yet. From then on the operator never touches `/api/bootstrap` - again. -2. **Rotation becomes real.** `POST /api/service-accounts/{id}/keys` + revoke the - old one — no destructive DB-level workaround, no re-triggering a single-shot - endpoint that can't fire twice. -3. **Cross-namespace credential mirroring is no longer needed at all.** - `terdut-operator`'s current design holds every credential — instance- and - team-scoped alike — privately in the operator's own namespace, never in - the namespace of the CR each one authenticates for; reconciliation happens - entirely inside the operator's controller loop, so no CR owner ever needs - read access to a terdut-server credential regardless of same- or - cross-namespace `serverRef`. Team scoping is still what bounds the blast - radius of any individual credential: a leaked team-scoped key exposes - exactly one team's resources, never the whole server, which is what makes - holding many credentials in one place (the operator's namespace) an - acceptable trade rather than reintroducing the mirrored design's - server-admin-equivalent-everywhere problem. -4. **A human can get a first login on a no-OIDC, operator-managed install — - without ever touching `AdminOnly` or `/api/admin/settings`.** This was - filed as `terdut-server#23` ("no API path to create a human login after - bootstrap") and diagnosed, at the time, as this server needing to let a - service account through `AdminOnly`. It doesn't: the fix lives entirely - in `terdut-operator`, because a team-scoped credential was *already* - owner-equivalent for `POST /api/teams/{teamID}/invites`, and invite - redemption (`POST /api/signup` with an `invite` token) bypasses - `signup_mode` entirely — `terdut-operator` just never grew a feature to - use either fact. Its `TerdutTeam` controller now mints and surfaces one - via its own existing team-scoped credential (`spec.invite`, - `status.inviteSecretRef`, see that repo's own docs), so a human joins a - CRD-managed team by a real invite link, the same way anyone else would. - `terdut-server#23` is closed with this note once that feature ships — its - named routes stay human-only, correctly, not a gap. - -## Suggested sequencing - -Land this before `terdut-operator` implements any bootstrap/credential-handling -code — that code would otherwise be written against the current one-shot, -user-only credential model as a known-temporary workaround, which is wasted -effort on a repo that currently has zero implementation to begin with. +Where a service account acts on an incident (acknowledge, resolve), the +timeline and `acknowledged_by` record it through parallel `*_service_account_id` +columns, never as a user. diff --git a/TEAM-LOOKUP.md b/TEAM-LOOKUP.md deleted file mode 100644 index 67419cd..0000000 --- a/TEAM-LOOKUP.md +++ /dev/null @@ -1,73 +0,0 @@ -# Team lookup for service accounts: closing terdut-operator's create-path crash window - -This is a design note for a feature, not an implementation plan — same posture as -`SERVICE-ACCOUNTS.md`, and raised for the same reason: `terdut-operator`'s `TerdutTeam` -controller (ROADMAP.md Stage 2, a separate repo, no shared code) hit a gap this server has -no answer for yet. - -## The problem - -`POST /api/teams` (`handleCreateTeam`, confirmed against `internal/api/teams.go`) lets an -instance-scoped service account create a team — it has its own explicit -`isInstanceServiceAccount(...)` branch alongside the human-user path, not gated by -`AdminOnly`. If that call succeeds server-side but the caller (`TerdutTeam`'s controller) -crashes before persisting the resulting team ID locally, a retry's `POST` 409s on the name's -unique constraint (confirmed: the `isUniqueViolation` branch in the same handler). - -Recovering from that 409 means looking the team up by name, and nothing today permits that -for a service account: - -- `GET /api/teams` (`handleListTeams`) answers "what teams does the *caller* belong to", via - a `team_members` join keyed on `userFromContext`'s `caller.ID` — confirmed against source. - A service account is never a member of anything, so this always returns empty for one, - regardless of what exists. -- `GET /api/admin/teams` (`handleAdminListTeams`) is gated by `AdminOnly`, and `AdminOnly`'s - actual code (`internal/api/middleware.go`) checks only `userFromContext(...).IsAdmin` — no - branch for a service account at all, confirmed against source. This contradicts - `SERVICE-ACCOUNTS.md`'s own text, which claims "an instance-scoped [service account - satisfies] `AdminOnly` for team-creation/listing purposes" — that claim doesn't match this - endpoint's actual, shipped code. (Team *creation* is fine: `handleCreateTeam` isn't behind - `AdminOnly` at all, it has its own check. Only the listing half of that sentence is wrong.) - -This is exactly the shape of gap `SERVICE-ACCOUNTS.md`'s own `GET /api/service-accounts?name=` -closed for service accounts themselves (confirmed: that endpoint's own comment — -"the name lookup is open to any authenticated caller... what lets a service account find its -own account on the 403 that follows a second POST"). Teams never got the equivalent, because -nothing needed it until an operator started creating them unattended. - -## Goals - -- A service-account-accessible way to look up one team by exact name, mirroring - `GET /api/service-accounts?name=` as closely as possible — same shape, same reasoning, - same low sensitivity of what it discloses. -- No change to today's behavior for an empty/no-name request. - -## Proposed shape - -Extend `GET /api/teams` itself, the same way `handleListServiceAccounts` already branches on -a `?name=` query param, rather than adding a new route: - -- `name` unset (today's behavior, unchanged): the caller's own teams, via `team_members`. -- `name=` set: look up that one team by exact name — a one-or-zero-length array, not - an error on no match, mirroring `GET /api/service-accounts?name=`'s own response shape and - status codes exactly. Deliberately **not** gated by `isInstanceServiceAccount` or - `AdminOnly`: a human caller who's already a member sees this same information in their own - team list regardless, and a non-member learning only that a name is taken — not who's in - the team, not any of its data — is the same low-sensitivity disclosure - `GET /api/service-accounts?name=` already accepts for service-account names. - -## What this unblocks - -Directly resolves the crash-window gap in `terdut-operator`'s `TerdutTeam` controller: on a -409 from `POST /api/teams`, `GET /api/teams?name=` — authenticated with the -same instance-scoped credential that just got the 409 — finds the id, and the controller -proceeds as if its own create had returned it directly. The same adopt-on-409 pattern already -proven for service accounts (that repo's `DESIGN.md` §6 point 1, §5's general rule), not a -new one. - -## Suggested sequencing - -Land this before `TerdutTeam`'s create path is implemented — the same reasoning -`SERVICE-ACCOUNTS.md` gave for its own sequencing: writing that code against today's gap as a -"known-temporary workaround" is wasted effort when the fix is this small and this -well-precedented. diff --git a/docs/README.md b/docs/README.md new file mode 100644 index 0000000..c59199e --- /dev/null +++ b/docs/README.md @@ -0,0 +1,21 @@ +# Terminal Duty documentation + +The [README](../README.md) is the short tour. These pages hold the detail. + +**Running it** +- [Deployment](./deployment.md): Docker, the Helm chart, the database, backups, and the operator. +- [Configuration](./configuration.md): environment variables and settings. +- [Single sign-on](./single-sign-on.md): OIDC, group mapping, the terminal device flow. + +**Using it** +- [The web UI](./web-ui.md): sessions, the Team and Admin tabs. +- [Alertmanager configuration](./alertmanager.md): routes, integration keys and webhooks. +- [Alerts and incidents](./incidents.md): correlation, lifecycle, on-call assignment, stale-alert expiry. +- [Push notifications](./notifications.md): ntfy pages and acknowledging from them. +- [Escalation](./escalation.md): ladders, repeats and the fallback topic. +- [Dead man's switches](./dead-mans-switch.md): noticing that alerts stopped arriving. + +**Integrating and contributing** +- [API reference](./api.md): every endpoint, authentication and error shape. +- [Service accounts](../SERVICE-ACCOUNTS.md): non-human credentials for automation. +- [Development and releasing](./development.md): tests, the CI gate, the release pipeline. diff --git a/docs/alertmanager.md b/docs/alertmanager.md new file mode 100644 index 0000000..a153e72 --- /dev/null +++ b/docs/alertmanager.md @@ -0,0 +1,62 @@ +# Alertmanager configuration + +_Pointing Alertmanager at the server._ Back to the [README](../README.md) and the [documentation index](./README.md). + +Alerts arrive on a team's **integration key**, which says both that the sender +may post and which team the alerts belong to. Mint one as an owner of the team: + +```bash +curl -X POST https://terdut.example.com/api/teams/1/integrations \ + -H "Authorization: Bearer $TERDUT_API_KEY" \ + -H 'Content-Type: application/json' \ + -d '{"name":"prod alertmanager"}' +``` + +The response carries the key and the full URL **once**; only a SHA-256 hash is +stored. Put it in your `alertmanager.yml`: + +```yaml +receivers: + - name: terdut + webhook_configs: + - url: http://terdut-server:8080/api/integrations//alertmanager + send_resolved: true + +route: + receiver: terdut +``` + +The whole URL is a credential, so treat it like one. Alertmanager 0.26 and +later can read it from a file with `url_file:` instead, which keeps it out of +your configuration repository: + +```yaml + - url_file: /etc/alertmanager/secrets/terdut-webhook-url/url + send_resolved: true +``` + +The webhook endpoint requires no authentication. + +If you use the [dead man's switch](./dead-mans-switch.md) — and the default configuration does — give +the heartbeat a route of its own, because the deadline is only as tight as the interval feeding it: + +```yaml +route: + receiver: terdut + repeat_interval: 4h + routes: + - matchers: [ 'alertname = "Watchdog"' ] + receiver: terdut + group_wait: 0s + group_interval: 1m + repeat_interval: 1m +``` + +That delivers a heartbeat every **2 minutes**, not every minute. Alertmanager only reconsiders a +group every `group_interval`, and at exactly one elapsed interval `repeat_interval` has not *quite* +passed, so the send slips to the next tick — equal values give 2×. Two minutes against the 15 minute +default is seven heartbeats per window, which is the point; use `group_interval: 30s` if you want +the numbers to mean what they say. + +kube-prometheus-stack users get the `Watchdog` alert (`expr: vector(1)`) for free; it just needs +routing to terdut rather than to `null`. diff --git a/docs/api.md b/docs/api.md new file mode 100644 index 0000000..6af26a0 --- /dev/null +++ b/docs/api.md @@ -0,0 +1,436 @@ +# API reference + +_The REST API._ Back to the [README](../README.md) and the [documentation index](./README.md). + +## Authentication + +All endpoints except `/api/bootstrap`, `/api/integrations/{key}/alertmanager`, +`/api/notify/ack/{token}`, `/api/login`, `/api/logout`, `/api/auth/config`, +`/api/version`, `/api/oidc/login`, `/api/oidc/callback`, `/api/oidc/device` and +`/api/oidc/device/token` require either an API key: + +``` +Authorization: Bearer +``` + +or the web UI's session cookie. A request that carries an `Authorization` header +is judged on that header alone. + +Two kinds of user exist. An **administrator** manages accounts: creating and +deleting users, setting anybody's password, minting keys for anybody, and +granting the flag itself. Everybody else works incidents — acknowledging, +assigning, snoozing, resolving, noting — and manages their own account and +nobody else's. An API key carries exactly the rights of the user it belongs to. + +A third principal, the **service account**, exists for automation (a +Kubernetes operator, most likely) that needs to manage teams, escalation +policies, dead man's switches and integrations without impersonating a human. +It is not a user — it never signs in, never appears in a team's member list, +and never holds the administrator flag — and its key is prefixed `tdsa_` so it +reads as one at a glance in a log line. See [Service accounts](#service-accounts). + +**Getting an account.** The first one comes from `/api/bootstrap`. After that +it depends on `signup_mode`, an administrator setting: + +- `invite_only` (the default) — a team owner mints a link with + `POST /api/teams/{teamID}/invites`, and the person who opens it picks a + username and password and lands in that team with the role the link carries. + Links are single-use unless told otherwise, expire after seven days, and can + be revoked before that. +- `open` — anybody who can reach the server can create an account, and must + name a team, which they then own. + +Invites are **links, not email**: this server has no SMTP, and adding it to send +one message would be a subsystem to run, secure and monitor. Send the link +however you already talk to the person. + +A domain-restricted third mode was considered and dropped: with no email there +is nothing to verify an address against, so it would only check the domain of a +string somebody typed. + +The first user, from `/api/bootstrap`, is an administrator. Users created +afterwards are not, until an administrator says so. An install always keeps at +least one: the last administrator can be neither deleted nor demoted, and +nobody can delete or demote themselves. + +Endpoints that require the flag answer `403` with +`{"error":"administrator access required"}`. + +**Teams** are the unit of tenancy, and are a separate axis from the administrator +flag. A team owns its incidents, alerts, schedule and integrations, and a user +sees exactly the teams they belong to. Within a team an **owner** configures it +(schedule, integrations, membership) and a **member** works its incidents. + +An administrator crosses that line in one direction only. They **configure any +team** without being in it — every owner-only endpoint accepts the flag, because +otherwise a team whose last owner left could never be repaired. They do **not +read any team**: the queue, the alerts and the incidents are filtered by real +membership, so an administrator sees a team's work only by joining it, which is +a membership change and shows up as one. Administration is about accounts and +the shape of a team, not about reading other people's incidents. + +Anything belonging to a team you are not in answers `404`, not `403`: whether an +incident exists is itself something only its team should learn. + +**Operator mode** (`TERDUT_OPERATOR_MODE`, see [Configuration](./configuration.md#configuration)) +declares this install gitops-managed. When it is on, a session or a user's own +API key gets `403 {"error": "...", "reason": "operator_managed"}` on every +write this page marks **owner**-gated under Teams below (creating, renaming +or deleting a team; its OIDC group binding; its escalation ladder; its dead +man's switches; its integrations) — a service account's writes are unaffected. +Team membership and invites are deliberately excluded: they are never +gitops-managed, in operator mode or out of it. `GET /api/auth/config` reports +`operator_mode` so a client can grey those sections out before a write is ever +attempted. + +| Method | Path | Description | +|---|---|---| +| `GET` | `/api/auth/config` | How to sign in: `{"password_login", "oidc": {"enabled","name"}, "device_login", "operator_mode"}`. No session needed | +| `GET` | `/api/version` | `{"version"}` — this build's version string. No session needed, the same as `/healthz` | +| `POST` | `/api/login` | `{"username","password"}` → sets the session cookie, returns `{user, has_password}`. `429` after too many failures; `403` when `TERDUT_PASSWORD_LOGIN=false` | +| `GET` | `/api/oidc/login` | Starts a single sign-on sign-in: redirects the browser to the provider. `?next=/path` is where to land afterwards; only a path on this server is honoured. Only exists when SSO is configured | +| `POST` | `/api/oidc/device` | Starts a device login: returns `{device_code, user_code, verification_url, interval, expires_in}`. Only exists when SSO is configured | +| `POST` | `/api/oidc/device/token` | `{"device_code"}` → `202 {"status":"pending"}`, then `200` with the session cookie once approved (once only). `410` with `{"error":"expired"}` or `{"error":"denied"}`; `429 {"error":"slow_down"}` if polled faster than `interval` | +| `POST` | `/api/oidc/device/approve` | **session** — `{"user_code"}`. Approves a pending device login as the caller. `403` for an API key; `404` for an unknown, expired or already decided code | +| `POST` | `/api/oidc/device/deny` | **session** — `{"user_code"}`. Refuses it | +| `GET` | `/api/oidc/callback` | Where the provider sends the browser back. Sets the session cookie and redirects to `/`, or to `/?sso_error=` — one of `denied`, `expired`, `failed`, `unavailable`, `not_allowed`, `no_email`, `email_conflict`, `disabled`, `not_bootstrapped` (no user exists on this install yet — sign in again once something has called `/api/bootstrap`) | +| `POST` | `/api/logout` | Ends the session and clears the cookie | +| `GET` | `/api/me` | The caller: `{user, has_password}` | + +## Users + +| Method | Path | Description | +|---|---|---| +**admin** marks an endpoint that requires the administrator flag; **self or +admin** marks one you may use on your own account and an administrator may use +on anybody's. + +| Method | Path | Who | Description | +|---|---|---|---| +| `GET` | `/api/signup` | — | Whether sign-up is open, and whether `?invite=` is usable. No session needed: the caller has no account yet | +| `POST` | `/api/signup` | — | Create an account `{"username","email","password","invite"?,"team_name"?}` and sign in. `403` without a usable invite when the mode is invite-only | +| `POST` | `/api/bootstrap` | — | Create first user + API key `{"username","email","password"?}` (only works on empty DB). The user is an administrator | +| `GET` | `/api/users` | any | List users. Open to everybody: the queue's assignment control and the schedule both have to name people | +| `GET` | `/api/users/{id}/teams` | self or admin | The teams that user is in, each with their role. `/api/teams` is always about the caller; this one answers it about somebody else, for the admin page's per-user view. `404` for a user who does not exist, so "no teams" and "no such person" are distinguishable | +| `POST` | `/api/users` | **admin** | Create user `{"username","email"}`. Not an administrator | +| `DELETE` | `/api/users/{id}` | **admin** | Delete user (cascades to keys). `409` for yourself or the last administrator | +| `PUT` | `/api/users/{id}/admin` | **admin** | Grant or revoke the administrator flag `{"is_admin"}`. `409` for yourself, the last administrator, or an administrator granted by single sign-on | +| `PUT` | `/api/users/{id}/disabled` | **admin** | Take an account out of use, or put it back `{"disabled"}`. `409` for yourself or the last administrator | +| `PUT` | `/api/users/{id}/notify` | self or admin | Set push notification target `{"ntfy_topic"}` — empty string clears it | +| `PUT` | `/api/users/{id}/password` | self or admin | Set web UI password `{"password","current_password"}`. `current_password` is required only when changing your own existing password. Ends the user's other sessions | +| `POST` | `/api/users/{id}/api-keys` | self or admin | Issue API key `{"name"}` — key shown once | +| `DELETE` | `/api/users/{id}/api-keys/{keyID}` | self or admin | Revoke API key | + +## Administration + +| Method | Path | Who | Description | +|---|---|---|---| +| `GET` | `/api/admin/teams` | **admin** | Every team on the server, with its member and open-incident counts. `/api/teams` answers "what am I in"; this answers "what is there" | +| `GET` | `/api/admin/teams/{teamID}` | **admin** | One team and who is in it: `{"team", "members"}`. `404` for a team that does not exist. `GET /api/teams/{teamID}/members` is **member**-only and still `404`s an administrator from outside the team — reading a team's shape and reading its work are different questions, so they are different endpoints | +| `GET` | `/api/admin/settings` | **admin** | The editable settings with their bounds, plus the environment-configured ones, read-only. Never credentials | +| `PUT` | `/api/admin/settings` | **admin** | Change one or more `{"key": seconds}`, or `{"signup_mode": "open"\|"invite_only"}`. `400` for an unknown key or a value outside its bounds | + +## Service accounts + +A service account is a scoped, non-human credential for automation — not a +`users` row, so it never signs in, is never a team member, and never carries +the administrator flag. Two scopes: + +- **instance** — the same reach system administration has over teams: create + one, and mint a **team**-scoped account against any of them. There is no + cap on how many instance-scoped accounts exist, but ordinarily there is one, + belonging to whatever is provisioning this install end to end. +- **team** — owner-equivalent for that one team, and nothing else: every + **owner**-gated endpoint under [Teams](#teams), membership and invites + included. Nothing narrower is enforced server-side; what actually keeps + membership out of automation's hands is that no operator built against this + scope should ever call those two endpoints — see + [operator mode](#authentication) and [`SERVICE-ACCOUNTS.md`](../SERVICE-ACCOUNTS.md)'s note on this. + +A key is shown once, at creation or rotation, and only its hash is stored — +the same handling as a user's API key. Losing it means minting a new one; +there is no way to recover a raw key from the server. + +| Method | Path | Who | Description | +|---|---|---|---| +| `GET` | `/api/service-accounts` | **admin** | Every service account. Pass `?name=` instead to look one up by its exact name — open to **any** authenticated caller (human or service account), since it returns no key material and is how an account finds its own id | +| `POST` | `/api/service-accounts` | owner\* | Create one and mint its first key `{"name","scope","team_id"?}` (`team_id` required for `scope:"team"`, absent for `scope:"instance"`). Returns `{"service_account", "key"}` — `key.key` shown once | +| `POST` | `/api/service-accounts/{id}/keys` | owner\* | Mint an additional key `{"name"}` — rotation without recreating the account. Shown once | +| `DELETE` | `/api/service-accounts/{id}/keys/{keyID}` | owner\* | Revoke one key | + +\* For an **instance**-scoped account: a system administrator only. For a +**team**-scoped account: a system administrator, that team's own human owner, +an instance-scoped service account (minting a narrower credential for a team +it just created), or — for the two key endpoints only — the account rotating +or revoking its own key, which is not a privilege escalation, the same +reasoning a user's own API keys rest on. + +## Alert ingestion + +Alerts arrive on a team's integration key. The key is both the credential and the +routing: it says that the sender may post, and which team the alerts belong to. +Create one with `POST /api/teams/{teamID}/integrations`, which returns the key +and the full URL once and stores only a SHA-256 hash. + +| Method | Path | Description | +|---|---|---| +| `POST` | `/api/integrations/{key}/alertmanager` | Alertmanager v4 webhook receiver for the key's team. `401` for an unknown key | + +This is the only way in. The pre-teams `POST /api/alertmanager/webhook` took no +credential at all — anything able to reach the port could open an incident — +and was removed in v0.13.0 once senders had moved onto keys. + +## Teams + +**owner** below means an owner of that team, a system administrator (who +passes every one of these without being a member), or that team's own +team-scoped [service account](#service-accounts) — including membership and +invites, technically, though no automation this scope was designed for +(a Kubernetes operator's CRDs, see [`SERVICE-ACCOUNTS.md`](../SERVICE-ACCOUNTS.md)) ever models team +membership or would call those two. See [Authentication](#authentication). +**member** means membership and nothing else: an administrator who is not in +the team gets the same `404` as anybody else. + +| Method | Path | Who | Description | +|---|---|---|---| +| `GET` | `/api/teams` | any | The caller's own teams, each with their role | +| `POST` | `/api/teams` | any | Create a team `{"name"}`; a human creator becomes its first owner. An instance-scoped [service account](#service-accounts) may also create one, and it gets no owner at all — expected for a team an operator is about to hand a team-scoped credential to, not an orphaned team a human made | +| `PUT` | `/api/teams/{teamID}` | **owner** | Rename it `{"name"}`. `409` if the name is taken | +| `DELETE` | `/api/teams/{teamID}` | **owner** | Delete a team and everything under it. `409` while it has open incidents | +| `GET` | `/api/teams/{teamID}/members` | member | Who is in the team, with `status` (`oncall` if the rota has them today, `unpageable` when a page to them would go nowhere — even if they are on call — else `reachable`), `on_call`, `next_shift` (first rota day after today), `pageable` and `problem` (`has no ntfy topic` / `account is disabled`; never the topic itself) and `last_active_at` (their newest session or API-key use). Every member sees the same list | +| `POST` | `/api/teams/{teamID}/members` | **owner** | Add a member, or change their role `{"user_id","role"}`. `409` when it would demote the last owner, or the membership is managed by single sign-on | +| `DELETE` | `/api/teams/{teamID}/members/{userID}` | **owner** | Remove a member. `409` for the last owner, or a membership managed by single sign-on | +| `GET` | `/api/teams/{teamID}/oidc-groups` | member | Which groups control this team's membership: `{"member_group","owner_group"}`. An empty string means no group grants that role here | +| `PUT` | `/api/teams/{teamID}/oidc-groups` | **owner** | Set them. An empty string clears a binding | +| `GET` | `/api/teams/{teamID}/integrations` | member | List integrations. Never returns keys. Each carries `status` (`active` if its key posted within 24h, `quiet` if it has but not lately, `never`), `last_used_at` (last webhook, usable or not), `last_alert_at` (when an alert last arrived on it) and `alerts_24h` (distinct alerts it refreshed in the last day). Alerts delivered before the source was recorded (migration 010) have none, so the last two fill in as Alertmanager re-sends them | +| `PATCH` | `/api/teams/{teamID}/integrations/{integrationID}` | **owner** | Rename `{"name"}`. The key does not change | +| `POST` | `/api/teams/{teamID}/integrations` | **owner** | Mint an integration `{"name","kind"}` — key and URL shown once | +| `DELETE` | `/api/teams/{teamID}/integrations/{integrationID}` | **owner** | Revoke an integration. Alerts it delivered stay, unattributed | +| `GET` | `/api/teams/{teamID}/invites` | **owner** | The team's invite links, with their uses and expiry. Never the tokens | +| `POST` | `/api/teams/{teamID}/invites` | **owner** | Mint one `{"role","max_uses"}` — the full URL is returned once | +| `DELETE` | `/api/teams/{teamID}/invites/{inviteID}` | **owner** | Revoke a link before it expires | +| `GET` | `/api/teams/{teamID}/escalation` | member | The team's [escalation ladder](./escalation.md#escalation) `{repeat_count, fallback_topic, levels[], last_escalated_at?, last_escalated_incident_id?}`. Empty levels means the team has none. Each level also carries `status` (`ready`, `escalating` when an unanswered incident has climbed to it, `unreachable` when nobody on it could be woken), `waiting` (ids of the open incidents on it) and, per target, `username` (who it means today — the person on call, for a rota target), `reachable` and `problem`. The extra fields are output only; `PUT` takes the plain shape | +| `PUT` | `/api/teams/{teamID}/escalation` | **owner** | Replace it wholesale. `400` for a level with no targets or no timeout — a rung that pages nobody is a silence with a number on it | +| `GET` | `/api/teams/{teamID}/deadman/switches` | member | The team's [dead man's switches](./dead-mans-switch.md), each `{id, name, matcher, timeout_seconds, severity, status, last_heartbeat_at, last_triggered_at, open_incident_id, sources[]}`. `status` is `healthy`, `dead` or `dormant`; `sources` has one entry per heartbeat fingerprint. Empty when the team watches nothing | +| `POST` | `/api/teams/{teamID}/deadman/switches` | **owner** | Add one: `{name?, matcher, timeout_seconds, severity?}`. `400` when the matcher names no `alertname` or holds several, or the timeout is not positive — a switch that silently watches nothing is the failure this feature exists to prevent | +| `PUT` | `/api/teams/{teamID}/deadman/switches/{switchID}` | **owner** | Replace one in place, same body and validation as create. Its id is unchanged — for an automated caller reconciling a spec change, unlike delete-and-recreate | +| `DELETE` | `/api/teams/{teamID}/deadman/switches/{switchID}` | **owner** | Stop watching. An incident it opened stays open. `404` for a switch of another team | + +## Notifications + +| Method | Path | Description | +|---|---|---| +| `POST` | `/api/notify/ack/{token}` | Acknowledge an incident from a push notification's Acknowledge button. No auth: the token in the path is the credential — one incident, one action, 24 hours, idempotent. Must stay publicly reachable | + +## Incidents + +| Method | Path | Description | +|---|---|---| +| `GET` | `/api/incidents` | List incidents. Filters: `?status=triggered\|acknowledged\|resolved`, `?severity=`, `?assigned_to=`, `?archived=true`, `?snoozed=true`, `?from=YYYY-MM-DD`, `?to=YYYY-MM-DD`, `?sort=severity`, `?cluster=`, `?limit=` (default 50, max 500) | +| `GET` | `/api/incidents/clusters` | The distinct `cluster` values on the caller's incidents from the last 90 days, sorted (`?team_id=` narrows it). An empty array when nothing carries the label | +| `GET` | `/api/incidents/{id}` | Get single incident, with its alerts inline | +| `GET` | `/api/incidents/{id}/alerts` | Alerts under this incident | +| `GET` | `/api/incidents/{id}/timeline` | Full event history, chronological | +| `POST` | `/api/incidents/{id}/acknowledge` | Acknowledge (stamps authed user + time) | +| `DELETE` | `/api/incidents/{id}/acknowledge` | Clear acknowledgement, back to `triggered` | +| `POST` | `/api/incidents/{id}/resolve` | Close by hand — **terminal**, see above | +| `POST` | `/api/incidents/{id}/assign` | Reassign `{"user_id"}` | +| `POST` | `/api/incidents/{id}/snooze` | Hide until `{"until": RFC3339}` or `{"duration": "2h"}` | +| `DELETE` | `/api/incidents/{id}/snooze` | Un-snooze | +| `POST` | `/api/incidents/{id}/archive` | Archive (hides from the default list) | +| `DELETE` | `/api/incidents/{id}/archive` | Un-archive | +| `POST` | `/api/incidents/{id}/notes` | Add a note `{"content"}` | +| `DELETE` | `/api/incidents/{id}/notes/{eventID}` | Delete own note | + +With no `?status=` filter, `GET /api/incidents` returns **open** incidents only — +the queue an on-call person wants. Currently snoozed and archived incidents are +excluded unless asked for. Actions that only make sense on an open incident +return `409` once it is resolved. + +Notes are ordinary timeline events of type `note`; only they are deletable, and +only by their author. The rest of the timeline is a record of what happened. + +### The incident object + +| Field | Type | Notes | +|---|---|---| +| `id` | integer | Server-assigned | +| `group_key` | string | Alertmanager's `groupKey` — opaque, treat as an identifier | +| `title` | string | Rendered from `groupLabels` | +| `group_labels` | object | String→string, as sent by Alertmanager | +| `status` | string | `"triggered"`, `"acknowledged"` or `"resolved"` | +| `severity` | string | *optional* — high-water mark across the incident's alerts; never lowered | +| `triggered_at` | timestamp | When the incident opened | +| `acknowledged_by_id` / `acknowledged_by` / `acknowledged_at` | | *optional* — user id, username, time | +| `assigned_to_id` / `assigned_to` | | *optional* — user id, username | +| `snoozed_until` | timestamp | *optional* — a value in the past reads as not snoozed | +| `resolved_at` | timestamp | *optional* | +| `resolution_source` | string | *optional* — `"alerts"`, `"manual"` or `"recovered"` | +| `archived_at` | timestamp | *optional* | +| `alerts` | array | Only on `GET /api/incidents/{id}` | + +Treat `resolution_source` as an open set, as with the alert field of the same +name: degrade unknown values to "resolved, reason unknown". + +### The timeline event object + +| Field | Type | Notes | +|---|---|---| +| `id` | integer | | +| `incident_id` | integer | | +| `type` | string | See below — treat as an open set | +| `user_id` / `username` | | *optional* — absent when the server acted rather than a person | +| `alert_id` | integer | *optional* — the alert an `alert_added` / `alert_resolved` event refers to | +| `detail` | string | *optional* — the note body, the snooze deadline, etc. | +| `created_at` | timestamp | | + +Types written today: `triggered`, `alert_added`, `alert_resolved`, +`acknowledged`, `unacknowledged`, `assigned`, `archived`, `unarchived`, `snoozed`, +`unsnoozed`, `resolved`, `note`, `notified`, `notify_failed`, `deadman_silent`. On an +`assigned` event `user_id` is the **assignee**, not the actor; the actor is in +`actor_user_id`/`actor_username` or `actor_service_account_id`/`actor_service_account_name` +(absent on assignments made before they were recorded). New types may be added; render +unknown ones generically rather than dropping them. + +On `notified` and `notify_failed`, `detail` carries the notification kind +(`triggered` | `reminder` | `resolved`), and on a failure the reason after it. +`user_id` is who was paged — absent means the page went to the shared fallback +topic and so belongs to nobody. The topic itself is never written to the +timeline: it is a shared secret with the ntfy server, and every API key can read +this. + +## Alerts + +Alerts are read-only. Everything a person does happens on the incident. + +| Method | Path | Description | +|---|---|---| +| `GET` | `/api/alerts` | List alerts. Filters: `?status=firing\|resolved`, `?name=`, `?incident_id=`, `?archived=true`, `?from=YYYY-MM-DD`, `?to=YYYY-MM-DD`, `?limit=` (default 50, max 500) | +| `GET` | `/api/alerts/{id}` | Get single alert | + +Archived alerts are hidden from `GET /api/alerts` unless `?archived=true` is +passed; alert archiving is automatic housekeeping by the sweeper, not a user +action. Resolved alerts carry `resolution_source`: `"alertmanager"` for a real +resolved webhook, `"expiry"` when the sweeper inferred it (see +[Stale alert expiry](./incidents.md#stale-alert-expiry)), `"deadman"` for a heartbeat declared +dead (see [Dead man's switch](./dead-mans-switch.md)). + +### The alert object + +Returned by `GET /api/alerts` (as an array) and `GET /api/alerts/{id}`. +Timestamps are RFC 3339 in UTC. Fields marked *optional* are omitted entirely +when unset, so clients must treat them as nullable. + +| Field | Type | Notes | +|---|---|---| +| `id` | integer | Server-assigned; stable for the life of the row | +| `fingerprint` | string | Alertmanager's fingerprint — the upsert key | +| `name` | string | From the `alertname` label | +| `status` | string | `"firing"` or `"resolved"` | +| `labels` | object | String→string, as sent by Alertmanager | +| `annotations` | object | String→string, as sent by Alertmanager | +| `starts_at` | timestamp | When the alert instance began, **per Prometheus** | +| `ends_at` | timestamp | *optional* — absent while no end is known | +| `generator_url` | string | Link back to the originating Prometheus | +| `received_at` | timestamp | When the server last accepted a webhook for this alert — see below | +| `incident_id` | integer | *optional* — the most recent incident this alert belongs to | +| `resolution_source` | string | *optional* — `"alertmanager"`, `"expiry"` or `"deadman"` | +| `archived_at` | timestamp | *optional* — set while archived | + +#### `received_at` is a liveness heartbeat + +`starts_at` comes from Prometheus and **never changes** for the lifetime of an +alert instance. It says when the problem began, not whether it is still +happening — an alert that started twelve days ago looks identical whether +Alertmanager refreshed it a minute ago or went silent a week ago. + +`received_at` is the field that answers "is this still live". It is set to the +server's clock on **every accepted webhook** for that fingerprint, including the +unchanged firing notifications Alertmanager re-sends every `repeat_interval`. +Clients may rely on this: + +- **A firing alert whose `received_at` is advancing is still being refreshed.** + Stale-dating it against `repeat_interval` is a valid liveness check, and it is + what the built-in sweeper does (see + [Stale alert expiry](./incidents.md#stale-alert-expiry)). +- **`received_at` tracks accepted payloads, not delivery attempts.** A retry + that describes an older instance than the stored one is discarded, and a + discarded payload does not move `received_at`. +- **It stops advancing once the alert resolves,** because Alertmanager stops + re-sending. On an alert resolved by the sweeper + (`"resolution_source": "expiry"`) it therefore marks the last time + Alertmanager was actually heard from, which is earlier than `ends_at`. + +`GET /api/alerts` is ordered by `received_at` descending — most recently +refreshed first — and the `?from=` / `?to=` filters on both the alert and stats +endpoints select on `received_at`, not `starts_at`. + +#### `resolution_source` says how much to trust `ends_at` + +An alert can leave the firing state two ways, and `resolution_source` records +which happened. Clients may rely on this: + +- **Absent while firing.** It is set only on resolve, and a re-fire under the + same fingerprint clears it again, so its presence always agrees with + `"status": "resolved"`. +- **`"alertmanager"` — a real resolved webhook arrived.** `ends_at` is the end + time Alertmanager reported. It is an observed value and can be displayed as + fact. +- **`"expiry"` — the sweeper inferred the resolve** because Alertmanager stopped + refreshing the alert (see [Stale alert expiry](./incidents.md#stale-alert-expiry)). Nothing + ever reported an end, so **`ends_at` is approximate**: it is either the stale + `endsAt` watermark from the last notification, or — when that notification + carried none — the time the sweep ran, which lags the last real contact by up + to `TERDUT_STALE_AFTER` plus a sweep interval. Treat it as "no later than", + not as when the problem stopped. + + On these alerts `received_at` is the more truthful signal: it marks the last + time Alertmanager was actually heard from. Surfacing the distinction is + worthwhile, since `"expiry"` can also mean the alert is still firing and the + notification path broke. + +- **`"deadman"` — a heartbeat was declared dead** (see + [Dead man's switch](./dead-mans-switch.md)). Like `"expiry"`, an inference from + silence rather than an observed end, so `ends_at` is approximate — but a much + tighter one, bounded by the switch's timeout. It is also the one resolution + a re-fire under the same `starts_at` can undo, since the switch coming back is + exactly the evidence that the inference was wrong. + +Treat the value as an open set and tolerate ones you do not recognise — new +sources may be added, and unknown values should degrade to "resolved, reason +unknown" rather than being rejected. + +## On-call schedule + +| Method | Path | Description | +|---|---|---| +Each team keeps its own rota, so two teams can have two different people on call +on the same day. The person taking a shift has to be in the team — paging +somebody who cannot open the incident is worse than paging nobody. + +| Method | Path | Who | Description | +|---|---|---|---| +| `POST` | `/api/teams/{teamID}/schedule` | **owner** | Assign user to dates `{"user_id", "dates":["YYYY-MM-DD",...], "replace"}` — all-or-nothing | +| `GET` | `/api/teams/{teamID}/schedule` | member | List entries. Filters: `?from=YYYY-MM-DD`, `?to=YYYY-MM-DD` | +| `DELETE` | `/api/teams/{teamID}/schedule/{id}` | **owner** | Remove schedule entry | +| `GET` | `/api/schedule/current` | any | Who is on call today (UTC) in **every** team the caller is in — one entry per team, `[]` when nobody anywhere | + +## Statistics + +Every figure counts the caller's own teams only: a report that counted other +teams' incidents would leak their volume, and their alert names through the +top-alerts list, and would not be a number about the reader's work anyway. + +All stat endpoints accept optional `?from=YYYY-MM-DD` and `?to=YYYY-MM-DD`, and exclude archived rows to match the default list views. Alert stats filter on `received_at`; incident stats filter on `triggered_at`. + +| Method | Path | Description | +|---|---|---| +| `GET` | `/api/stats/incidents` | `{total, triggered, acknowledged, resolved, mtta_seconds, mttr_seconds}` | +| `GET` | `/api/stats/alerts` | `{total, firing, resolved}` counts | +| `GET` | `/api/stats/alerts/top` | Most frequent alert names. `?limit=` (default 10, max 100) | +| `GET` | `/api/stats/alerts/by-hour` | Count per hour-of-day (UTC), all 24 slots returned | +| `GET` | `/api/stats/alerts/by-day` | Count per day-of-week, all 7 slots with names returned | + +`mtta_seconds` (time to acknowledge) and `mttr_seconds` (time to resolve) are +averages over incidents that have actually been acknowledged or resolved, and are +**null** until there are any — null means "no data", not zero. diff --git a/docs/configuration.md b/docs/configuration.md new file mode 100644 index 0000000..b89a49a --- /dev/null +++ b/docs/configuration.md @@ -0,0 +1,49 @@ +# Configuration + +_Environment variables and settings._ Back to the [README](../README.md) and the [documentation index](./README.md). + +Two kinds of setting, split by who changes them and how often. + +**Where the server is plugged in** stays in the environment: the listen address, +the database DSN, the ntfy URL and token, the public URL. They are needed before +the database is open, and two of them are credentials. + +**How the server behaves** lives in the database and is edited by an +administrator in the web UI or through `PUT /api/admin/settings`, taking effect +on the next sweep rather than at the next restart. The variables below marked +**seed** are the value each of those starts from: written once, on first start, +and never overwritten afterwards — a redeploy cannot put a chart's default back +over an administrator's edit. + +| Variable | Default | Description | +|---|---|---| +| `TERDUT_ADDR` | `:8080` | TCP address to listen on | +| `TERDUT_DB_DSN` | — | **Required.** Postgres connection string, e.g. `postgres://terdut:secret@localhost:5432/terdut?sslmode=require` | +| `TERDUT_ARCHIVE_AFTER` | `168h` (7d) | **seed.** How long a resolved alert or incident stays in the default list before being auto-archived | +| `TERDUT_STALE_AFTER` | `6h` | **seed.** How long a firing alert may go without a refreshing webhook before it is treated as resolved — **must exceed your Alertmanager `repeat_interval`** | +| `TERDUT_NTFY_URL` | — | ntfy server to publish push notifications to. Empty disables notifications entirely | +| `TERDUT_NTFY_TOKEN` | — | Bearer token for an access-controlled ntfy | +| `TERDUT_NTFY_FALLBACK_TOPIC` | — | Topic used when nobody is on call | +| `TERDUT_PUBLIC_URL` | — | Base URL a phone uses to reach this server: the notification's link into the web UI, its Acknowledge button, and whether the session cookie is `Secure` | +| `TERDUT_NOTIFY_REPEAT` | `15m` | **seed.** How long an incident may sit unacknowledged before it is paged again. `0` notifies once and never repeats | +| `TERDUT_PASSWORD_LOGIN` | `true` | `false` refuses password login and password sign-up (`403`), leaving single sign-on the only way in. Refused at startup unless SSO is configured | +| `TERDUT_TRUSTED_PROXIES` | `1` | How many reverse proxies in front of the server append to `X-Forwarded-For`; the per-address rate limits use the entry that many hops from the right. `0` ignores the header | +| `TERDUT_OPERATOR_KEY` | — | At least 32 characters. When set, the instance-scoped service account `terdut-operator` is created if missing and its `seed` key replaced with this value at every start — how terdut-operator authenticates without a bootstrap handshake. An instance-scoped account acts as owner of every team (team configuration) but is not a member of any, so it reads no incidents | +| `TERDUT_OPERATOR_MODE` | `false` | Declares this install gitops-managed: a session's or a user's own API key's writes to teams, escalation policies, dead man's switches and integrations are refused (`403 reason:"operator_managed"`); a [service account](./api.md#service-accounts)'s are not. Team membership and the schedule stay editable regardless | +| `TERDUT_OIDC_ISSUER` | — | Turns single sign-on on. The provider's issuer URL; discovery is read from `/.well-known/openid-configuration`. See [Single sign-on](./single-sign-on.md#single-sign-on-oidc) | +| `TERDUT_OIDC_CLIENT_ID` / `TERDUT_OIDC_CLIENT_SECRET` | — | **Required with an issuer.** The confidential client registered at the provider. Keep the secret in a Secret, not in values | +| `TERDUT_OIDC_NAME` | `SSO` | What the sign-in button calls the provider | +| `TERDUT_OIDC_SCOPES` | `openid profile email` | Scopes requested, comma or space separated. Authentik puts `groups` behind `profile` | +| `TERDUT_OIDC_USERNAME_CLAIM` / `_EMAIL_CLAIM` / `_GROUPS_CLAIM` | `preferred_username` / `email` / `groups` | ID token claims read for the username, email and groups | +| `TERDUT_OIDC_TRUST_EMAIL` | `false` | Link a first sign-in to an existing local user by email even if the provider does not mark the address verified | +| `TERDUT_OIDC_ALLOWED_GROUPS` | — | Comma-separated. Only people in one of these may sign in. Empty admits everybody the provider authenticates | +| `TERDUT_OIDC_ADMIN_GROUP` | — | Members are system administrators | +| `TERDUT_OIDC_SESSION_MAX_AGE` | `12h` | Hard ceiling on a session made by an SSO sign-in | + +Durations use Go syntax (`30m`, `12h`, `168h`). An unparseable value falls back to the default. + +Note that `TERDUT_STALE_AFTER` and a dead man's switch timeout point in opposite directions. Staleness +is a generous grace period around a `repeat_interval` you do not control; a dead man's switch is a +deadline you set deliberately, and the heartbeat's route is configured to beat faster than it. + +In the Helm chart the two sweeper durations are set via `sweeper.staleAfter` and `sweeper.archiveAfter`, notifications via the `notify.*` values, single sign-on via `oidc.*` and `passwordLogin`, and operator mode via `operatorMode`. diff --git a/docs/dead-mans-switch.md b/docs/dead-mans-switch.md new file mode 100644 index 0000000..bd2a43b --- /dev/null +++ b/docs/dead-mans-switch.md @@ -0,0 +1,81 @@ +# Dead man's switches + +_Detecting that alerts have stopped arriving._ Back to the [README](../README.md) and the [documentation index](./README.md). + +Everything above assumes alerts arrive. If Prometheus stops evaluating, or +Alertmanager cannot reach this server, nothing arrives — and silence looks +exactly like everything being fine. A dead man's switch inverts the handling for +one designated alert so that silence is the signal: + +- **receiving** it opens no incident, and +- the **absence** of it does. + +kube-prometheus-stack already ships the alert for this. `Watchdog` is +`expr: vector(1)`, so it fires permanently and is re-sent forever; it is worth +nothing unless something downstream notices it stop. It is the usual first switch. + +**Switches belong to a team**, which decides which of its own alerts are +heartbeats and how long a silence has to last. Each **switch** is a row of its +own — a name, one matcher, a timeout and a severity — so switches in one team +can have different deadlines. An owner adds and removes them on **Team → +Switches**, which lists each with a status (**healthy**, **dead**, or +**dormant** until its first heartbeat), when it was last heard from, and when it +last opened an incident; a matcher that several clusters satisfy is broken down +per cluster. The API is `POST`/`DELETE /api/teams/{teamID}/deadman/switches`. A +missed heartbeat opens an incident in the team whose integration received it. +Removing a switch stops the watching; an incident it already opened stays open +until somebody resolves it. + +A new team watches nothing until its owner (or terdut-operator, from a +`TerdutTeam`) adds a switch: inheriting an install-wide heartbeat would page a +new team about a source it has never heard of. + +A matcher is a set of exact label conditions, one of which must be the +`alertname`, , one matcher per switch, `,` between the label conditions: + +``` +alertname=Watchdog,cluster=prod +``` + +**The unit of monitoring is the fingerprint, not the alert name.** Two clusters +sending the same `Watchdog` are two independent switches, so a healthy one can +never mask a dead one. + +## The lifecycle + +A switch is **dormant** until its first heartbeat arrives. A configured matcher +that has never been heard from opens nothing, so a fresh deploy or a restored +database does not page. It also means a matcher that never matches anything is +silently inert. + +Once armed, the sweeper declares it **dead** when either the heartbeat has not +been refreshed within the switch's `timeout_seconds`, or Alertmanager explicitly +resolved it — the sender saying the heartbeat stopped needs no further waiting. +That opens an incident at the switch's `severity`, assigned and paged like any +other, and marks the heartbeat alert `"resolution_source": "deadman"` so the +alert list stops claiming a dead switch is firing. + +It **recovers** when the heartbeat starts arriving again: the incident resolves +with `"resolution_source": "recovered"` and the all-clear goes to whoever was +paged. + +Resolving the incident by hand sticks, the same way it does for an alert-backed +one. While the switch stays silent nothing new opens — so a decommissioned +source is a one-time page rather than a nag. The switch **re-arms** on the next +heartbeat: come back and die again, and that is a new incident. + +## Two things to know + +A switch's timeout must be **shorter** than the `repeat_interval` of the +route carrying the heartbeat, which is the exact opposite of +`TERDUT_STALE_AFTER`. Inheriting a default `repeat_interval` of 4h gives you a +switch that takes four hours to notice anything, so give the heartbeat +[its own route](./alertmanager.md#alertmanager-configuration). Matched alerts are exempt from +stale-alert expiry — a heartbeat answers to its own timeout and nothing else. + +A dead man's switch incident has **no member alerts**: +`GET /api/incidents/{id}/alerts` returns an empty list. There is no alert +describing the problem, because the problem is that no alert arrived. What +happened is on the timeline instead, as a `deadman_silent` event carrying the age +of the last heartbeat, and the heartbeat's labels are on the incident's +`group_labels`. diff --git a/docs/deployment.md b/docs/deployment.md new file mode 100644 index 0000000..20e1ae9 --- /dev/null +++ b/docs/deployment.md @@ -0,0 +1,74 @@ +# Deployment + +_Running the server in a container and on Kubernetes with the Helm chart._ Back to the [README](../README.md) and the [documentation index](./README.md). + +## Docker + +```bash +docker build -t terdut-server . +docker run -p 8080:8080 \ + -e TERDUT_DB_DSN='postgres://terdut:secret@host.docker.internal:5432/terdut?sslmode=disable' \ + terdut-server +``` + +The server creates its own schema on startup and needs a reachable Postgres; it stores nothing on +disk, so there is no volume to mount. + +## Kubernetes + +A Helm chart is published from this repository as an OCI artifact, versioned in lockstep +with the app — chart `x.y.z` is always app `vx.y.z`: + +```bash +helm upgrade --install terdut-server oci://git.ryuvia.com/niklas/terdut-server \ + --version 0.9.2 \ + --namespace terdut-server --create-namespace \ + --set networking.hostname=terdut.example.com +``` + +The chart expects a [Gateway API](https://gateway-api.sigs.k8s.io/) Gateway named `envoy-main` in +the `envoy-gateway-system` namespace to already exist — it renders an `HTTPRoute` against it rather +than an `Ingress`. TLS is terminated at the gateway, so the server itself never sees a certificate. + +| Value | Default | Description | +|---|---|---| +| `networking.hostname` | `terdut.example.com` | Hostname the `HTTPRoute` serves | +| `networking.listener` | `""` | Gateway listener (`sectionName`) to bind to. Empty attaches to every matching listener, **including plaintext HTTP** — set it to the HTTPS listener's name to serve TLS only | +| `networking.servicePort` | `8080` | Port the route forwards to; keep in sync with `service.port` | +| `bootstrap.enabled` | `true` | Runs a post-install hook that creates the first user and stores its API key in the `-admin-key` Secret. Already-bootstrapped servers are left alone | +| `database.dsn` | `""` | **Required.** Postgres DSN, with no password in it. The chart provisions no database | +| `database.passwordSecret.name` | `""` | Secret supplying `PGPASSWORD`. With the Zalando postgres operator, the Secret it generates for the role | +| `database.passwordSecret.key` | `password` | Key within that Secret | + +The API key travels in an `Authorization: Bearer` header, so set `networking.listener` whenever the +hostname is reachable outside a trusted network. + +### The database + +The chart provisions no database: it takes a DSN and expects a Postgres that already exists. In this +cluster the wrapper chart declares an `acid.zalan.do/v1 postgresql` CR; anywhere else, any reachable +Postgres 14+ will do. + +The DSN carries no password. pgx falls back to libpq's environment variables for whatever the DSN +leaves out, so the password arrives as `PGPASSWORD` from a Secret and never appears in values, in +the rendered manifest or in `kubectl describe pod`. With the postgres operator that Secret is the +one it generates for the role, so a rebuild mints a new password with nothing to keep in sync — +the same wiring miniflux uses. + +The server migrates its own schema on startup, so a new database only has to exist and be writable. + +### Backups + +Postgres is backed up where it runs, not from here. The database pod carries a +[k8up](https://k8up.io/) `k8up.io/backupcommand` annotation that streams a `pg_dump`, the same way +gitea and immich do in this cluster. + +## On Kubernetes with the operator + +[terdut-operator](https://git.ryuvia.com/niklas/terdut-operator) runs a server for you from a +`TerdutServer` object and manages its teams, escalation ladders, dead man's switches and alert +sources as Kubernetes objects. It hands the server a generated key through `TERDUT_OPERATOR_KEY` +(see [Configuration](./configuration.md) and [`SERVICE-ACCOUNTS.md`](../SERVICE-ACCOUNTS.md)), and +the server then treats configuration as operator-managed (`TERDUT_OPERATOR_MODE`), refusing edits +made by hand in the web UI. Use the Helm chart above for a plain install, the operator when you +want that configuration in gitops. diff --git a/docs/development.md b/docs/development.md new file mode 100644 index 0000000..ae4b154 --- /dev/null +++ b/docs/development.md @@ -0,0 +1,93 @@ +# Development and releasing + +_Building, testing and releasing the server._ Back to the [README](../README.md) and the [documentation index](./README.md). + +## Upgrading + +The schema is a single baseline (`internal/db/migrations/001_schema.sql`) and no +release has shipped yet, so there is no upgrade path from earlier development +databases: start from an empty one. Changes after the first release arrive as +new numbered migrations. + +## Development + +```bash +make test-db # start a local Postgres for the tests (podman or docker) +make test # run all tests +go build ./... # compile all packages +go run ./cmd/terdut # run locally (needs TERDUT_DB_DSN) +``` + +The tests need a real Postgres, because the server does — there is no in-memory Postgres. +`TERDUT_TEST_DSN` says where it is, `make test-db` starts +one on port 5433 and prints the DSN, and `make test-db-stop` removes it. Each test gets its +own schema on that server, so tests cannot see each other's rows. An unset `TERDUT_TEST_DSN` +fails the suite rather than skipping it: a run that quietly tests nothing is worse than one +that does not run. + +`make fmt lint test helm-lint` is the gate. It mirrors `.gitea/workflows/ci.yaml` step for +step, so a green run here means a green pipeline — with one deliberate exception: `make test` +adds `-race`, which CI does not. The sweeper, the notifier goroutine and the dead man's switch +sweep all run concurrently against the same database, and a race between them would surface as +a flaky incident in production rather than as a red build. + +The web UI lives in `internal/web/static/` as plain HTML, CSS and ES modules, +embedded into the binary with `go:embed`. It has no build step and no npm, so +editing a file and restarting the server is the whole loop. + +## Releasing + +``` +push or PR → ci.yaml gofmt, go vet, go test -race + govulncheck, gitleaks + helm lint + render +push tag vX.Y.Z → release.yaml the same gate, then publish: + git.ryuvia.com/niklas/terdut-server:vX.Y.Z + oci://git.ryuvia.com/niklas/terdut-server X.Y.Z + then trivy-scan the pushed image +PR to Ryuvia/charts → bump the wrapper chart to X.Y.Z; on merge + Flux reconciles and the release rolls out +``` + +Both artifacts go to the **personal** Gitea namespace rather than `ryuvia`, because Gitea +scopes package visibility to the owner with no per-package override — so `ryuvia/*` is private +because the org is. Publishing to `niklas` keeps them anonymously pullable, which is why no +pull secret is needed in the cluster. Same reasoning, and the same choice, as riksdata and +rd-web. + +Saying **"Release"** runs all three rows: the `release` skill commits, pushes, tags, waits for +the pipeline, and opens the `Ryuvia/charts` PR, stopping before the merge. See +`~/.claude/skills/release/`, or `.release.conf` here for this repo's part of it. + +The chart is published **only** from the tag, by the `chart` job. There used to be a second +publisher on every `charts/**` push to main, and the two raced for the same chart version with +different answers — chart 0.9.0 went out reading `appVersion: "latest"` that way. One +publisher, triggered by the tag (`766f439`). The cost is that a chart-only change has no +version of its own and rides the next app tag. + +Both workflows are thin drivers over the Makefile: `ci.yaml` runs `make fmt lint test` and +`make helm-lint`, `release.yaml` adds `make binaries`, `make push`, `make helm-package` and +`make helm-push`. That is deliberate — it is what makes a green local gate and a green +pipeline the same code rather than two descriptions of it, and it is how riksdata and rd-web +have always worked. + +`make push` builds and pushes in one step, unlike those two, because the image is +`linux/amd64,linux/arm64` and buildx cannot load a multi-platform result into the local image +store. `make build` stays single-platform and local-only. Both refuse `VERSION=dev`: +publishing is one command, so it is also one command to run by accident. Publishing happens +by pushing a tag. + +Two things the release process needs to know about this repo: + +- **The image scan runs after publishing**, like riksdata's and rd-web's: trivy cannot read + a locally built image on this runner, so it pulls the pushed one. A red `scan-image` means + do not bump the wrapper chart to that version — it cannot unpublish anything. The image is + `FROM scratch`, so trivy sees exactly one target, the Go binary and its module graph. +- **The wrapper chart's `values.yaml` has two `tag:` lines** — the app image and the python + backup sidecar — so `chart-bump` is given `--image` to say which one moves. Once the wrapper + chart drops the sidecar and declares a `postgresql` CR instead, there is one `tag:` line + again, and `--image` becomes belt and braces. + +The wrapper chart must have **its own `version:` bumped in the same commit**. Flux reconciles +with `reconcileStrategy: ChartVersion`, so a chart whose version did not change produces no +new artifact and the change is never deployed — with no error anywhere. diff --git a/docs/escalation.md b/docs/escalation.md new file mode 100644 index 0000000..583aa2b --- /dev/null +++ b/docs/escalation.md @@ -0,0 +1,45 @@ +# Escalation + +_Escalation ladders: who is paged next when nobody acknowledges._ Back to the [README](../README.md) and the [documentation index](./README.md). + +Without a ladder, an unacknowledged incident re-pages the same topic every +`notify_repeat` forever. That is a louder version of the same silence: if the +person on call is asleep, out of signal, or has left, nothing else happens. + +A team can configure an ordered ladder instead. Each level has a timeout and a +set of targets, and a target is either a named person or **whoever the team's +rota says is on call today** — the target that keeps working when the rota +changes and nobody remembers to edit the policy. + +``` +level 1 5m oncall the rota gets first refusal +level 2 5m user:bob then a named second + then repeat_count more rounds + then the team's fallback topic, once +``` + +When a level's timeout passes with the incident still `triggered`, the next +level is paged. Off the end of the ladder the whole thing runs again +`repeat_count` times, and after that the team's `fallback_topic` is paged once +as the end of the line. The incident stays open throughout: running out of +people to wake is not the same as somebody answering. + +**Acknowledging or resolving stops it**, which is the point — continuing to wake +people after somebody has said "I have this" is how a tool teaches people to +mute it. **Snoozing pauses it**: a deliberate "not now" holds the ladder where +it is, and it resumes when the snooze runs out. + +Every step is on the incident's timeline with the level and the names it woke, +so somebody reading it afterwards can tell why their phone rang at 04:00. A +level whose targets are all unreachable — no ntfy topic, a disabled account, an +empty rota — is recorded as `nobody reachable` and the ladder moves on rather +than stalling on a rung that cannot ring. + +**Reminders and escalation never both run.** A team with a ladder gets +escalation; a team without keeps the reminder behaviour exactly as it was. Two +pages for one silence is the surest way to get a tool muted. + +The ladder's `fallback_topic` is per team, unlike `TERDUT_NTFY_FALLBACK_TOPIC`, +which is the install-wide topic used when an incident opens with nobody on call. +They answer different questions: one is "nobody was scheduled", the other is +"everybody scheduled has been tried". diff --git a/docs/images/admin.png b/docs/images/admin.png new file mode 100644 index 0000000..ea088af Binary files /dev/null and b/docs/images/admin.png differ diff --git a/docs/images/alerts.png b/docs/images/alerts.png new file mode 100644 index 0000000..f556dcd Binary files /dev/null and b/docs/images/alerts.png differ diff --git a/docs/images/incident-note.png b/docs/images/incident-note.png new file mode 100644 index 0000000..0b4cb16 Binary files /dev/null and b/docs/images/incident-note.png differ diff --git a/docs/images/mobile-incident.png b/docs/images/mobile-incident.png new file mode 100644 index 0000000..4762343 Binary files /dev/null and b/docs/images/mobile-incident.png differ diff --git a/docs/images/mobile-queue.png b/docs/images/mobile-queue.png new file mode 100644 index 0000000..b9fb72a Binary files /dev/null and b/docs/images/mobile-queue.png differ diff --git a/docs/images/oncall.png b/docs/images/oncall.png new file mode 100644 index 0000000..0a2a667 Binary files /dev/null and b/docs/images/oncall.png differ diff --git a/docs/images/queue-dark.png b/docs/images/queue-dark.png new file mode 100644 index 0000000..75d231b Binary files /dev/null and b/docs/images/queue-dark.png differ diff --git a/docs/images/queue-light.png b/docs/images/queue-light.png new file mode 100644 index 0000000..8c05295 Binary files /dev/null and b/docs/images/queue-light.png differ diff --git a/docs/images/stats.png b/docs/images/stats.png new file mode 100644 index 0000000..68816a7 Binary files /dev/null and b/docs/images/stats.png differ diff --git a/docs/images/team-deadman.png b/docs/images/team-deadman.png new file mode 100644 index 0000000..e958cb4 Binary files /dev/null and b/docs/images/team-deadman.png differ diff --git a/docs/images/team-escalation.png b/docs/images/team-escalation.png new file mode 100644 index 0000000..45d88ca Binary files /dev/null and b/docs/images/team-escalation.png differ diff --git a/docs/incidents.md b/docs/incidents.md new file mode 100644 index 0000000..77cf82d --- /dev/null +++ b/docs/incidents.md @@ -0,0 +1,113 @@ +# Alerts and incidents + +_How alerts become incidents and how incidents are worked, notified, escalated and expired._ Back to the [README](../README.md) and the [documentation index](./README.md). + +There are two objects, and the difference between them is the whole design. + +**An alert is Alertmanager's record.** It has two states, `firing` and +`resolved`, one row per fingerprint, and no human ever writes to it. The API +exposes alerts read-only. + +**An incident is the work item.** It goes `triggered → acknowledged → resolved`, +carries an assignee, a snooze, notes and a timeline, and is the only thing people +act on. Many alerts belong to one incident. + +## Correlation uses Alertmanager's `groupKey` + +Alertmanager has already grouped alerts according to the `group_by` routing tree +you configured, and it sends the resulting `groupKey` and `groupLabels` on every +webhook. Incidents adopt that answer rather than re-grouping alerts a second +time — if you want different correlation, change `group_by` in +`alertmanager.yml` and terdut follows. + +At most one incident is open per `groupKey` at a time. Alerts firing in a group +that already has an open incident join it. The incident's `severity` is a +high-water mark — the highest `severity` label any of its alerts has carried — so +an incident that hit `critical` still reads as critical after the critical alert +clears. + +## Several clusters, one team + +A team with one Alertmanager per Kubernetes cluster, each posting to its own +source, needs two settings or the clusters run together. + +1. Give every alert a `cluster` label at the source. In Prometheus that is + `externalLabels: {cluster: prod-eu}` (kube-prometheus-stack: + `prometheus.prometheusSpec.externalLabels`). +2. Add `cluster` to `group_by` in `alertmanager.yml`. + +The second one is the one that matters. Incidents are matched on the team and +Alertmanager's `groupKey`, and the `groupKey` does not include external labels: +without `cluster` in `group_by`, the same alert in two clusters has the same +key and joins one incident. With it, each cluster gets its own, `cluster` is in +the incident's `group_labels`, and the web UI shows it as a coloured chip on the +queue, the incident and the alert list, instead of leaving it in the title. +An alert that is not grouped by `cluster` still shows the chip on the alert +list, which reads the label from the alert itself. + +The queue has a cluster dropdown once there are two or more values to choose +between. It filters on the incident's `cluster` group label +(`GET /api/incidents?cluster=...`), so it only sees incidents grouped by it. + +## An incident opens only on a new occurrence + +An incident opens when an alert **transitions into firing**: a fingerprint that +was never seen, an alert with a newer `startsAt`, or a resolved alert that +started again. The unchanged firing notifications Alertmanager re-sends every +`repeat_interval` are none of those, and open nothing. + +This is what makes closing an incident by hand mean something. Without the rule, +`POST /api/incidents/{id}/resolve` would be undone by the next re-send of an +alert that never stopped firing. + +## Leaving the open state + +- **Automatically**, once every alert under the incident has stopped firing — + whether by a resolved webhook or by the sweeper's + [stale-alert expiry](#stale-alert-expiry). The incident gets + `"resolution_source": "alerts"`. +- **By hand**, via `POST /api/incidents/{id}/resolve` + (`"resolution_source": "manual"`). This is **terminal**: a later occurrence in + that group opens a *new* incident rather than reopening this one. If the alert + underneath never stops firing, the incident stays closed — that is what + resolving by hand asserts. +- **On recovery**, for a [dead man's switch](./dead-mans-switch.md) incident whose + heartbeat started arriving again (`"resolution_source": "recovered"`). These + incidents have no member alerts, so the automatic cascade above cannot reach + them. + +To quieten an incident you expect to come back, snooze it instead +(`POST /api/incidents/{id}/snooze`). A snooze hides the incident from the default +list without closing it, and expires by simply falling into the past. + +## On-call assignment + +A new incident is assigned to whoever holds today's schedule entry at the moment +it opens (`GET /api/schedule/current`). If nobody is scheduled it opens +unassigned. Reassign with `POST /api/incidents/{id}/assign`. + +One person holds a given day, so `POST /api/schedule` refuses a date somebody +already has: taking a shift off the person expecting to be paged for it should +not be something a plain call does by accident. Pass `"replace": true` to take +them anyway. Either way the whole request is one transaction — a week where some +days are free and some are taken moves as a unit, and a failure leaves the rota +exactly as it was rather than with a hole in it. + +## Stale alert expiry + +A resolved webhook is the only signal that an alert has stopped firing, so a +notification that is dropped, silenced, or lost to a restart would otherwise pin +that alert as firing forever. A background sweeper resolves firing alerts that +Alertmanager has stopped refreshing, using either signal: + +- the `endsAt` watermark on the last notification has passed, or +- no webhook has refreshed the alert within `TERDUT_STALE_AFTER`. + +Alertmanager re-sends firing notifications every `repeat_interval`, which is what +keeps a live alert fresh — so `TERDUT_STALE_AFTER` must be comfortably larger +than your `repeat_interval` (default 4h), or live alerts will be resolved +prematurely. Alerts resolved this way are marked `"resolution_source": "expiry"` +to distinguish them from a real Alertmanager resolve (`"alertmanager"`). + +An expiry cascades: once it leaves an incident with nothing firing under it, the +incident resolves too, in the same sweep. diff --git a/docs/notifications.md b/docs/notifications.md new file mode 100644 index 0000000..d455c4a --- /dev/null +++ b/docs/notifications.md @@ -0,0 +1,62 @@ +# Push notifications + +_Pages through ntfy, who gets them and how to acknowledge from the notification._ Back to the [README](../README.md) and the [documentation index](./README.md). + +With `TERDUT_NTFY_URL` set, an incident that opens is pushed to the on-call +person's phone through [ntfy](https://ntfy.sh). Everybody sets their own topic +under *Account* in the web UI, where a **Send a test push** button proves it +before an incident has to; `PUT /api/users/{id}/notify` is the same thing over +the API, and an administrator may set somebody else's. A user with no topic +falls back to `TERDUT_NTFY_FALLBACK_TOPIC`, as does an incident that opens with +nobody on call. If neither yields a topic, nothing is queued. + +The **server** is the install's one ntfy, from `TERDUT_NTFY_URL`, and is not +something a user picks. Only the topic is per-person. + +A topic is a shared secret with the ntfy server: anyone who knows it can both +read the pages and publish to it, so an unguessable one is worth the trouble. +That is also why the topic never appears in an incident's timeline, which every +API key can read. + +Three things get pushed: + +- **triggered** — an incident opened. Priority follows severity (`critical` maps + to ntfy's max priority, the one that overrides the phone's quiet settings). +- **reminder** — the incident is still `triggered` after `TERDUT_NOTIFY_REPEAT`. + Repeats until somebody acts. Acknowledging, snoozing, resolving or archiving + all stop it — snooze is the mute button. +- **resolved** — every alert under the incident stopped firing. Only sent to + whoever was paged in the first place, and only for the automatic cascade: + resolving by hand pushes nothing, since the person who did it already knows. + +Notifications carry an **Acknowledge** button that acknowledges the incident +without opening anything. It POSTs to `/api/notify/ack/{token}`, an +unauthenticated route authorised by the 256-bit token in its path — minted fresh +per notification, scoped to one incident and one action, and valid for 24 hours. +A real API key is never put in a notification, because the message is stored on +the ntfy server and cached on the device. + +The token is **not** consumed by use. Acknowledging is idempotent, so a token +stays valid for its full 24 hours and a second tap is a no-op that reports the +incident's current state rather than an error — which is what you want when a +tap is retried on a flaky mobile connection. What bounds it is scope, not a use +count: one incident, one action, one day. Expired tokens are purged by the +sweeper. + +Two consequences worth planning for: + +- `/api/notify/ack/{token}` **must stay publicly reachable**, or the button will + not work when the responder is off your network. +- Notifications sent to the fallback topic carry **no** Acknowledge button. The + topic is shared, and a button on it would let any subscriber acknowledge as + somebody else. + +Delivery is a queue, not an inline call: the webhook writes a row and a +background notifier sends it within 30 seconds, retrying with exponential +backoff up to 8 attempts. Nothing about ingestion blocks on ntfy being reachable. + +Every delivery is recorded on the incident's timeline: a `notified` event once +ntfy accepts the publish, and a `notify_failed` event when a notification +exhausts its retries. Written from the result rather than at enqueue, so the +timeline says what actually happened — and a page that never landed is visible +instead of looking the same as one that did. diff --git a/docs/single-sign-on.md b/docs/single-sign-on.md new file mode 100644 index 0000000..6432c0c --- /dev/null +++ b/docs/single-sign-on.md @@ -0,0 +1,105 @@ +# Single sign-on (OIDC) + +_Signing in through an OpenID Connect provider, and mapping its groups to teams and administrators._ Back to the [README](../README.md) and the [documentation index](./README.md). + +terdut can sign people in through any OpenID Connect provider; the examples use +[Authentik](https://goauthentik.io/). Groups at the provider decide who may sign +in, which teams they belong to and whether they administer the install, much as +Grafana's OAuth role and org mapping does. Password login keeps working alongside +it unless you turn it off. + +**At the provider**, create an OAuth2/OpenID provider and an application for it: +a *confidential* client, redirect URI `/api/oidc/callback`, and +the `openid`, `profile` and `email` scopes. The issuer is the application's, e.g. +`https://auth.example.com/application/o/terdut/`. Then set: + +```sh +TERDUT_PUBLIC_URL=https://terdut.example.com +TERDUT_OIDC_ISSUER=https://auth.example.com/application/o/terdut/ +TERDUT_OIDC_CLIENT_ID=terdut +TERDUT_OIDC_CLIENT_SECRET=... +TERDUT_OIDC_ALLOWED_GROUPS=terdut-users,terdut-admins +TERDUT_OIDC_ADMIN_GROUP=terdut-admins +``` + +Which team a group grants is not server-wide config: each team names its own +group(s), set by that team's own owner (or an administrator) from its Members +tab, or `PUT /api/teams/{teamID}/oidc-groups {"member_group":"sre","owner_group":"sre-leads"}`. +A team must already exist before a group can grant access to it — the sync +never creates one. + +The web UI's sign-in page shows a "Sign in with " button (a plain link to +`/api/oidc/login`) above the password form, or instead of it when +`TERDUT_PASSWORD_LOGIN=false`; it asks `GET /api/auth/config` what the server offers +(`password_login`, `oidc.enabled`, `oidc.name`). A refused sign-in comes back to that +page with the reason spelled out. Access the groups grant is badged **SSO** on the +Team, Admin and per-user pages, with its edit and remove controls disabled, and the +Account page does not offer to set a password nobody could use. + +**What a sign-in does** + +1. *Who.* The provider's `(issuer, subject)` is the identity. The first time, a + user is found by email — only when the provider marks it verified, or + `TERDUT_OIDC_TRUST_EMAIL` is set — or created with no password. A username taken + by somebody else gets a numeric suffix (`alice-2`). Username and email follow the + provider at each sign-in. Authentik reports `email_verified` as false unless + configured otherwise, so linking existing users usually needs + `TERDUT_OIDC_TRUST_EMAIL=true`. +2. *Whether.* With `TERDUT_OIDC_ALLOWED_GROUPS` set, somebody in none of them is + refused and nothing is created. +3. *What.* The administrator flag follows `TERDUT_OIDC_ADMIN_GROUP`. Team roles + follow each team's own `oidc_member_group`/`oidc_owner_group`; where both of a + team's groups match, the owner group wins. + +**Managed access.** What the sync grants is marked as managed by single sign-on, +and only that is ever changed by it. It is added at sign-in, and removed at the +next sign-in after the group is gone, even if that leaves a team without an owner +(an administrator can always repair a team) — the provider is the source of truth +for what it grants, so the last-owner and last-administrator guards do not apply. +Memberships and administrators added by hand are left alone; the exception is a +hand-added member whose team's own group grants a *higher* role, who is raised and +from then on managed. Editing managed access by hand (`POST` or `DELETE` on a +team's members, revoking an SSO-granted administrator) is refused with `409`, since +the next sign-in would undo it. + +> **Upgrading past migration 013: reconfigure every team's groups.** +> `TERDUT_OIDC_GROUP_MAPPINGS` is gone, and the sync no longer creates a team by +> name. Group-to-team-role mapping is now each team's own setting — an owner sets +> it from the Members tab, or `PUT /api/teams/{teamID}/oidc-groups`. Until a team's +> owner does that, an OIDC-sourced membership in it is dropped at that user's next +> SSO sign-in, the same as any other loss of group access. Set every team's groups +> before affected users next sign in, to avoid a visible gap in access. + +**How fast changes arrive.** Groups are read only at sign-in. A session made by an +SSO sign-in has a hard ceiling (`TERDUT_OIDC_SESSION_MAX_AGE`, default 12h) that +sliding never extends, so a change at the provider reaches terdut within that time. +Password sessions are unaffected. + +> **API keys are not revoked when somebody is removed at the provider.** terdut +> holds no refresh token and never asks the provider again, so a person removed +> from every allowed group loses their sessions within `TERDUT_OIDC_SESSION_MAX_AGE` +> and cannot sign in again, but keeps any API key they made (the TUI and scripts use +> them) until an administrator disables the user in terdut. + +**Signing in from a terminal.** A client with no browser of its own, such as the +TUI over SSH, signs in with a device code, run by terdut itself so the terminal +never talks to the provider: + +1. The terminal calls `POST /api/oidc/device` and shows the person a link + (`/device?code=XXXX-XXXX`) and the code. +2. On any device the person opens the link, signs in (by the provider or by + password, whatever the login page offers), sees the code and the account, and + presses **Approve**. Only a browser session can approve; an API key cannot. +3. The terminal polls `POST /api/oidc/device/token` every 5 seconds and is given the + ordinary `terdut_session` cookie once. A person who signs in through the provider + gets the same `TERDUT_OIDC_SESSION_MAX_AGE` ceiling on the terminal's session as + on their browser's. + +A login expires after 10 minutes. `GET /api/auth/config` reports `device_login`. + +**If the provider is down**, terdut still starts (discovery is fetched on first +use) and password login is the way in. With `TERDUT_PASSWORD_LOGIN=false` that way +is closed: set it back to `true`. The first administrator comes from the bootstrap +endpoint, and stays a manual administrator that no group can revoke; on an SSO-only +install set `bootstrap.enabled: false` in the chart if you don't want that account, +or keep it and never give it a password. diff --git a/docs/web-ui.md b/docs/web-ui.md new file mode 100644 index 0000000..5c20c3f --- /dev/null +++ b/docs/web-ui.md @@ -0,0 +1,90 @@ +# The web UI + +_What the web UI offers, how sign-in and sessions work, and the Team and Admin tabs._ Back to the [README](../README.md) and the [documentation index](./README.md). + +The server serves a web UI at `/`: the incident queue, each incident's alerts +and timeline with every action (acknowledge, assign, snooze, note, resolve, +archive), who is on call, the alert feed, and an *Account* tab for your own +password and the ntfy topic your pages go to. It is built for a phone first. On a phone +it navigates through a hamburger menu and has a sticky action bar, it follows the +system's dark mode, and it can be added to the home screen. From 900px wide it switches +to a sidebar with the queue and the incident side by side. The Stats page shows +incident counts, MTTA and MTTR, and alert frequency by name, hour and day over a +chosen range. + +You sign in with a username and password. Users have no password until one is +set, and a user without one can only use API keys: + +```bash +# an admin sets someone's first password with their API key +curl -X PUT http://localhost:8080/api/users/2/password \ + -H "Authorization: Bearer $KEY" -H "Content-Type: application/json" \ + -d '{"password": ""}' +``` + +After that, users change it themselves under *Account*. Changing your own +password requires the current one. + +How a browser stays signed in: + +- A successful login sets an `HttpOnly`, `SameSite=Lax` session cookie. It lasts + 30 days and slides forward while it is used, so an on-call phone stays signed + in. +- The cookie is marked `Secure` when `TERDUT_PUBLIC_URL` starts with `https://`, + so set it to the HTTPS address. TLS terminates at the gateway and the server + itself only ever sees plain HTTP. +- Requests authenticated by the cookie are checked for cross-origin use (Go's + `http.CrossOriginProtection`). That is the CSRF guard. Bearer-key clients are + not affected. +- Setting a password signs that user out everywhere else. +- Ten failed logins for one username within 15 minutes lock that username for + the rest of the window. + +With `TERDUT_PUBLIC_URL` set, tapping a push notification opens the incident in +the web UI (`/incidents/{id}`). + +A **Team** tab holds everything a team owns, in five sub-sections with a URL +each and a strip across the top to move between them: the on-call rota +(`/team/rota`), the membership (`/team/members`), the escalation ladder +(`/team/escalation`), the alert sources with their keys (`/team/sources`) and +the dead man's switches (`/team/deadman`). `/team` itself is an overview — who +is on call today, how many members and owners, how many ladder levels, how many +keys and how many switches — so a page fetches only what it shows. An owner +edits it; a member sees the same pages read-only, because the server refuses +their writes anyway. Somebody in more than one team picks between them above +the strip, since the choice changes the subject of all five. + +The rota is a month at a time, one coloured initial per day with a legend +underneath, and it says how many days are left uncovered — the question a rota +is read for is who holds which stretch, and a run of one colour answers it +where a list of dates does not. An owner taps a day to hand it to somebody or +empty it, and fills a whole shift from the range form folded in below. + +The **Admin** tab appears only for a system administrator, and holds what +belongs to the whole server rather than to one team. It has three sub-sections, +each with a URL of its own and a strip across the top to move between them: +every team (`/admin/teams`), every user (`/admin/users`), and the settings that +used to be environment variables (`/admin/settings`). `/admin` itself is an +overview — how many of each, and what each section is for. Adding somebody is +minting them an invite link into a team, rather than creating a bare account: +the person who accepts it picks their own password, so one never passes through +an administrator, and the link carries the team, so they land somewhere with a +queue in it. That happens on the team's own page, since an invite is a fact +about a team; the user list points there rather than asking which team beside a +form. + +A name in the team list opens **that team's page**, at `/admin/teams/{id}`: when it +was created, how many are in it and how much is open, a field to rename it, the +members with their roles, the invites into it, and deletion. The member list is the +one thing there that needed a new endpoint — `GET /api/teams/{id}/members` is +member-only and answers `404` to an administrator who is not in the team, which is +the rule and not an oversight, so the page reads `GET /api/admin/teams/{id}` instead. +An administrator still sees none of that team's incidents, alerts or rota. + +A name in the user list opens **that person's page**, at `/admin/users/{id}`: their +email and when they joined, where their notifications go, whether they are an +administrator, whether the account is disabled, the teams they are in with their +role in each, a password field for a first or forgotten one, and deletion. It is +the one place membership is edited from the person's side — the Team tab answers +"who is in this team", and answering "which teams is this person in" there means +visiting each team in turn. diff --git a/internal/db/migrations/001_baseline.sql b/internal/db/migrations/001_baseline.sql deleted file mode 100644 index 2cfb54e..0000000 --- a/internal/db/migrations/001_baseline.sql +++ /dev/null @@ -1,182 +0,0 @@ --- The Postgres baseline: the schema as it stood at the end of the SQLite line, --- in one file rather than ten. --- --- The ten SQLite migrations are in git history up to the commit that introduced --- this one, and they replay against nothing here: their shape was incremental --- (columns added, then dropped again in 008) and 008's backfill rewrote data --- that a Postgres install never had. An existing SQLite database is carried over --- by scripts/sqlite-to-postgres.go, which copies rows into this schema. --- --- Two conventions inherited deliberately: --- --- * Timestamps are BIGINT unix seconds, not timestamptz. Everything in Go --- already speaks epochs, and converting was a second change riding along --- with the port. Worth revisiting on its own. --- --- * Ids are GENERATED BY DEFAULT, not ALWAYS, so the migration script can --- insert rows with their original ids and keep every foreign key intact. --- setval at the end of the copy puts the sequences past them. - -CREATE TABLE users ( - id BIGINT GENERATED BY DEFAULT AS IDENTITY PRIMARY KEY, - username TEXT NOT NULL UNIQUE, - email TEXT NOT NULL UNIQUE, - created_at BIGINT NOT NULL DEFAULT FLOOR(EXTRACT(EPOCH FROM now()))::bigint, - -- Where this user's notifications go. NULL means they get none; incidents - -- assigned to them fall back to the configured fallback topic. - ntfy_topic TEXT, - -- NULL means the user has no password and can only use API keys. - password_hash TEXT -); - -CREATE TABLE api_keys ( - id BIGINT GENERATED BY DEFAULT AS IDENTITY PRIMARY KEY, - user_id BIGINT NOT NULL REFERENCES users(id) ON DELETE CASCADE, - key_hash TEXT NOT NULL UNIQUE, - name TEXT NOT NULL, - created_at BIGINT NOT NULL DEFAULT FLOOR(EXTRACT(EPOCH FROM now()))::bigint, - last_used_at BIGINT -); - --- A session is a browser's credential, the cookie counterpart of an API key: --- only the hash of the token is stored. expires_at slides forward while the --- session is in use, so an on-call phone stays signed in. -CREATE TABLE sessions ( - id BIGINT GENERATED BY DEFAULT AS IDENTITY PRIMARY KEY, - token_hash TEXT NOT NULL UNIQUE, - user_id BIGINT NOT NULL REFERENCES users(id) ON DELETE CASCADE, - created_at BIGINT NOT NULL, - last_seen_at BIGINT NOT NULL, - expires_at BIGINT NOT NULL, - user_agent TEXT -); - -CREATE INDEX idx_sessions_user ON sessions(user_id); - --- The machine-owned signal record: what Alertmanager says is true right now. --- Workflow state lives on incidents, never here, because the webhook upsert owns --- these rows and would overwrite it. -CREATE TABLE alerts ( - id BIGINT GENERATED BY DEFAULT AS IDENTITY PRIMARY KEY, - fingerprint TEXT NOT NULL UNIQUE, - name TEXT NOT NULL, - status TEXT NOT NULL CHECK (status IN ('firing', 'resolved')), - labels JSONB NOT NULL DEFAULT '{}'::jsonb, - annotations JSONB NOT NULL DEFAULT '{}'::jsonb, - starts_at BIGINT NOT NULL, - ends_at BIGINT, - generator_url TEXT NOT NULL DEFAULT '', - received_at BIGINT NOT NULL DEFAULT FLOOR(EXTRACT(EPOCH FROM now()))::bigint, - archived_at BIGINT, - -- Why the alert left the firing state: 'alertmanager' when a resolved - -- webhook set it, 'expiry' when the sweeper inferred it from staleness. - resolution_source TEXT -); - -CREATE INDEX alerts_status_idx ON alerts(status); -CREATE INDEX alerts_name_idx ON alerts(name); -CREATE INDEX alerts_received_at_idx ON alerts(received_at DESC); -CREATE INDEX alerts_archived_at_idx ON alerts(archived_at); - -CREATE TABLE schedule_entries ( - id BIGINT GENERATED BY DEFAULT AS IDENTITY PRIMARY KEY, - user_id BIGINT NOT NULL REFERENCES users(id) ON DELETE CASCADE, - date TEXT NOT NULL UNIQUE, -- YYYY-MM-DD; one person per day - created_at BIGINT NOT NULL DEFAULT FLOOR(EXTRACT(EPOCH FROM now()))::bigint -); - -CREATE INDEX schedule_entries_date_idx ON schedule_entries(date); - --- The human work item: what people acknowledge, assign, snooze, discuss and --- resolve. Correlation uses Alertmanager's own groupKey, so incidents follow the --- group_by routing tree the operator already tuned. -CREATE TABLE incidents ( - id BIGINT GENERATED BY DEFAULT AS IDENTITY PRIMARY KEY, - group_key TEXT NOT NULL, -- Alertmanager groupKey, opaque - title TEXT NOT NULL, -- rendered from group_labels - group_labels JSONB NOT NULL DEFAULT '{}'::jsonb, - status TEXT NOT NULL CHECK (status IN ('triggered', 'acknowledged', 'resolved')), - severity TEXT, -- highest `severity` label across firing members - triggered_at BIGINT NOT NULL, - acknowledged_by BIGINT REFERENCES users(id) ON DELETE SET NULL, - acknowledged_at BIGINT, - assigned_to BIGINT REFERENCES users(id) ON DELETE SET NULL, - snoozed_until BIGINT, - resolved_at BIGINT, - resolution_source TEXT, -- 'alerts' | 'manual' - archived_at BIGINT -); - --- Load-bearing: at most one OPEN incident per group_key. This is what makes --- "resolved incident + a new alert occurrence = a new incident" work, and it is --- the constraint the webhook's find-or-open lookup relies on. -CREATE UNIQUE INDEX incidents_open_group_key_idx ON incidents(group_key) WHERE resolved_at IS NULL; -CREATE INDEX incidents_status_idx ON incidents(status); -CREATE INDEX incidents_triggered_at_idx ON incidents(triggered_at DESC); -CREATE INDEX incidents_archived_at_idx ON incidents(archived_at); - --- Membership is historical, not a pointer on alerts: one alert row (one --- fingerprint) resolves and re-fires over time and belongs to a different --- incident each occurrence. -CREATE TABLE incident_alerts ( - incident_id BIGINT NOT NULL REFERENCES incidents(id) ON DELETE CASCADE, - alert_id BIGINT NOT NULL REFERENCES alerts(id) ON DELETE CASCADE, - added_at BIGINT NOT NULL DEFAULT FLOOR(EXTRACT(EPOCH FROM now()))::bigint, - PRIMARY KEY (incident_id, alert_id) -); - -CREATE INDEX incident_alerts_alert_id_idx ON incident_alerts(alert_id); - --- The timeline. Append-only, and the only history this server keeps: alert rows --- are mutated in place, so without this there is no record that anything --- happened. Notes are events too, so one query renders the whole story. -CREATE TABLE incident_events ( - id BIGINT GENERATED BY DEFAULT AS IDENTITY PRIMARY KEY, - incident_id BIGINT NOT NULL REFERENCES incidents(id) ON DELETE CASCADE, - -- triggered | alert_added | alert_resolved | acknowledged | unacknowledged - -- | assigned | snoozed | unsnoozed | resolved | note | notified | notify_failed - type TEXT NOT NULL, - user_id BIGINT REFERENCES users(id) ON DELETE SET NULL, -- NULL = the server acted - alert_id BIGINT REFERENCES alerts(id) ON DELETE SET NULL, - detail TEXT, - created_at BIGINT NOT NULL DEFAULT FLOOR(EXTRACT(EPOCH FROM now()))::bigint -); - -CREATE INDEX incident_events_incident_idx ON incident_events(incident_id, created_at); - --- Delivery is an outbox rather than an inline HTTP call: a POST made while --- holding the webhook's transaction would hold a connection open across a --- network round trip. The webhook inserts a row; the notifier goroutine --- delivers it. -CREATE TABLE notifications ( - id BIGINT GENERATED BY DEFAULT AS IDENTITY PRIMARY KEY, - incident_id BIGINT NOT NULL REFERENCES incidents(id) ON DELETE CASCADE, - -- Nullable: a notification sent to the fallback topic belongs to nobody, - -- because nobody was on call when the incident opened. - user_id BIGINT REFERENCES users(id) ON DELETE SET NULL, - topic TEXT NOT NULL, -- resolved at enqueue: who was on call then - kind TEXT NOT NULL CHECK (kind IN ('triggered', 'reminder', 'resolved')), - created_at BIGINT NOT NULL, - send_after BIGINT NOT NULL, -- retry backoff watermark - attempts BIGINT NOT NULL DEFAULT 0, - sent_at BIGINT, - last_error TEXT -- kept after the last attempt, for debugging -); - --- The delivery loop's only query: what is due and still unsent. -CREATE INDEX notifications_pending_idx ON notifications(send_after) WHERE sent_at IS NULL; --- Reminders and resolved notices both look up an incident's newest row. -CREATE INDEX notifications_incident_idx ON notifications(incident_id, id DESC); - --- A notification body is stored on the ntfy server and cached on the device, so --- a real API key must never appear in one. Each delivery mints its own token --- instead: one incident, one action, one day. -CREATE TABLE incident_ack_tokens ( - token_hash TEXT PRIMARY KEY, -- SHA-256 of the raw token, as with api_keys - incident_id BIGINT NOT NULL REFERENCES incidents(id) ON DELETE CASCADE, - user_id BIGINT NOT NULL REFERENCES users(id) ON DELETE CASCADE, - created_at BIGINT NOT NULL, - expires_at BIGINT NOT NULL -); - -CREATE INDEX incident_ack_tokens_expires_idx ON incident_ack_tokens(expires_at); diff --git a/internal/db/migrations/002_admin_role.sql b/internal/db/migrations/002_admin_role.sql deleted file mode 100644 index 99cc095..0000000 --- a/internal/db/migrations/002_admin_role.sql +++ /dev/null @@ -1,25 +0,0 @@ --- A system administrator role, and the first thing in this server that one user --- can do and another cannot. --- --- Until now every authenticated caller could create and delete users, set --- anybody's password and mint API keys for anybody — auth.go said so in a --- comment. That was defensible with one operator and a hand-made account; it is --- not once people sign themselves up (see #7). --- --- EVERY EXISTING USER BECOMES AN ADMIN. They already hold these powers, so --- this migration changes nobody's access: it names what is already true, and --- leaves demotion as a deliberate act somebody performs afterwards. The --- alternative — promoting only user 1 — would silently strip the others, and --- could leave an install whose only admin is an account nobody has a password --- for. --- --- New users are not admins: the column defaults to false, and the only ways to --- become one are this backfill, the bootstrap endpoint, or an existing admin --- granting it. -ALTER TABLE users ADD COLUMN is_admin BOOLEAN NOT NULL DEFAULT false; - -UPDATE users SET is_admin = true; - --- The queue's assignment dropdown and the on-call schedule read every user, and --- the admin screens in #5 will filter on this. -CREATE INDEX users_is_admin_idx ON users(is_admin) WHERE is_admin; diff --git a/internal/db/migrations/003_teams.sql b/internal/db/migrations/003_teams.sql deleted file mode 100644 index ea1358d..0000000 --- a/internal/db/migrations/003_teams.sql +++ /dev/null @@ -1,103 +0,0 @@ --- Teams: the unit of tenancy. Everything a person works on now belongs to one. --- --- Until this migration the install was one shared space — every user saw every --- alert and every incident, and the Alertmanager webhook was unauthenticated, so --- anything that could reach the port could open an incident for everybody. --- --- The shape, in one paragraph: a team owns its incidents, alerts, schedule and --- integrations. A user belongs to as many teams as they like, with a role in --- each: an `owner` configures the team, a `member` works its incidents. An --- integration key is what an alert arrives on, and the key is what says which --- team the alert belongs to. --- --- EVERYTHING EXISTING MOVES INTO ONE DEFAULT TEAM, and every existing user --- becomes an owner of it. That keeps an upgrade a no-op for the people using it: --- the same queue, the same schedule, the same incidents, with a name on them. - -CREATE TABLE teams ( - id BIGINT GENERATED BY DEFAULT AS IDENTITY PRIMARY KEY, - name TEXT NOT NULL UNIQUE, - created_at BIGINT NOT NULL DEFAULT FLOOR(EXTRACT(EPOCH FROM now()))::bigint -); - --- role is free text with a CHECK rather than an enum, so adding a third role --- later is a migration and not a type rewrite. -CREATE TABLE team_members ( - team_id BIGINT NOT NULL REFERENCES teams(id) ON DELETE CASCADE, - user_id BIGINT NOT NULL REFERENCES users(id) ON DELETE CASCADE, - role TEXT NOT NULL CHECK (role IN ('owner', 'member')), - joined_at BIGINT NOT NULL DEFAULT FLOOR(EXTRACT(EPOCH FROM now()))::bigint, - PRIMARY KEY (team_id, user_id) -); - -CREATE INDEX team_members_user_idx ON team_members(user_id); - --- How alerts get in, and the only thing that says which team they belong to. --- The key is stored as a SHA-256 hash, like api_keys and the ack tokens: a --- leaked database gives nobody the ability to post alerts. -CREATE TABLE integrations ( - id BIGINT GENERATED BY DEFAULT AS IDENTITY PRIMARY KEY, - team_id BIGINT NOT NULL REFERENCES teams(id) ON DELETE CASCADE, - kind TEXT NOT NULL CHECK (kind IN ('alertmanager')), - name TEXT NOT NULL, - key_hash TEXT NOT NULL UNIQUE, - created_at BIGINT NOT NULL DEFAULT FLOOR(EXTRACT(EPOCH FROM now()))::bigint, - last_used_at BIGINT -); - -CREATE INDEX integrations_team_idx ON integrations(team_id); - --- --------------------------------------------------------------------------- --- The default team, and everything that already exists moving into it. --- --- Created unconditionally, even on an empty install, so there is always a team --- for the bootstrap user to land in and for the first integration to hang off. --- --------------------------------------------------------------------------- - -INSERT INTO teams (name) VALUES ('Default'); - -INSERT INTO team_members (team_id, user_id, role) -SELECT (SELECT id FROM teams WHERE name = 'Default'), id, 'owner' FROM users; - --- --------------------------------------------------------------------------- --- team_id on everything a team owns. --- --- Added nullable, backfilled, then made NOT NULL: adding a NOT NULL column with --- no default to a table with rows is rejected, and a DEFAULT pointing at the --- default team would quietly keep working after the default team is gone. --- --------------------------------------------------------------------------- - -ALTER TABLE alerts ADD COLUMN team_id BIGINT REFERENCES teams(id) ON DELETE CASCADE; -ALTER TABLE incidents ADD COLUMN team_id BIGINT REFERENCES teams(id) ON DELETE CASCADE; -ALTER TABLE schedule_entries ADD COLUMN team_id BIGINT REFERENCES teams(id) ON DELETE CASCADE; - -UPDATE alerts SET team_id = (SELECT id FROM teams WHERE name = 'Default'); -UPDATE incidents SET team_id = (SELECT id FROM teams WHERE name = 'Default'); -UPDATE schedule_entries SET team_id = (SELECT id FROM teams WHERE name = 'Default'); - -ALTER TABLE alerts ALTER COLUMN team_id SET NOT NULL; -ALTER TABLE incidents ALTER COLUMN team_id SET NOT NULL; -ALTER TABLE schedule_entries ALTER COLUMN team_id SET NOT NULL; - --- --------------------------------------------------------------------------- --- The uniqueness rules were all written for one tenant, and every one of them --- is wrong now: two teams monitoring two clusters legitimately see the same --- fingerprint, the same groupKey, and want somebody on call on the same day. --- --------------------------------------------------------------------------- - -ALTER TABLE alerts DROP CONSTRAINT alerts_fingerprint_key; -CREATE UNIQUE INDEX alerts_team_fingerprint_idx ON alerts(team_id, fingerprint); - -DROP INDEX incidents_open_group_key_idx; --- Still load-bearing, now per team: at most one OPEN incident per group_key --- within a team. This is what makes "resolved incident + a new alert occurrence --- = a new incident" work, and what the webhook's find-or-open lookup relies on. -CREATE UNIQUE INDEX incidents_open_group_key_idx - ON incidents(team_id, group_key) WHERE resolved_at IS NULL; - -ALTER TABLE schedule_entries DROP CONSTRAINT schedule_entries_date_key; -CREATE UNIQUE INDEX schedule_entries_team_date_idx ON schedule_entries(team_id, date); - --- The list views all filter by team first. -CREATE INDEX alerts_team_received_idx ON alerts(team_id, received_at DESC); -CREATE INDEX incidents_team_triggered_idx ON incidents(team_id, triggered_at DESC); diff --git a/internal/db/migrations/004_team_deadman.sql b/internal/db/migrations/004_team_deadman.sql deleted file mode 100644 index 3a066fd..0000000 --- a/internal/db/migrations/004_team_deadman.sql +++ /dev/null @@ -1,39 +0,0 @@ --- Dead man's switches become a team's own configuration. --- --- They were three environment variables — TERDUT_DEADMAN_MATCHERS, _TIMEOUT and --- _SEVERITY — which made them one setting for the whole install. That was the --- last piece of the alerting path a team could not control: a team could take --- its own alerts on its own key and still not say which of them were --- heartbeats, or how long a silence had to last before somebody was paged. --- --- One row per team rather than one row per switch. The unit of monitoring is --- still the fingerprint, as it always was — two clusters sending the same --- heartbeat alertname are two independent switches — and the matcher string --- keeps the format the environment variable used, so a value can be moved from --- one to the other unchanged. --- --- No rows are seeded here: a migration cannot read the environment. The server --- inserts a row per team at startup from its own configuration, and the same --- values therefore carry forward into the first team's row without anybody --- retyping them. See seedDeadmanConfigs. -CREATE TABLE deadman_configs ( - team_id BIGINT PRIMARY KEY REFERENCES teams(id) ON DELETE CASCADE, - - -- ";" separates matchers, "," the label conditions within one, "=" is exact - -- equality: `alertname=Watchdog,cluster=prod; alertname=EdgeHeartbeat`. - -- Every matcher must name an alertname. Empty watches nothing. - matchers TEXT NOT NULL DEFAULT '', - - -- Seconds rather than a Go duration string: the column is compared and - -- arithmetic is done on it, and a value that has to be parsed before it can - -- be believed is a value that can be stored unparseable. Zero disables the - -- team's switches entirely. - timeout_seconds BIGINT NOT NULL DEFAULT 0, - - -- The severity these incidents open at. They have no member alerts to - -- derive one from, and a heartbeat's own severity label is meaningless — - -- Watchdog ships as "none". - severity TEXT NOT NULL DEFAULT 'critical', - - updated_at BIGINT NOT NULL DEFAULT FLOOR(EXTRACT(EPOCH FROM now()))::bigint -); diff --git a/internal/db/migrations/005_settings.sql b/internal/db/migrations/005_settings.sql deleted file mode 100644 index c9e060c..0000000 --- a/internal/db/migrations/005_settings.sql +++ /dev/null @@ -1,35 +0,0 @@ --- Settings that an administrator can change without a redeploy, and the flag --- that takes an account out of use without deleting it. --- --- Three of the server's tunables were environment variables, which meant --- changing how long an incident waits before it is paged again required editing --- a chart, merging it, and waiting for a reconcile. They are behaviour, not --- infrastructure, and the difference is who needs to change them and how often. --- --- What stays in the environment: the ntfy URL and token, the database DSN, the --- listen address and the public URL. Those are where the server is plugged in --- rather than how it behaves, they are needed before the database is open, and --- two of them are credentials. --- --- Key/value rather than a column per setting. A settings table with one row and --- a column per knob needs a migration for every new knob, and #6 and #7 will --- both add some. The cost is that values are text and the accessor has to say --- what type it wanted; settings.go does that in one place. --- --- No rows are seeded here: a migration cannot read the environment. The server --- inserts each key from its own configuration at startup, once, so an install --- that upgrades keeps exactly the behaviour it had. See SeedSettings. -CREATE TABLE settings ( - key TEXT PRIMARY KEY, - value TEXT NOT NULL, - updated_at BIGINT NOT NULL DEFAULT FLOOR(EXTRACT(EPOCH FROM now()))::bigint -); - --- Disabling an account rather than deleting it: the person has left, or the --- credential is suspect, and their incidents, acknowledgements and timeline --- entries must stay exactly where they are. Deleting a user nulls their --- acknowledged_by and assigned_to, which quietly rewrites history. --- --- A disabled user cannot sign in and their API keys stop working, but they are --- still a name the timeline can show and still a member of their teams. -ALTER TABLE users ADD COLUMN disabled_at BIGINT; diff --git a/internal/db/migrations/006_escalation.sql b/internal/db/migrations/006_escalation.sql deleted file mode 100644 index 245567d..0000000 --- a/internal/db/migrations/006_escalation.sql +++ /dev/null @@ -1,95 +0,0 @@ --- Escalation: page somebody else when the first person does not answer. --- --- This is the gap the whole multi-tenancy line of work was opened to close. --- Until now an unacknowledged incident re-paged the same topic every --- notify_repeat forever, which is a louder version of the same silence: if the --- person on call is asleep, has no signal, or has left, nothing else happens. --- --- Shape: one policy per team, an ordered list of levels, each level with a --- timeout and a set of targets. When a level's timeout passes and the incident --- is still triggered, the next level is paged. When the last level passes, the --- chain repeats repeat_count times, and then the team's fallback topic is paged --- once as the end of the line. --- --- A team WITHOUT a policy keeps exactly today's behaviour: page the assignee, --- then remind on the same topic. Escalation is opt-in per team, and the two --- never both run for one incident -- see enqueueReminders. -CREATE TABLE escalation_policies ( - -- One per team for now, hence the team as the key rather than an id with a - -- unique index: routing different alerts to different chains needs the - -- alert to carry something to route ON, which is a separate question. - team_id BIGINT PRIMARY KEY REFERENCES teams(id) ON DELETE CASCADE, - - -- How many extra times to run the whole chain after it has been walked - -- once. 0 means walk it once and stop at the fallback. - repeat_count BIGINT NOT NULL DEFAULT 0 CHECK (repeat_count >= 0 AND repeat_count <= 10), - - -- Where the last page goes when every level has been tried. Per team now: - -- TERDUT_NTFY_FALLBACK_TOPIC was one topic for the whole install, which in - -- a multi-team server pages the wrong people. Empty means the chain simply - -- ends. - fallback_topic TEXT NOT NULL DEFAULT '', - - updated_at BIGINT NOT NULL DEFAULT FLOOR(EXTRACT(EPOCH FROM now()))::bigint -); - -CREATE TABLE escalation_levels ( - id BIGINT GENERATED BY DEFAULT AS IDENTITY PRIMARY KEY, - team_id BIGINT NOT NULL REFERENCES escalation_policies(team_id) ON DELETE CASCADE, - -- 1-based, dense. The API rewrites the whole ladder on every edit rather - -- than patching one rung, so there is no way to leave a gap. - position BIGINT NOT NULL, - -- How long this level has to produce an acknowledgement before the next one - -- is paged. Seconds, like every other duration in this schema. - timeout_seconds BIGINT NOT NULL CHECK (timeout_seconds > 0), - - UNIQUE (team_id, position) -); - --- Who a level pages. Either a named person, or whoever the team's rota says is --- on call today -- which is the target that keeps working when the rota --- changes and nobody remembers to edit the policy. -CREATE TABLE escalation_targets ( - id BIGINT GENERATED BY DEFAULT AS IDENTITY PRIMARY KEY, - level_id BIGINT NOT NULL REFERENCES escalation_levels(id) ON DELETE CASCADE, - kind TEXT NOT NULL CHECK (kind IN ('user', 'oncall')), - -- Set for kind='user', NULL for kind='oncall'. - user_id BIGINT REFERENCES users(id) ON DELETE CASCADE, - - CHECK ((kind = 'user' AND user_id IS NOT NULL) OR (kind = 'oncall' AND user_id IS NULL)) -); - -CREATE INDEX escalation_targets_level_idx ON escalation_targets(level_id); - --- --------------------------------------------------------------------------- --- Where an incident is in its chain. --- --- On the incident rather than in a side table: it is read on every notifier --- tick alongside the incident's status, and one row per incident is exactly --- what the state is. --- --------------------------------------------------------------------------- - --- 0 means no level has been paged yet, which is the state of every incident --- that existed before escalation and of every incident in a team with no --- policy. 1 is the first level. -ALTER TABLE incidents ADD COLUMN escalation_level BIGINT NOT NULL DEFAULT 0; - --- When the current level was entered, and therefore what its timeout is --- measured from. NULL while escalation_level is 0. -ALTER TABLE incidents ADD COLUMN escalation_level_at BIGINT; - --- How many times the chain has been walked in full. Compared against the --- policy's repeat_count. -ALTER TABLE incidents ADD COLUMN escalation_round BIGINT NOT NULL DEFAULT 0; - --- The notifier's escalation query: incidents still waiting, oldest level first. -CREATE INDEX incidents_escalation_idx - ON incidents(escalation_level_at) - WHERE resolved_at IS NULL AND status = 'triggered'; - --- 'escalated' joins the outbox kinds: a page that went out because nobody --- answered the last one, which is worth telling apart from the first page and --- from a reminder when reading the timeline or debugging a delivery. -ALTER TABLE notifications DROP CONSTRAINT notifications_kind_check; -ALTER TABLE notifications ADD CONSTRAINT notifications_kind_check - CHECK (kind IN ('triggered', 'reminder', 'resolved', 'escalated')); diff --git a/internal/db/migrations/007_signup_invites.sql b/internal/db/migrations/007_signup_invites.sql deleted file mode 100644 index c615353..0000000 --- a/internal/db/migrations/007_signup_invites.sql +++ /dev/null @@ -1,49 +0,0 @@ --- Self-service sign-up, and the invite links that make it useful. --- --- Until now the only way to get an account was for somebody who already had one --- to create it, and the login page told people to "ask an admin". That is a --- workable arrangement for one operator and an impossible one for a team. --- --- An invite is a link, not an email: this server has no SMTP and adding it to --- send one message would be a new subsystem to run, secure and monitor. The --- person inviting sends the link however they already talk to the person they --- are inviting. -CREATE TABLE invites ( - id BIGINT GENERATED BY DEFAULT AS IDENTITY PRIMARY KEY, - - -- SHA-256 of the raw token, like api_keys, the integration keys and the - -- acknowledgement tokens. A leaked database hands nobody an account. - token_hash TEXT NOT NULL UNIQUE, - - -- Which team the invitee lands in, and as what. An invite always names a - -- team: an account in no team sees an empty queue and can be paged by - -- nobody, which is not a state to invite somebody into. - team_id BIGINT NOT NULL REFERENCES teams(id) ON DELETE CASCADE, - role TEXT NOT NULL CHECK (role IN ('owner', 'member')), - - created_by BIGINT REFERENCES users(id) ON DELETE SET NULL, - created_at BIGINT NOT NULL DEFAULT FLOOR(EXTRACT(EPOCH FROM now()))::bigint, - - -- Invites expire. A link that works forever is a credential nobody - -- remembers issuing, sitting in a chat log. - expires_at BIGINT NOT NULL, - - -- Single-use by default: max_uses 1. A team onboarding six people at once - -- can raise it rather than minting six links. - max_uses BIGINT NOT NULL DEFAULT 1 CHECK (max_uses > 0 AND max_uses <= 100), - uses BIGINT NOT NULL DEFAULT 0, - - -- Revoked by hand, separately from expiry, so "this link is no longer - -- wanted" and "this link timed out" stay distinguishable in the listing. - revoked_at BIGINT -); - -CREATE INDEX invites_team_idx ON invites(team_id); - --- Who redeemed which invite. Kept after the invite is gone — the answer to "how --- did this account get here" should outlive the link that made it. -ALTER TABLE users ADD COLUMN invited_via BIGINT REFERENCES invites(id) ON DELETE SET NULL; - --- Where a person is in the first-run checklist, so it can be resumed and --- dismissed rather than nagging forever. One row per user, created on demand. -ALTER TABLE users ADD COLUMN onboarding_dismissed_at BIGINT; diff --git a/internal/db/migrations/008_incident_signature.sql b/internal/db/migrations/008_incident_signature.sql deleted file mode 100644 index 3e63b31..0000000 --- a/internal/db/migrations/008_incident_signature.sql +++ /dev/null @@ -1,23 +0,0 @@ --- Similar incidents: a signature per incident, so "has this happened before" --- is an indexed equality instead of a search. --- --- The signature is the alert name plus the group labels that identify WHAT is --- broken, minus the ones that only say WHERE it happened to run this time --- (instance, pod, ...). Two incidents with the same signature in the same team --- are the same problem for a responder's purposes. --- --- Computed in Go for new incidents (incidentSignature in incident_store.go). --- The backfill below MUST produce the same string; keep the volatile list in --- both places in step. -ALTER TABLE incidents ADD COLUMN signature TEXT NOT NULL DEFAULT ''; - -UPDATE incidents SET signature = - COALESCE(NULLIF(group_labels->>'alertname', ''), title) || '|' || - COALESCE(( - SELECT string_agg(e.k || '=' || e.v, ',' ORDER BY e.k) - FROM jsonb_each_text(incidents.group_labels) AS e(k, v) - WHERE e.k <> 'alertname' - AND e.k NOT IN ('instance', 'pod', 'pod_name', 'pod_ip', 'container', 'container_name', 'endpoint') - ), ''); - -CREATE INDEX incidents_signature_idx ON incidents(team_id, signature, triggered_at DESC); diff --git a/internal/db/migrations/009_deadman_switches.sql b/internal/db/migrations/009_deadman_switches.sql deleted file mode 100644 index e693d46..0000000 --- a/internal/db/migrations/009_deadman_switches.sql +++ /dev/null @@ -1,54 +0,0 @@ --- Dead man's switches become rows of their own. --- --- 004 kept a team's switches in one string with one timeout and one severity, --- which was enough to configure them and not enough to show them: there was no --- thing to list, nothing to hang a status on, and every switch in a team had to --- share a deadline. A row per switch gives each its own name, matcher, timeout --- and severity, and gives the Team → Switches page something to be a list of. --- --- The matcher keeps the syntax the string used, one matcher per row: --- `alertname=Watchdog,cluster=prod`. The unit of monitoring is still the --- fingerprint, so a matcher that many clusters satisfy is still one switch row --- watching several independent heartbeats. -CREATE TABLE deadman_switches ( - id BIGSERIAL PRIMARY KEY, - team_id BIGINT NOT NULL REFERENCES teams(id) ON DELETE CASCADE, - - -- What the owner calls it. Defaults to the matcher when they do not say. - name TEXT NOT NULL, - - -- "," separates the label conditions, "=" is exact equality, and alertname is - -- mandatory: it is what keeps the sweeper's candidate query on an index. - matcher TEXT NOT NULL, - - -- Seconds of silence before the switch is declared dead. Never zero: a switch - -- that cannot fire is deleted, not disabled. - timeout_seconds BIGINT NOT NULL CHECK (timeout_seconds > 0), - - -- The severity its incidents open at. See 004 for why they carry their own. - severity TEXT NOT NULL DEFAULT 'critical', - - created_at BIGINT NOT NULL DEFAULT FLOOR(EXTRACT(EPOCH FROM now()))::bigint -); - -CREATE INDEX deadman_switches_team_idx ON deadman_switches (team_id); - --- Carry every team's configuration over, one row per matcher. A team whose --- timeout was zero had switches turned off, which is now "no rows". -INSERT INTO deadman_switches (team_id, name, matcher, timeout_seconds, severity) -SELECT c.team_id, btrim(m), btrim(m), c.timeout_seconds, c.severity -FROM deadman_configs c, - LATERAL regexp_split_to_table(c.matchers, ';') AS m -WHERE c.timeout_seconds > 0 - AND btrim(m) <> '' -ORDER BY c.team_id; - --- The server seeds environment defaults into teams once, and remembers that it --- did. An install that had a row per team was already seeded; without this --- marker the first start after upgrading would seed teams that had switched --- theirs off. -INSERT INTO settings (key, value) -SELECT 'deadman_seeded', '1' -WHERE EXISTS (SELECT 1 FROM deadman_configs); - -DROP TABLE deadman_configs; diff --git a/internal/db/migrations/010_alert_source.sql b/internal/db/migrations/010_alert_source.sql deleted file mode 100644 index f397fad..0000000 --- a/internal/db/migrations/010_alert_source.sql +++ /dev/null @@ -1,21 +0,0 @@ --- Which alert source an alert last arrived on. --- --- Team -> Sources shows when each source last posted, which integrations --- already knew (last_used_at, stamped on every webhook). What it could not say --- was what a source delivered: an alert never recorded the key it came in on, so --- "prod alertmanager" and "staging alertmanager" were indistinguishable once --- inside. This column is that link, and lets the page show each source's last --- alert and how many alerts it has kept fresh over the past day. --- --- Last sender wins: every accepted payload restamps it, the way it advances --- received_at. Two sources posting the same fingerprint into one team is --- already one alert, and it is attributed to whichever spoke last. --- --- Nullable, and not backfilled. Alerts that arrived before this migration have --- no source, and NULL says so honestly rather than guessing. It heals by itself: --- Alertmanager re-sends every alert each repeat_interval, and each re-send is an --- accepted payload. Deleting a source keeps its alerts, unattributed. -ALTER TABLE alerts ADD COLUMN integration_id BIGINT REFERENCES integrations(id) ON DELETE SET NULL; - -CREATE INDEX alerts_integration_idx ON alerts (integration_id, received_at) - WHERE integration_id IS NOT NULL; diff --git a/internal/db/migrations/011_oidc.sql b/internal/db/migrations/011_oidc.sql deleted file mode 100644 index 557d72c..0000000 --- a/internal/db/migrations/011_oidc.sql +++ /dev/null @@ -1,60 +0,0 @@ --- Single sign-on through an OpenID Connect provider (Authentik, and anything --- else that speaks OIDC). --- --- Four things change, and none of them touches a password user: every new column --- has a default that says "this is how it has always worked". --- --- 1. user_identities says which provider account a user is. It is keyed on --- (issuer, subject), never on email or username: those are mutable at the --- provider, and a recycled address must not inherit somebody's account. A --- user can have several identities (a second provider later), and none at all --- (a local, password-only user), which is why this is a table and not two --- columns on users. --- --- 2. team_members.source and users.admin_source record who granted a role. 'oidc' --- rows are owned by the group sync: it adds them when a group grants access --- and removes them when it stops, and nothing else may edit them. 'manual' rows --- are everything that existed before this migration, and are never touched by --- the sync. Without the marker the sync could not tell a membership it created --- from one an owner added by hand, and would have to either leave stale access --- behind or delete people it had no business deleting. --- --- 3. sessions.max_expires_at is a hard ceiling on a session's life. Ordinary --- sessions slide for as long as they are used; a session made by an SSO login --- must not, because the login is the only moment the groups are re-read. --- Capping the session is what makes "removed from the group in the provider" --- take effect within a bounded time. NULL means no ceiling. --- --- 4. oidc_logins holds a login that has been started and not yet finished: the --- state, nonce and PKCE verifier the callback must see again. A row rather --- than a signed cookie, so it survives a restart and needs no signing key. --- Only the hash of the state is stored, like every other token here; the --- nonce and verifier are useless without the state that names the row. -CREATE TABLE user_identities ( - id BIGINT GENERATED BY DEFAULT AS IDENTITY PRIMARY KEY, - user_id BIGINT NOT NULL REFERENCES users(id) ON DELETE CASCADE, - issuer TEXT NOT NULL, - subject TEXT NOT NULL, - created_at BIGINT NOT NULL DEFAULT FLOOR(EXTRACT(EPOCH FROM now()))::bigint, - last_login_at BIGINT NOT NULL DEFAULT FLOOR(EXTRACT(EPOCH FROM now()))::bigint, - UNIQUE (issuer, subject) -); - -CREATE INDEX user_identities_user_idx ON user_identities (user_id); - -ALTER TABLE team_members - ADD COLUMN source TEXT NOT NULL DEFAULT 'manual' CHECK (source IN ('manual', 'oidc')); - -ALTER TABLE users - ADD COLUMN admin_source TEXT NOT NULL DEFAULT 'manual' CHECK (admin_source IN ('manual', 'oidc')); - -ALTER TABLE sessions ADD COLUMN max_expires_at BIGINT; - -CREATE TABLE oidc_logins ( - state_hash TEXT PRIMARY KEY, - nonce TEXT NOT NULL, - pkce_verifier TEXT NOT NULL, - expires_at BIGINT NOT NULL -); - -CREATE INDEX oidc_logins_expires_idx ON oidc_logins (expires_at); diff --git a/internal/db/migrations/012_device_login.sql b/internal/db/migrations/012_device_login.sql deleted file mode 100644 index 1764e62..0000000 --- a/internal/db/migrations/012_device_login.sql +++ /dev/null @@ -1,40 +0,0 @@ --- Signing in from a terminal, for clients that cannot open a browser on the --- machine they run on (the TUI over SSH is the reason). --- --- The flow is the OAuth device authorization grant, run by terdut itself rather --- than the identity provider, so the terminal never talks to the provider and --- the server issues its ordinary session at the end: --- --- 1. The terminal asks for a login and gets two secrets: a device code it --- keeps and polls with, and a short user code it shows the person. --- 2. The person opens the verification URL on any device, signs in by whatever --- means the server offers, sees the user code, and approves it. --- 3. The terminal's next poll finds the row approved and is given a session. --- --- Only the hash of the device code is stored, like every other token here: the --- device code is what earns a session, so a database read must not yield one. --- The user code is shown on screens and typed by people, so it is stored as is; --- on its own it can only be approved, never redeemed. --- --- user_id is the person who approved. It is empty until then, and the session --- is minted at redemption, not at approval: an approval nobody collects must not --- leave a live session lying about. --- --- last_polled_at lets the server refuse a client that polls faster than the --- interval it was told. -CREATE TABLE device_logins ( - device_hash TEXT PRIMARY KEY, - user_code TEXT NOT NULL UNIQUE, - status TEXT NOT NULL DEFAULT 'pending' CHECK (status IN ('pending', 'approved', 'denied')), - user_id BIGINT REFERENCES users(id) ON DELETE CASCADE, - expires_at BIGINT NOT NULL, - last_polled_at BIGINT NOT NULL DEFAULT 0 -); - -CREATE INDEX device_logins_expires_idx ON device_logins (expires_at); - --- Where to send the browser once a single sign-on login completes. A person who --- opens /device?code=... without a session has to sign in first and then come --- back to it, and the same is true of any other deep link. Validated when it is --- stored: only a path on this server is ever kept. -ALTER TABLE oidc_logins ADD COLUMN next TEXT NOT NULL DEFAULT '/'; diff --git a/internal/db/migrations/013_oidc_team_groups.sql b/internal/db/migrations/013_oidc_team_groups.sql deleted file mode 100644 index d269402..0000000 --- a/internal/db/migrations/013_oidc_team_groups.sql +++ /dev/null @@ -1,26 +0,0 @@ --- Per-team OIDC group configuration, replacing the global --- TERDUT_OIDC_GROUP_MAPPINGS env var. --- --- Group -> team -> role used to be one global list an operator set for the --- whole install, matched against a team by name, and the sync would create --- the team if no team by that name existed yet. That put the decision of --- which group controls a team in the server's environment rather than the --- team's own hands, meant changing it needed an env var edit and a restart, --- and let a typo in a team name silently create a stray team. --- --- Each team now names, itself, which group grants membership and which --- grants ownership. Nullable: most teams need neither. No uniqueness --- constraint on either column — two teams may legitimately watch the same --- provider group (a broad team and a narrower one both keyed off overlapping --- groups is a choice for their owners to make, not one the schema should --- refuse). --- --- BREAKING CHANGE, deliberately not auto-migrated: TERDUT_OIDC_GROUP_MAPPINGS --- stops being read as of this version, and the sync no longer creates a team --- by name. Every team's group binding must be set again through --- PUT /api/teams/{teamID}/oidc-groups. Until an owner does that, an --- OIDC-sourced membership in that team is dropped at that user's next SSO --- sign-in, the same way any other loss of group access is handled. See the --- README's OIDC section. -ALTER TABLE teams ADD COLUMN oidc_member_group TEXT; -ALTER TABLE teams ADD COLUMN oidc_owner_group TEXT; diff --git a/internal/db/migrations/014_service_accounts.sql b/internal/db/migrations/014_service_accounts.sql deleted file mode 100644 index bd93644..0000000 --- a/internal/db/migrations/014_service_accounts.sql +++ /dev/null @@ -1,43 +0,0 @@ --- Service accounts: a scoped, non-human credential for automation (e.g. --- terdut-operator) that needs to manage teams, escalation policies, dead --- man's switches, integrations and OIDC group bindings without impersonating --- a human user. See SERVICE-ACCOUNTS.md for the design this implements. --- --- Deliberately not a users row: no password_hash, no is_admin, no --- user_identities linkage, so a service account can never be pulled into --- OIDC group sync or password login, and is never mistaken for a human in an --- audit trail. --- --- scope is 'instance' (acts with the same reach system administration has --- over teams: create one, list them, mint a 'team'-scoped account against --- any of them) or 'team' (acts as that one team's owner, and nothing else). --- The CHECK ties team_id's presence to scope directly, rather than leaving it --- to application code to keep the two consistent. -CREATE TABLE service_accounts ( - id BIGINT GENERATED BY DEFAULT AS IDENTITY PRIMARY KEY, - name TEXT NOT NULL UNIQUE, - scope TEXT NOT NULL CHECK (scope IN ('instance', 'team')), - team_id BIGINT REFERENCES teams(id) ON DELETE CASCADE, - created_by BIGINT REFERENCES users(id) ON DELETE SET NULL, - created_at BIGINT NOT NULL DEFAULT FLOOR(EXTRACT(EPOCH FROM now()))::bigint, - CONSTRAINT service_accounts_scope_team_id_chk CHECK ( - (scope = 'team' AND team_id IS NOT NULL) OR - (scope = 'instance' AND team_id IS NULL) - ) -); - -CREATE INDEX service_accounts_team_id_idx ON service_accounts(team_id); - --- One account, many keys: rotation is minting a new one and revoking the --- old, the same shape api_keys already has, so an account's identity and --- audit history survive a rotation instead of being recreated by it. -CREATE TABLE service_account_keys ( - id BIGINT GENERATED BY DEFAULT AS IDENTITY PRIMARY KEY, - service_account_id BIGINT NOT NULL REFERENCES service_accounts(id) ON DELETE CASCADE, - key_hash TEXT NOT NULL UNIQUE, - name TEXT NOT NULL, - created_at BIGINT NOT NULL DEFAULT FLOOR(EXTRACT(EPOCH FROM now()))::bigint, - last_used_at BIGINT -); - -CREATE INDEX service_account_keys_service_account_id_idx ON service_account_keys(service_account_id); diff --git a/internal/db/migrations/015_incident_service_account_actors.sql b/internal/db/migrations/015_incident_service_account_actors.sql deleted file mode 100644 index b225f34..0000000 --- a/internal/db/migrations/015_incident_service_account_actors.sql +++ /dev/null @@ -1,39 +0,0 @@ --- Service-account actors on incident mutations (terdut-server#25). A --- team-scoped service account acknowledging/resolving/snoozing/noting an --- incident is not a users row, so it cannot be written into --- acknowledged_by/incident_events.user_id — doing so either violates the --- users(id) FK (new rows) or, for incident_events.user_id, silently matches --- zero rows on delete. These columns are the service-account-shaped parallel --- to the existing human ones: nullable, mutually exclusive with their human --- counterpart, ON DELETE SET NULL so a deleted service account doesn't take --- the incident history with it. -ALTER TABLE incidents - ADD COLUMN acknowledged_by_service_account_id BIGINT - REFERENCES service_accounts(id) ON DELETE SET NULL; - -ALTER TABLE incident_events - ADD COLUMN service_account_id BIGINT - REFERENCES service_accounts(id) ON DELETE SET NULL; - --- At most one actor kind per row: both NULL ("the server acted") is valid, --- exactly one set is valid, both set is a bug this constraint refuses to --- store rather than silently accepting. -ALTER TABLE incidents - ADD CONSTRAINT incidents_ack_actor_xor_chk CHECK ( - acknowledged_by IS NULL OR acknowledged_by_service_account_id IS NULL - ); - -ALTER TABLE incident_events - ADD CONSTRAINT incident_events_actor_xor_chk CHECK ( - user_id IS NULL OR service_account_id IS NULL - ); - -CREATE INDEX incidents_acknowledged_by_service_account_id_idx - ON incidents(acknowledged_by_service_account_id); -CREATE INDEX incident_events_service_account_id_idx - ON incident_events(service_account_id); - --- assigned_to_service_account_id is deliberately not added here: it would sit --- unpopulated until handleIncidentAssign itself tracks an actor, which is a --- separate, pre-existing gap (it records the assignee today, never the --- actor, for humans either) tracked in its own follow-up issue. diff --git a/internal/db/migrations/016_rate_limit_counters.sql b/internal/db/migrations/016_rate_limit_counters.sql deleted file mode 100644 index 5553401..0000000 --- a/internal/db/migrations/016_rate_limit_counters.sql +++ /dev/null @@ -1,16 +0,0 @@ --- Backs the rate limiters (failed logins, sign-ups, OIDC/device start) with --- Postgres instead of an in-memory map, now that the server runs more than --- one replica in production (v0.37.0): a counter that only ever sees its own --- pod's traffic quietly let every one of these limits through multiplied by --- the replica count. --- --- window_start is the start of the current fixed window for key, in the same --- "unix seconds" shape every other timestamp in this schema uses. The window --- resets rather than slides, matching the in-memory limiter it replaces: --- once a key's window is older than the limiter's window length, the next --- failure starts a fresh one instead of extending the stale one. -CREATE TABLE rate_limit_counters ( - key TEXT PRIMARY KEY, - window_start BIGINT NOT NULL, - count INT NOT NULL -); diff --git a/internal/db/migrations/017_api_key_expiry.sql b/internal/db/migrations/017_api_key_expiry.sql deleted file mode 100644 index 17f8ab5..0000000 --- a/internal/db/migrations/017_api_key_expiry.sql +++ /dev/null @@ -1,7 +0,0 @@ --- Optional expiry on a user's own API keys. NULL (the existing default for --- every row already in this table) means "never expires" -- the same --- behavior these keys have always had, so no existing integration breaks. --- Service account keys are deliberately NOT touched: they are a different --- table, managed by automation, and already distinguished by their own --- "tdsa_" prefix. -ALTER TABLE api_keys ADD COLUMN expires_at BIGINT; diff --git a/internal/db/migrations/018_incident_event_actor.sql b/internal/db/migrations/018_incident_event_actor.sql deleted file mode 100644 index cfee52f..0000000 --- a/internal/db/migrations/018_incident_event_actor.sql +++ /dev/null @@ -1,21 +0,0 @@ --- Who performed an assignment (terdut-server#35). On an 'assigned' event --- incident_events.user_id is the assignee, so the actor needs columns of its --- own. Only populated for 'assigned' events; every other event type keeps --- using user_id/service_account_id for the actor. Older 'assigned' rows stay --- NULL (the actor was never recorded). Same shape as migration 015: nullable, --- mutually exclusive, ON DELETE SET NULL. --- --- assigned_to_service_account_id is still deliberately not added: making --- service accounts assignable is a separate change (request body, assignee --- picker, notifier, filters). -ALTER TABLE incident_events - ADD COLUMN actor_user_id BIGINT REFERENCES users(id) ON DELETE SET NULL, - ADD COLUMN actor_service_account_id BIGINT REFERENCES service_accounts(id) ON DELETE SET NULL; - -ALTER TABLE incident_events - ADD CONSTRAINT incident_events_assign_actor_xor_chk CHECK ( - actor_user_id IS NULL OR actor_service_account_id IS NULL - ); - -CREATE INDEX incident_events_actor_user_id_idx ON incident_events(actor_user_id); -CREATE INDEX incident_events_actor_service_account_id_idx ON incident_events(actor_service_account_id);