Rewrite the README as highlights with screenshots; move the detail into docs/
The README was 1,240 lines of reference material and still described a SQLite quick start. It is now a short tour (highlights, screenshots of the web UI, an accurate quick start against Postgres), and each topic has its own page under docs/ with an index: deployment, configuration, Alertmanager, incidents, notifications, escalation, dead man's switches, single sign-on, web UI, API and development. SERVICE-ACCOUNTS.md is rewritten from a proposal into a reference, and TEAM-LOOKUP.md is gone with the endpoint it described. The "Upgrading to ..." sections for an unreleased product are dropped. Claude-Session: https://claude.ai/code/session_016mBLURvJoMuUEr9cB2RpUN
This commit is contained in:
Niklas Ye
@@ -0,0 +1,21 @@
|
||||
# Terminal Duty documentation
|
||||
|
||||
The [README](../README.md) is the short tour. These pages hold the detail.
|
||||
|
||||
**Running it**
|
||||
- [Deployment](./deployment.md): Docker, the Helm chart, the database, backups, and the operator.
|
||||
- [Configuration](./configuration.md): environment variables and settings.
|
||||
- [Single sign-on](./single-sign-on.md): OIDC, group mapping, the terminal device flow.
|
||||
|
||||
**Using it**
|
||||
- [The web UI](./web-ui.md): sessions, the Team and Admin tabs.
|
||||
- [Alertmanager configuration](./alertmanager.md): routes, integration keys and webhooks.
|
||||
- [Alerts and incidents](./incidents.md): correlation, lifecycle, on-call assignment, stale-alert expiry.
|
||||
- [Push notifications](./notifications.md): ntfy pages and acknowledging from them.
|
||||
- [Escalation](./escalation.md): ladders, repeats and the fallback topic.
|
||||
- [Dead man's switches](./dead-mans-switch.md): noticing that alerts stopped arriving.
|
||||
|
||||
**Integrating and contributing**
|
||||
- [API reference](./api.md): every endpoint, authentication and error shape.
|
||||
- [Service accounts](../SERVICE-ACCOUNTS.md): non-human credentials for automation.
|
||||
- [Development and releasing](./development.md): tests, the CI gate, the release pipeline.
|
||||
@@ -0,0 +1,62 @@
|
||||
# Alertmanager configuration
|
||||
|
||||
_Pointing Alertmanager at the server._ Back to the [README](../README.md) and the [documentation index](./README.md).
|
||||
|
||||
Alerts arrive on a team's **integration key**, which says both that the sender
|
||||
may post and which team the alerts belong to. Mint one as an owner of the team:
|
||||
|
||||
```bash
|
||||
curl -X POST https://terdut.example.com/api/teams/1/integrations \
|
||||
-H "Authorization: Bearer $TERDUT_API_KEY" \
|
||||
-H 'Content-Type: application/json' \
|
||||
-d '{"name":"prod alertmanager"}'
|
||||
```
|
||||
|
||||
The response carries the key and the full URL **once**; only a SHA-256 hash is
|
||||
stored. Put it in your `alertmanager.yml`:
|
||||
|
||||
```yaml
|
||||
receivers:
|
||||
- name: terdut
|
||||
webhook_configs:
|
||||
- url: http://terdut-server:8080/api/integrations/<key>/alertmanager
|
||||
send_resolved: true
|
||||
|
||||
route:
|
||||
receiver: terdut
|
||||
```
|
||||
|
||||
The whole URL is a credential, so treat it like one. Alertmanager 0.26 and
|
||||
later can read it from a file with `url_file:` instead, which keeps it out of
|
||||
your configuration repository:
|
||||
|
||||
```yaml
|
||||
- url_file: /etc/alertmanager/secrets/terdut-webhook-url/url
|
||||
send_resolved: true
|
||||
```
|
||||
|
||||
The webhook endpoint requires no authentication.
|
||||
|
||||
If you use the [dead man's switch](./dead-mans-switch.md) — and the default configuration does — give
|
||||
the heartbeat a route of its own, because the deadline is only as tight as the interval feeding it:
|
||||
|
||||
```yaml
|
||||
route:
|
||||
receiver: terdut
|
||||
repeat_interval: 4h
|
||||
routes:
|
||||
- matchers: [ 'alertname = "Watchdog"' ]
|
||||
receiver: terdut
|
||||
group_wait: 0s
|
||||
group_interval: 1m
|
||||
repeat_interval: 1m
|
||||
```
|
||||
|
||||
That delivers a heartbeat every **2 minutes**, not every minute. Alertmanager only reconsiders a
|
||||
group every `group_interval`, and at exactly one elapsed interval `repeat_interval` has not *quite*
|
||||
passed, so the send slips to the next tick — equal values give 2×. Two minutes against the 15 minute
|
||||
default is seven heartbeats per window, which is the point; use `group_interval: 30s` if you want
|
||||
the numbers to mean what they say.
|
||||
|
||||
kube-prometheus-stack users get the `Watchdog` alert (`expr: vector(1)`) for free; it just needs
|
||||
routing to terdut rather than to `null`.
|
||||
+436
@@ -0,0 +1,436 @@
|
||||
# API reference
|
||||
|
||||
_The REST API._ Back to the [README](../README.md) and the [documentation index](./README.md).
|
||||
|
||||
## Authentication
|
||||
|
||||
All endpoints except `/api/bootstrap`, `/api/integrations/{key}/alertmanager`,
|
||||
`/api/notify/ack/{token}`, `/api/login`, `/api/logout`, `/api/auth/config`,
|
||||
`/api/version`, `/api/oidc/login`, `/api/oidc/callback`, `/api/oidc/device` and
|
||||
`/api/oidc/device/token` require either an API key:
|
||||
|
||||
```
|
||||
Authorization: Bearer <api-key>
|
||||
```
|
||||
|
||||
or the web UI's session cookie. A request that carries an `Authorization` header
|
||||
is judged on that header alone.
|
||||
|
||||
Two kinds of user exist. An **administrator** manages accounts: creating and
|
||||
deleting users, setting anybody's password, minting keys for anybody, and
|
||||
granting the flag itself. Everybody else works incidents — acknowledging,
|
||||
assigning, snoozing, resolving, noting — and manages their own account and
|
||||
nobody else's. An API key carries exactly the rights of the user it belongs to.
|
||||
|
||||
A third principal, the **service account**, exists for automation (a
|
||||
Kubernetes operator, most likely) that needs to manage teams, escalation
|
||||
policies, dead man's switches and integrations without impersonating a human.
|
||||
It is not a user — it never signs in, never appears in a team's member list,
|
||||
and never holds the administrator flag — and its key is prefixed `tdsa_` so it
|
||||
reads as one at a glance in a log line. See [Service accounts](#service-accounts).
|
||||
|
||||
**Getting an account.** The first one comes from `/api/bootstrap`. After that
|
||||
it depends on `signup_mode`, an administrator setting:
|
||||
|
||||
- `invite_only` (the default) — a team owner mints a link with
|
||||
`POST /api/teams/{teamID}/invites`, and the person who opens it picks a
|
||||
username and password and lands in that team with the role the link carries.
|
||||
Links are single-use unless told otherwise, expire after seven days, and can
|
||||
be revoked before that.
|
||||
- `open` — anybody who can reach the server can create an account, and must
|
||||
name a team, which they then own.
|
||||
|
||||
Invites are **links, not email**: this server has no SMTP, and adding it to send
|
||||
one message would be a subsystem to run, secure and monitor. Send the link
|
||||
however you already talk to the person.
|
||||
|
||||
A domain-restricted third mode was considered and dropped: with no email there
|
||||
is nothing to verify an address against, so it would only check the domain of a
|
||||
string somebody typed.
|
||||
|
||||
The first user, from `/api/bootstrap`, is an administrator. Users created
|
||||
afterwards are not, until an administrator says so. An install always keeps at
|
||||
least one: the last administrator can be neither deleted nor demoted, and
|
||||
nobody can delete or demote themselves.
|
||||
|
||||
Endpoints that require the flag answer `403` with
|
||||
`{"error":"administrator access required"}`.
|
||||
|
||||
**Teams** are the unit of tenancy, and are a separate axis from the administrator
|
||||
flag. A team owns its incidents, alerts, schedule and integrations, and a user
|
||||
sees exactly the teams they belong to. Within a team an **owner** configures it
|
||||
(schedule, integrations, membership) and a **member** works its incidents.
|
||||
|
||||
An administrator crosses that line in one direction only. They **configure any
|
||||
team** without being in it — every owner-only endpoint accepts the flag, because
|
||||
otherwise a team whose last owner left could never be repaired. They do **not
|
||||
read any team**: the queue, the alerts and the incidents are filtered by real
|
||||
membership, so an administrator sees a team's work only by joining it, which is
|
||||
a membership change and shows up as one. Administration is about accounts and
|
||||
the shape of a team, not about reading other people's incidents.
|
||||
|
||||
Anything belonging to a team you are not in answers `404`, not `403`: whether an
|
||||
incident exists is itself something only its team should learn.
|
||||
|
||||
**Operator mode** (`TERDUT_OPERATOR_MODE`, see [Configuration](./configuration.md#configuration))
|
||||
declares this install gitops-managed. When it is on, a session or a user's own
|
||||
API key gets `403 {"error": "...", "reason": "operator_managed"}` on every
|
||||
write this page marks **owner**-gated under Teams below (creating, renaming
|
||||
or deleting a team; its OIDC group binding; its escalation ladder; its dead
|
||||
man's switches; its integrations) — a service account's writes are unaffected.
|
||||
Team membership and invites are deliberately excluded: they are never
|
||||
gitops-managed, in operator mode or out of it. `GET /api/auth/config` reports
|
||||
`operator_mode` so a client can grey those sections out before a write is ever
|
||||
attempted.
|
||||
|
||||
| Method | Path | Description |
|
||||
|---|---|---|
|
||||
| `GET` | `/api/auth/config` | How to sign in: `{"password_login", "oidc": {"enabled","name"}, "device_login", "operator_mode"}`. No session needed |
|
||||
| `GET` | `/api/version` | `{"version"}` — this build's version string. No session needed, the same as `/healthz` |
|
||||
| `POST` | `/api/login` | `{"username","password"}` → sets the session cookie, returns `{user, has_password}`. `429` after too many failures; `403` when `TERDUT_PASSWORD_LOGIN=false` |
|
||||
| `GET` | `/api/oidc/login` | Starts a single sign-on sign-in: redirects the browser to the provider. `?next=/path` is where to land afterwards; only a path on this server is honoured. Only exists when SSO is configured |
|
||||
| `POST` | `/api/oidc/device` | Starts a device login: returns `{device_code, user_code, verification_url, interval, expires_in}`. Only exists when SSO is configured |
|
||||
| `POST` | `/api/oidc/device/token` | `{"device_code"}` → `202 {"status":"pending"}`, then `200` with the session cookie once approved (once only). `410` with `{"error":"expired"}` or `{"error":"denied"}`; `429 {"error":"slow_down"}` if polled faster than `interval` |
|
||||
| `POST` | `/api/oidc/device/approve` | **session** — `{"user_code"}`. Approves a pending device login as the caller. `403` for an API key; `404` for an unknown, expired or already decided code |
|
||||
| `POST` | `/api/oidc/device/deny` | **session** — `{"user_code"}`. Refuses it |
|
||||
| `GET` | `/api/oidc/callback` | Where the provider sends the browser back. Sets the session cookie and redirects to `/`, or to `/?sso_error=<code>` — one of `denied`, `expired`, `failed`, `unavailable`, `not_allowed`, `no_email`, `email_conflict`, `disabled`, `not_bootstrapped` (no user exists on this install yet — sign in again once something has called `/api/bootstrap`) |
|
||||
| `POST` | `/api/logout` | Ends the session and clears the cookie |
|
||||
| `GET` | `/api/me` | The caller: `{user, has_password}` |
|
||||
|
||||
## Users
|
||||
|
||||
| Method | Path | Description |
|
||||
|---|---|---|
|
||||
**admin** marks an endpoint that requires the administrator flag; **self or
|
||||
admin** marks one you may use on your own account and an administrator may use
|
||||
on anybody's.
|
||||
|
||||
| Method | Path | Who | Description |
|
||||
|---|---|---|---|
|
||||
| `GET` | `/api/signup` | — | Whether sign-up is open, and whether `?invite=` is usable. No session needed: the caller has no account yet |
|
||||
| `POST` | `/api/signup` | — | Create an account `{"username","email","password","invite"?,"team_name"?}` and sign in. `403` without a usable invite when the mode is invite-only |
|
||||
| `POST` | `/api/bootstrap` | — | Create first user + API key `{"username","email","password"?}` (only works on empty DB). The user is an administrator |
|
||||
| `GET` | `/api/users` | any | List users. Open to everybody: the queue's assignment control and the schedule both have to name people |
|
||||
| `GET` | `/api/users/{id}/teams` | self or admin | The teams that user is in, each with their role. `/api/teams` is always about the caller; this one answers it about somebody else, for the admin page's per-user view. `404` for a user who does not exist, so "no teams" and "no such person" are distinguishable |
|
||||
| `POST` | `/api/users` | **admin** | Create user `{"username","email"}`. Not an administrator |
|
||||
| `DELETE` | `/api/users/{id}` | **admin** | Delete user (cascades to keys). `409` for yourself or the last administrator |
|
||||
| `PUT` | `/api/users/{id}/admin` | **admin** | Grant or revoke the administrator flag `{"is_admin"}`. `409` for yourself, the last administrator, or an administrator granted by single sign-on |
|
||||
| `PUT` | `/api/users/{id}/disabled` | **admin** | Take an account out of use, or put it back `{"disabled"}`. `409` for yourself or the last administrator |
|
||||
| `PUT` | `/api/users/{id}/notify` | self or admin | Set push notification target `{"ntfy_topic"}` — empty string clears it |
|
||||
| `PUT` | `/api/users/{id}/password` | self or admin | Set web UI password `{"password","current_password"}`. `current_password` is required only when changing your own existing password. Ends the user's other sessions |
|
||||
| `POST` | `/api/users/{id}/api-keys` | self or admin | Issue API key `{"name"}` — key shown once |
|
||||
| `DELETE` | `/api/users/{id}/api-keys/{keyID}` | self or admin | Revoke API key |
|
||||
|
||||
## Administration
|
||||
|
||||
| Method | Path | Who | Description |
|
||||
|---|---|---|---|
|
||||
| `GET` | `/api/admin/teams` | **admin** | Every team on the server, with its member and open-incident counts. `/api/teams` answers "what am I in"; this answers "what is there" |
|
||||
| `GET` | `/api/admin/teams/{teamID}` | **admin** | One team and who is in it: `{"team", "members"}`. `404` for a team that does not exist. `GET /api/teams/{teamID}/members` is **member**-only and still `404`s an administrator from outside the team — reading a team's shape and reading its work are different questions, so they are different endpoints |
|
||||
| `GET` | `/api/admin/settings` | **admin** | The editable settings with their bounds, plus the environment-configured ones, read-only. Never credentials |
|
||||
| `PUT` | `/api/admin/settings` | **admin** | Change one or more `{"key": seconds}`, or `{"signup_mode": "open"\|"invite_only"}`. `400` for an unknown key or a value outside its bounds |
|
||||
|
||||
## Service accounts
|
||||
|
||||
A service account is a scoped, non-human credential for automation — not a
|
||||
`users` row, so it never signs in, is never a team member, and never carries
|
||||
the administrator flag. Two scopes:
|
||||
|
||||
- **instance** — the same reach system administration has over teams: create
|
||||
one, and mint a **team**-scoped account against any of them. There is no
|
||||
cap on how many instance-scoped accounts exist, but ordinarily there is one,
|
||||
belonging to whatever is provisioning this install end to end.
|
||||
- **team** — owner-equivalent for that one team, and nothing else: every
|
||||
**owner**-gated endpoint under [Teams](#teams), membership and invites
|
||||
included. Nothing narrower is enforced server-side; what actually keeps
|
||||
membership out of automation's hands is that no operator built against this
|
||||
scope should ever call those two endpoints — see
|
||||
[operator mode](#authentication) and [`SERVICE-ACCOUNTS.md`](../SERVICE-ACCOUNTS.md)'s note on this.
|
||||
|
||||
A key is shown once, at creation or rotation, and only its hash is stored —
|
||||
the same handling as a user's API key. Losing it means minting a new one;
|
||||
there is no way to recover a raw key from the server.
|
||||
|
||||
| Method | Path | Who | Description |
|
||||
|---|---|---|---|
|
||||
| `GET` | `/api/service-accounts` | **admin** | Every service account. Pass `?name=` instead to look one up by its exact name — open to **any** authenticated caller (human or service account), since it returns no key material and is how an account finds its own id |
|
||||
| `POST` | `/api/service-accounts` | owner\* | Create one and mint its first key `{"name","scope","team_id"?}` (`team_id` required for `scope:"team"`, absent for `scope:"instance"`). Returns `{"service_account", "key"}` — `key.key` shown once |
|
||||
| `POST` | `/api/service-accounts/{id}/keys` | owner\* | Mint an additional key `{"name"}` — rotation without recreating the account. Shown once |
|
||||
| `DELETE` | `/api/service-accounts/{id}/keys/{keyID}` | owner\* | Revoke one key |
|
||||
|
||||
\* For an **instance**-scoped account: a system administrator only. For a
|
||||
**team**-scoped account: a system administrator, that team's own human owner,
|
||||
an instance-scoped service account (minting a narrower credential for a team
|
||||
it just created), or — for the two key endpoints only — the account rotating
|
||||
or revoking its own key, which is not a privilege escalation, the same
|
||||
reasoning a user's own API keys rest on.
|
||||
|
||||
## Alert ingestion
|
||||
|
||||
Alerts arrive on a team's integration key. The key is both the credential and the
|
||||
routing: it says that the sender may post, and which team the alerts belong to.
|
||||
Create one with `POST /api/teams/{teamID}/integrations`, which returns the key
|
||||
and the full URL once and stores only a SHA-256 hash.
|
||||
|
||||
| Method | Path | Description |
|
||||
|---|---|---|
|
||||
| `POST` | `/api/integrations/{key}/alertmanager` | Alertmanager v4 webhook receiver for the key's team. `401` for an unknown key |
|
||||
|
||||
This is the only way in. The pre-teams `POST /api/alertmanager/webhook` took no
|
||||
credential at all — anything able to reach the port could open an incident —
|
||||
and was removed in v0.13.0 once senders had moved onto keys.
|
||||
|
||||
## Teams
|
||||
|
||||
**owner** below means an owner of that team, a system administrator (who
|
||||
passes every one of these without being a member), or that team's own
|
||||
team-scoped [service account](#service-accounts) — including membership and
|
||||
invites, technically, though no automation this scope was designed for
|
||||
(a Kubernetes operator's CRDs, see [`SERVICE-ACCOUNTS.md`](../SERVICE-ACCOUNTS.md)) ever models team
|
||||
membership or would call those two. See [Authentication](#authentication).
|
||||
**member** means membership and nothing else: an administrator who is not in
|
||||
the team gets the same `404` as anybody else.
|
||||
|
||||
| Method | Path | Who | Description |
|
||||
|---|---|---|---|
|
||||
| `GET` | `/api/teams` | any | The caller's own teams, each with their role |
|
||||
| `POST` | `/api/teams` | any | Create a team `{"name"}`; a human creator becomes its first owner. An instance-scoped [service account](#service-accounts) may also create one, and it gets no owner at all — expected for a team an operator is about to hand a team-scoped credential to, not an orphaned team a human made |
|
||||
| `PUT` | `/api/teams/{teamID}` | **owner** | Rename it `{"name"}`. `409` if the name is taken |
|
||||
| `DELETE` | `/api/teams/{teamID}` | **owner** | Delete a team and everything under it. `409` while it has open incidents |
|
||||
| `GET` | `/api/teams/{teamID}/members` | member | Who is in the team, with `status` (`oncall` if the rota has them today, `unpageable` when a page to them would go nowhere — even if they are on call — else `reachable`), `on_call`, `next_shift` (first rota day after today), `pageable` and `problem` (`has no ntfy topic` / `account is disabled`; never the topic itself) and `last_active_at` (their newest session or API-key use). Every member sees the same list |
|
||||
| `POST` | `/api/teams/{teamID}/members` | **owner** | Add a member, or change their role `{"user_id","role"}`. `409` when it would demote the last owner, or the membership is managed by single sign-on |
|
||||
| `DELETE` | `/api/teams/{teamID}/members/{userID}` | **owner** | Remove a member. `409` for the last owner, or a membership managed by single sign-on |
|
||||
| `GET` | `/api/teams/{teamID}/oidc-groups` | member | Which groups control this team's membership: `{"member_group","owner_group"}`. An empty string means no group grants that role here |
|
||||
| `PUT` | `/api/teams/{teamID}/oidc-groups` | **owner** | Set them. An empty string clears a binding |
|
||||
| `GET` | `/api/teams/{teamID}/integrations` | member | List integrations. Never returns keys. Each carries `status` (`active` if its key posted within 24h, `quiet` if it has but not lately, `never`), `last_used_at` (last webhook, usable or not), `last_alert_at` (when an alert last arrived on it) and `alerts_24h` (distinct alerts it refreshed in the last day). Alerts delivered before the source was recorded (migration 010) have none, so the last two fill in as Alertmanager re-sends them |
|
||||
| `PATCH` | `/api/teams/{teamID}/integrations/{integrationID}` | **owner** | Rename `{"name"}`. The key does not change |
|
||||
| `POST` | `/api/teams/{teamID}/integrations` | **owner** | Mint an integration `{"name","kind"}` — key and URL shown once |
|
||||
| `DELETE` | `/api/teams/{teamID}/integrations/{integrationID}` | **owner** | Revoke an integration. Alerts it delivered stay, unattributed |
|
||||
| `GET` | `/api/teams/{teamID}/invites` | **owner** | The team's invite links, with their uses and expiry. Never the tokens |
|
||||
| `POST` | `/api/teams/{teamID}/invites` | **owner** | Mint one `{"role","max_uses"}` — the full URL is returned once |
|
||||
| `DELETE` | `/api/teams/{teamID}/invites/{inviteID}` | **owner** | Revoke a link before it expires |
|
||||
| `GET` | `/api/teams/{teamID}/escalation` | member | The team's [escalation ladder](./escalation.md#escalation) `{repeat_count, fallback_topic, levels[], last_escalated_at?, last_escalated_incident_id?}`. Empty levels means the team has none. Each level also carries `status` (`ready`, `escalating` when an unanswered incident has climbed to it, `unreachable` when nobody on it could be woken), `waiting` (ids of the open incidents on it) and, per target, `username` (who it means today — the person on call, for a rota target), `reachable` and `problem`. The extra fields are output only; `PUT` takes the plain shape |
|
||||
| `PUT` | `/api/teams/{teamID}/escalation` | **owner** | Replace it wholesale. `400` for a level with no targets or no timeout — a rung that pages nobody is a silence with a number on it |
|
||||
| `GET` | `/api/teams/{teamID}/deadman/switches` | member | The team's [dead man's switches](./dead-mans-switch.md), each `{id, name, matcher, timeout_seconds, severity, status, last_heartbeat_at, last_triggered_at, open_incident_id, sources[]}`. `status` is `healthy`, `dead` or `dormant`; `sources` has one entry per heartbeat fingerprint. Empty when the team watches nothing |
|
||||
| `POST` | `/api/teams/{teamID}/deadman/switches` | **owner** | Add one: `{name?, matcher, timeout_seconds, severity?}`. `400` when the matcher names no `alertname` or holds several, or the timeout is not positive — a switch that silently watches nothing is the failure this feature exists to prevent |
|
||||
| `PUT` | `/api/teams/{teamID}/deadman/switches/{switchID}` | **owner** | Replace one in place, same body and validation as create. Its id is unchanged — for an automated caller reconciling a spec change, unlike delete-and-recreate |
|
||||
| `DELETE` | `/api/teams/{teamID}/deadman/switches/{switchID}` | **owner** | Stop watching. An incident it opened stays open. `404` for a switch of another team |
|
||||
|
||||
## Notifications
|
||||
|
||||
| Method | Path | Description |
|
||||
|---|---|---|
|
||||
| `POST` | `/api/notify/ack/{token}` | Acknowledge an incident from a push notification's Acknowledge button. No auth: the token in the path is the credential — one incident, one action, 24 hours, idempotent. Must stay publicly reachable |
|
||||
|
||||
## Incidents
|
||||
|
||||
| Method | Path | Description |
|
||||
|---|---|---|
|
||||
| `GET` | `/api/incidents` | List incidents. Filters: `?status=triggered\|acknowledged\|resolved`, `?severity=`, `?assigned_to=<user id>`, `?archived=true`, `?snoozed=true`, `?from=YYYY-MM-DD`, `?to=YYYY-MM-DD`, `?sort=severity`, `?cluster=<value of the cluster group label>`, `?limit=` (default 50, max 500) |
|
||||
| `GET` | `/api/incidents/clusters` | The distinct `cluster` values on the caller's incidents from the last 90 days, sorted (`?team_id=` narrows it). An empty array when nothing carries the label |
|
||||
| `GET` | `/api/incidents/{id}` | Get single incident, with its alerts inline |
|
||||
| `GET` | `/api/incidents/{id}/alerts` | Alerts under this incident |
|
||||
| `GET` | `/api/incidents/{id}/timeline` | Full event history, chronological |
|
||||
| `POST` | `/api/incidents/{id}/acknowledge` | Acknowledge (stamps authed user + time) |
|
||||
| `DELETE` | `/api/incidents/{id}/acknowledge` | Clear acknowledgement, back to `triggered` |
|
||||
| `POST` | `/api/incidents/{id}/resolve` | Close by hand — **terminal**, see above |
|
||||
| `POST` | `/api/incidents/{id}/assign` | Reassign `{"user_id"}` |
|
||||
| `POST` | `/api/incidents/{id}/snooze` | Hide until `{"until": RFC3339}` or `{"duration": "2h"}` |
|
||||
| `DELETE` | `/api/incidents/{id}/snooze` | Un-snooze |
|
||||
| `POST` | `/api/incidents/{id}/archive` | Archive (hides from the default list) |
|
||||
| `DELETE` | `/api/incidents/{id}/archive` | Un-archive |
|
||||
| `POST` | `/api/incidents/{id}/notes` | Add a note `{"content"}` |
|
||||
| `DELETE` | `/api/incidents/{id}/notes/{eventID}` | Delete own note |
|
||||
|
||||
With no `?status=` filter, `GET /api/incidents` returns **open** incidents only —
|
||||
the queue an on-call person wants. Currently snoozed and archived incidents are
|
||||
excluded unless asked for. Actions that only make sense on an open incident
|
||||
return `409` once it is resolved.
|
||||
|
||||
Notes are ordinary timeline events of type `note`; only they are deletable, and
|
||||
only by their author. The rest of the timeline is a record of what happened.
|
||||
|
||||
### The incident object
|
||||
|
||||
| Field | Type | Notes |
|
||||
|---|---|---|
|
||||
| `id` | integer | Server-assigned |
|
||||
| `group_key` | string | Alertmanager's `groupKey` — opaque, treat as an identifier |
|
||||
| `title` | string | Rendered from `groupLabels` |
|
||||
| `group_labels` | object | String→string, as sent by Alertmanager |
|
||||
| `status` | string | `"triggered"`, `"acknowledged"` or `"resolved"` |
|
||||
| `severity` | string | *optional* — high-water mark across the incident's alerts; never lowered |
|
||||
| `triggered_at` | timestamp | When the incident opened |
|
||||
| `acknowledged_by_id` / `acknowledged_by` / `acknowledged_at` | | *optional* — user id, username, time |
|
||||
| `assigned_to_id` / `assigned_to` | | *optional* — user id, username |
|
||||
| `snoozed_until` | timestamp | *optional* — a value in the past reads as not snoozed |
|
||||
| `resolved_at` | timestamp | *optional* |
|
||||
| `resolution_source` | string | *optional* — `"alerts"`, `"manual"` or `"recovered"` |
|
||||
| `archived_at` | timestamp | *optional* |
|
||||
| `alerts` | array | Only on `GET /api/incidents/{id}` |
|
||||
|
||||
Treat `resolution_source` as an open set, as with the alert field of the same
|
||||
name: degrade unknown values to "resolved, reason unknown".
|
||||
|
||||
### The timeline event object
|
||||
|
||||
| Field | Type | Notes |
|
||||
|---|---|---|
|
||||
| `id` | integer | |
|
||||
| `incident_id` | integer | |
|
||||
| `type` | string | See below — treat as an open set |
|
||||
| `user_id` / `username` | | *optional* — absent when the server acted rather than a person |
|
||||
| `alert_id` | integer | *optional* — the alert an `alert_added` / `alert_resolved` event refers to |
|
||||
| `detail` | string | *optional* — the note body, the snooze deadline, etc. |
|
||||
| `created_at` | timestamp | |
|
||||
|
||||
Types written today: `triggered`, `alert_added`, `alert_resolved`,
|
||||
`acknowledged`, `unacknowledged`, `assigned`, `archived`, `unarchived`, `snoozed`,
|
||||
`unsnoozed`, `resolved`, `note`, `notified`, `notify_failed`, `deadman_silent`. On an
|
||||
`assigned` event `user_id` is the **assignee**, not the actor; the actor is in
|
||||
`actor_user_id`/`actor_username` or `actor_service_account_id`/`actor_service_account_name`
|
||||
(absent on assignments made before they were recorded). New types may be added; render
|
||||
unknown ones generically rather than dropping them.
|
||||
|
||||
On `notified` and `notify_failed`, `detail` carries the notification kind
|
||||
(`triggered` | `reminder` | `resolved`), and on a failure the reason after it.
|
||||
`user_id` is who was paged — absent means the page went to the shared fallback
|
||||
topic and so belongs to nobody. The topic itself is never written to the
|
||||
timeline: it is a shared secret with the ntfy server, and every API key can read
|
||||
this.
|
||||
|
||||
## Alerts
|
||||
|
||||
Alerts are read-only. Everything a person does happens on the incident.
|
||||
|
||||
| Method | Path | Description |
|
||||
|---|---|---|
|
||||
| `GET` | `/api/alerts` | List alerts. Filters: `?status=firing\|resolved`, `?name=`, `?incident_id=`, `?archived=true`, `?from=YYYY-MM-DD`, `?to=YYYY-MM-DD`, `?limit=` (default 50, max 500) |
|
||||
| `GET` | `/api/alerts/{id}` | Get single alert |
|
||||
|
||||
Archived alerts are hidden from `GET /api/alerts` unless `?archived=true` is
|
||||
passed; alert archiving is automatic housekeeping by the sweeper, not a user
|
||||
action. Resolved alerts carry `resolution_source`: `"alertmanager"` for a real
|
||||
resolved webhook, `"expiry"` when the sweeper inferred it (see
|
||||
[Stale alert expiry](./incidents.md#stale-alert-expiry)), `"deadman"` for a heartbeat declared
|
||||
dead (see [Dead man's switch](./dead-mans-switch.md)).
|
||||
|
||||
### The alert object
|
||||
|
||||
Returned by `GET /api/alerts` (as an array) and `GET /api/alerts/{id}`.
|
||||
Timestamps are RFC 3339 in UTC. Fields marked *optional* are omitted entirely
|
||||
when unset, so clients must treat them as nullable.
|
||||
|
||||
| Field | Type | Notes |
|
||||
|---|---|---|
|
||||
| `id` | integer | Server-assigned; stable for the life of the row |
|
||||
| `fingerprint` | string | Alertmanager's fingerprint — the upsert key |
|
||||
| `name` | string | From the `alertname` label |
|
||||
| `status` | string | `"firing"` or `"resolved"` |
|
||||
| `labels` | object | String→string, as sent by Alertmanager |
|
||||
| `annotations` | object | String→string, as sent by Alertmanager |
|
||||
| `starts_at` | timestamp | When the alert instance began, **per Prometheus** |
|
||||
| `ends_at` | timestamp | *optional* — absent while no end is known |
|
||||
| `generator_url` | string | Link back to the originating Prometheus |
|
||||
| `received_at` | timestamp | When the server last accepted a webhook for this alert — see below |
|
||||
| `incident_id` | integer | *optional* — the most recent incident this alert belongs to |
|
||||
| `resolution_source` | string | *optional* — `"alertmanager"`, `"expiry"` or `"deadman"` |
|
||||
| `archived_at` | timestamp | *optional* — set while archived |
|
||||
|
||||
#### `received_at` is a liveness heartbeat
|
||||
|
||||
`starts_at` comes from Prometheus and **never changes** for the lifetime of an
|
||||
alert instance. It says when the problem began, not whether it is still
|
||||
happening — an alert that started twelve days ago looks identical whether
|
||||
Alertmanager refreshed it a minute ago or went silent a week ago.
|
||||
|
||||
`received_at` is the field that answers "is this still live". It is set to the
|
||||
server's clock on **every accepted webhook** for that fingerprint, including the
|
||||
unchanged firing notifications Alertmanager re-sends every `repeat_interval`.
|
||||
Clients may rely on this:
|
||||
|
||||
- **A firing alert whose `received_at` is advancing is still being refreshed.**
|
||||
Stale-dating it against `repeat_interval` is a valid liveness check, and it is
|
||||
what the built-in sweeper does (see
|
||||
[Stale alert expiry](./incidents.md#stale-alert-expiry)).
|
||||
- **`received_at` tracks accepted payloads, not delivery attempts.** A retry
|
||||
that describes an older instance than the stored one is discarded, and a
|
||||
discarded payload does not move `received_at`.
|
||||
- **It stops advancing once the alert resolves,** because Alertmanager stops
|
||||
re-sending. On an alert resolved by the sweeper
|
||||
(`"resolution_source": "expiry"`) it therefore marks the last time
|
||||
Alertmanager was actually heard from, which is earlier than `ends_at`.
|
||||
|
||||
`GET /api/alerts` is ordered by `received_at` descending — most recently
|
||||
refreshed first — and the `?from=` / `?to=` filters on both the alert and stats
|
||||
endpoints select on `received_at`, not `starts_at`.
|
||||
|
||||
#### `resolution_source` says how much to trust `ends_at`
|
||||
|
||||
An alert can leave the firing state two ways, and `resolution_source` records
|
||||
which happened. Clients may rely on this:
|
||||
|
||||
- **Absent while firing.** It is set only on resolve, and a re-fire under the
|
||||
same fingerprint clears it again, so its presence always agrees with
|
||||
`"status": "resolved"`.
|
||||
- **`"alertmanager"` — a real resolved webhook arrived.** `ends_at` is the end
|
||||
time Alertmanager reported. It is an observed value and can be displayed as
|
||||
fact.
|
||||
- **`"expiry"` — the sweeper inferred the resolve** because Alertmanager stopped
|
||||
refreshing the alert (see [Stale alert expiry](./incidents.md#stale-alert-expiry)). Nothing
|
||||
ever reported an end, so **`ends_at` is approximate**: it is either the stale
|
||||
`endsAt` watermark from the last notification, or — when that notification
|
||||
carried none — the time the sweep ran, which lags the last real contact by up
|
||||
to `TERDUT_STALE_AFTER` plus a sweep interval. Treat it as "no later than",
|
||||
not as when the problem stopped.
|
||||
|
||||
On these alerts `received_at` is the more truthful signal: it marks the last
|
||||
time Alertmanager was actually heard from. Surfacing the distinction is
|
||||
worthwhile, since `"expiry"` can also mean the alert is still firing and the
|
||||
notification path broke.
|
||||
|
||||
- **`"deadman"` — a heartbeat was declared dead** (see
|
||||
[Dead man's switch](./dead-mans-switch.md)). Like `"expiry"`, an inference from
|
||||
silence rather than an observed end, so `ends_at` is approximate — but a much
|
||||
tighter one, bounded by the switch's timeout. It is also the one resolution
|
||||
a re-fire under the same `starts_at` can undo, since the switch coming back is
|
||||
exactly the evidence that the inference was wrong.
|
||||
|
||||
Treat the value as an open set and tolerate ones you do not recognise — new
|
||||
sources may be added, and unknown values should degrade to "resolved, reason
|
||||
unknown" rather than being rejected.
|
||||
|
||||
## On-call schedule
|
||||
|
||||
| Method | Path | Description |
|
||||
|---|---|---|
|
||||
Each team keeps its own rota, so two teams can have two different people on call
|
||||
on the same day. The person taking a shift has to be in the team — paging
|
||||
somebody who cannot open the incident is worse than paging nobody.
|
||||
|
||||
| Method | Path | Who | Description |
|
||||
|---|---|---|---|
|
||||
| `POST` | `/api/teams/{teamID}/schedule` | **owner** | Assign user to dates `{"user_id", "dates":["YYYY-MM-DD",...], "replace"}` — all-or-nothing |
|
||||
| `GET` | `/api/teams/{teamID}/schedule` | member | List entries. Filters: `?from=YYYY-MM-DD`, `?to=YYYY-MM-DD` |
|
||||
| `DELETE` | `/api/teams/{teamID}/schedule/{id}` | **owner** | Remove schedule entry |
|
||||
| `GET` | `/api/schedule/current` | any | Who is on call today (UTC) in **every** team the caller is in — one entry per team, `[]` when nobody anywhere |
|
||||
|
||||
## Statistics
|
||||
|
||||
Every figure counts the caller's own teams only: a report that counted other
|
||||
teams' incidents would leak their volume, and their alert names through the
|
||||
top-alerts list, and would not be a number about the reader's work anyway.
|
||||
|
||||
All stat endpoints accept optional `?from=YYYY-MM-DD` and `?to=YYYY-MM-DD`, and exclude archived rows to match the default list views. Alert stats filter on `received_at`; incident stats filter on `triggered_at`.
|
||||
|
||||
| Method | Path | Description |
|
||||
|---|---|---|
|
||||
| `GET` | `/api/stats/incidents` | `{total, triggered, acknowledged, resolved, mtta_seconds, mttr_seconds}` |
|
||||
| `GET` | `/api/stats/alerts` | `{total, firing, resolved}` counts |
|
||||
| `GET` | `/api/stats/alerts/top` | Most frequent alert names. `?limit=` (default 10, max 100) |
|
||||
| `GET` | `/api/stats/alerts/by-hour` | Count per hour-of-day (UTC), all 24 slots returned |
|
||||
| `GET` | `/api/stats/alerts/by-day` | Count per day-of-week, all 7 slots with names returned |
|
||||
|
||||
`mtta_seconds` (time to acknowledge) and `mttr_seconds` (time to resolve) are
|
||||
averages over incidents that have actually been acknowledged or resolved, and are
|
||||
**null** until there are any — null means "no data", not zero.
|
||||
@@ -0,0 +1,49 @@
|
||||
# Configuration
|
||||
|
||||
_Environment variables and settings._ Back to the [README](../README.md) and the [documentation index](./README.md).
|
||||
|
||||
Two kinds of setting, split by who changes them and how often.
|
||||
|
||||
**Where the server is plugged in** stays in the environment: the listen address,
|
||||
the database DSN, the ntfy URL and token, the public URL. They are needed before
|
||||
the database is open, and two of them are credentials.
|
||||
|
||||
**How the server behaves** lives in the database and is edited by an
|
||||
administrator in the web UI or through `PUT /api/admin/settings`, taking effect
|
||||
on the next sweep rather than at the next restart. The variables below marked
|
||||
**seed** are the value each of those starts from: written once, on first start,
|
||||
and never overwritten afterwards — a redeploy cannot put a chart's default back
|
||||
over an administrator's edit.
|
||||
|
||||
| Variable | Default | Description |
|
||||
|---|---|---|
|
||||
| `TERDUT_ADDR` | `:8080` | TCP address to listen on |
|
||||
| `TERDUT_DB_DSN` | — | **Required.** Postgres connection string, e.g. `postgres://terdut:secret@localhost:5432/terdut?sslmode=require` |
|
||||
| `TERDUT_ARCHIVE_AFTER` | `168h` (7d) | **seed.** How long a resolved alert or incident stays in the default list before being auto-archived |
|
||||
| `TERDUT_STALE_AFTER` | `6h` | **seed.** How long a firing alert may go without a refreshing webhook before it is treated as resolved — **must exceed your Alertmanager `repeat_interval`** |
|
||||
| `TERDUT_NTFY_URL` | — | ntfy server to publish push notifications to. Empty disables notifications entirely |
|
||||
| `TERDUT_NTFY_TOKEN` | — | Bearer token for an access-controlled ntfy |
|
||||
| `TERDUT_NTFY_FALLBACK_TOPIC` | — | Topic used when nobody is on call |
|
||||
| `TERDUT_PUBLIC_URL` | — | Base URL a phone uses to reach this server: the notification's link into the web UI, its Acknowledge button, and whether the session cookie is `Secure` |
|
||||
| `TERDUT_NOTIFY_REPEAT` | `15m` | **seed.** How long an incident may sit unacknowledged before it is paged again. `0` notifies once and never repeats |
|
||||
| `TERDUT_PASSWORD_LOGIN` | `true` | `false` refuses password login and password sign-up (`403`), leaving single sign-on the only way in. Refused at startup unless SSO is configured |
|
||||
| `TERDUT_TRUSTED_PROXIES` | `1` | How many reverse proxies in front of the server append to `X-Forwarded-For`; the per-address rate limits use the entry that many hops from the right. `0` ignores the header |
|
||||
| `TERDUT_OPERATOR_KEY` | — | At least 32 characters. When set, the instance-scoped service account `terdut-operator` is created if missing and its `seed` key replaced with this value at every start — how terdut-operator authenticates without a bootstrap handshake. An instance-scoped account acts as owner of every team (team configuration) but is not a member of any, so it reads no incidents |
|
||||
| `TERDUT_OPERATOR_MODE` | `false` | Declares this install gitops-managed: a session's or a user's own API key's writes to teams, escalation policies, dead man's switches and integrations are refused (`403 reason:"operator_managed"`); a [service account](./api.md#service-accounts)'s are not. Team membership and the schedule stay editable regardless |
|
||||
| `TERDUT_OIDC_ISSUER` | — | Turns single sign-on on. The provider's issuer URL; discovery is read from `<issuer>/.well-known/openid-configuration`. See [Single sign-on](./single-sign-on.md#single-sign-on-oidc) |
|
||||
| `TERDUT_OIDC_CLIENT_ID` / `TERDUT_OIDC_CLIENT_SECRET` | — | **Required with an issuer.** The confidential client registered at the provider. Keep the secret in a Secret, not in values |
|
||||
| `TERDUT_OIDC_NAME` | `SSO` | What the sign-in button calls the provider |
|
||||
| `TERDUT_OIDC_SCOPES` | `openid profile email` | Scopes requested, comma or space separated. Authentik puts `groups` behind `profile` |
|
||||
| `TERDUT_OIDC_USERNAME_CLAIM` / `_EMAIL_CLAIM` / `_GROUPS_CLAIM` | `preferred_username` / `email` / `groups` | ID token claims read for the username, email and groups |
|
||||
| `TERDUT_OIDC_TRUST_EMAIL` | `false` | Link a first sign-in to an existing local user by email even if the provider does not mark the address verified |
|
||||
| `TERDUT_OIDC_ALLOWED_GROUPS` | — | Comma-separated. Only people in one of these may sign in. Empty admits everybody the provider authenticates |
|
||||
| `TERDUT_OIDC_ADMIN_GROUP` | — | Members are system administrators |
|
||||
| `TERDUT_OIDC_SESSION_MAX_AGE` | `12h` | Hard ceiling on a session made by an SSO sign-in |
|
||||
|
||||
Durations use Go syntax (`30m`, `12h`, `168h`). An unparseable value falls back to the default.
|
||||
|
||||
Note that `TERDUT_STALE_AFTER` and a dead man's switch timeout point in opposite directions. Staleness
|
||||
is a generous grace period around a `repeat_interval` you do not control; a dead man's switch is a
|
||||
deadline you set deliberately, and the heartbeat's route is configured to beat faster than it.
|
||||
|
||||
In the Helm chart the two sweeper durations are set via `sweeper.staleAfter` and `sweeper.archiveAfter`, notifications via the `notify.*` values, single sign-on via `oidc.*` and `passwordLogin`, and operator mode via `operatorMode`.
|
||||
@@ -0,0 +1,81 @@
|
||||
# Dead man's switches
|
||||
|
||||
_Detecting that alerts have stopped arriving._ Back to the [README](../README.md) and the [documentation index](./README.md).
|
||||
|
||||
Everything above assumes alerts arrive. If Prometheus stops evaluating, or
|
||||
Alertmanager cannot reach this server, nothing arrives — and silence looks
|
||||
exactly like everything being fine. A dead man's switch inverts the handling for
|
||||
one designated alert so that silence is the signal:
|
||||
|
||||
- **receiving** it opens no incident, and
|
||||
- the **absence** of it does.
|
||||
|
||||
kube-prometheus-stack already ships the alert for this. `Watchdog` is
|
||||
`expr: vector(1)`, so it fires permanently and is re-sent forever; it is worth
|
||||
nothing unless something downstream notices it stop. It is the usual first switch.
|
||||
|
||||
**Switches belong to a team**, which decides which of its own alerts are
|
||||
heartbeats and how long a silence has to last. Each **switch** is a row of its
|
||||
own — a name, one matcher, a timeout and a severity — so switches in one team
|
||||
can have different deadlines. An owner adds and removes them on **Team →
|
||||
Switches**, which lists each with a status (**healthy**, **dead**, or
|
||||
**dormant** until its first heartbeat), when it was last heard from, and when it
|
||||
last opened an incident; a matcher that several clusters satisfy is broken down
|
||||
per cluster. The API is `POST`/`DELETE /api/teams/{teamID}/deadman/switches`. A
|
||||
missed heartbeat opens an incident in the team whose integration received it.
|
||||
Removing a switch stops the watching; an incident it already opened stays open
|
||||
until somebody resolves it.
|
||||
|
||||
A new team watches nothing until its owner (or terdut-operator, from a
|
||||
`TerdutTeam`) adds a switch: inheriting an install-wide heartbeat would page a
|
||||
new team about a source it has never heard of.
|
||||
|
||||
A matcher is a set of exact label conditions, one of which must be the
|
||||
`alertname`, , one matcher per switch, `,` between the label conditions:
|
||||
|
||||
```
|
||||
alertname=Watchdog,cluster=prod
|
||||
```
|
||||
|
||||
**The unit of monitoring is the fingerprint, not the alert name.** Two clusters
|
||||
sending the same `Watchdog` are two independent switches, so a healthy one can
|
||||
never mask a dead one.
|
||||
|
||||
## The lifecycle
|
||||
|
||||
A switch is **dormant** until its first heartbeat arrives. A configured matcher
|
||||
that has never been heard from opens nothing, so a fresh deploy or a restored
|
||||
database does not page. It also means a matcher that never matches anything is
|
||||
silently inert.
|
||||
|
||||
Once armed, the sweeper declares it **dead** when either the heartbeat has not
|
||||
been refreshed within the switch's `timeout_seconds`, or Alertmanager explicitly
|
||||
resolved it — the sender saying the heartbeat stopped needs no further waiting.
|
||||
That opens an incident at the switch's `severity`, assigned and paged like any
|
||||
other, and marks the heartbeat alert `"resolution_source": "deadman"` so the
|
||||
alert list stops claiming a dead switch is firing.
|
||||
|
||||
It **recovers** when the heartbeat starts arriving again: the incident resolves
|
||||
with `"resolution_source": "recovered"` and the all-clear goes to whoever was
|
||||
paged.
|
||||
|
||||
Resolving the incident by hand sticks, the same way it does for an alert-backed
|
||||
one. While the switch stays silent nothing new opens — so a decommissioned
|
||||
source is a one-time page rather than a nag. The switch **re-arms** on the next
|
||||
heartbeat: come back and die again, and that is a new incident.
|
||||
|
||||
## Two things to know
|
||||
|
||||
A switch's timeout must be **shorter** than the `repeat_interval` of the
|
||||
route carrying the heartbeat, which is the exact opposite of
|
||||
`TERDUT_STALE_AFTER`. Inheriting a default `repeat_interval` of 4h gives you a
|
||||
switch that takes four hours to notice anything, so give the heartbeat
|
||||
[its own route](./alertmanager.md#alertmanager-configuration). Matched alerts are exempt from
|
||||
stale-alert expiry — a heartbeat answers to its own timeout and nothing else.
|
||||
|
||||
A dead man's switch incident has **no member alerts**:
|
||||
`GET /api/incidents/{id}/alerts` returns an empty list. There is no alert
|
||||
describing the problem, because the problem is that no alert arrived. What
|
||||
happened is on the timeline instead, as a `deadman_silent` event carrying the age
|
||||
of the last heartbeat, and the heartbeat's labels are on the incident's
|
||||
`group_labels`.
|
||||
@@ -0,0 +1,74 @@
|
||||
# Deployment
|
||||
|
||||
_Running the server in a container and on Kubernetes with the Helm chart._ Back to the [README](../README.md) and the [documentation index](./README.md).
|
||||
|
||||
## Docker
|
||||
|
||||
```bash
|
||||
docker build -t terdut-server .
|
||||
docker run -p 8080:8080 \
|
||||
-e TERDUT_DB_DSN='postgres://terdut:secret@host.docker.internal:5432/terdut?sslmode=disable' \
|
||||
terdut-server
|
||||
```
|
||||
|
||||
The server creates its own schema on startup and needs a reachable Postgres; it stores nothing on
|
||||
disk, so there is no volume to mount.
|
||||
|
||||
## Kubernetes
|
||||
|
||||
A Helm chart is published from this repository as an OCI artifact, versioned in lockstep
|
||||
with the app — chart `x.y.z` is always app `vx.y.z`:
|
||||
|
||||
```bash
|
||||
helm upgrade --install terdut-server oci://git.ryuvia.com/niklas/terdut-server \
|
||||
--version 0.9.2 \
|
||||
--namespace terdut-server --create-namespace \
|
||||
--set networking.hostname=terdut.example.com
|
||||
```
|
||||
|
||||
The chart expects a [Gateway API](https://gateway-api.sigs.k8s.io/) Gateway named `envoy-main` in
|
||||
the `envoy-gateway-system` namespace to already exist — it renders an `HTTPRoute` against it rather
|
||||
than an `Ingress`. TLS is terminated at the gateway, so the server itself never sees a certificate.
|
||||
|
||||
| Value | Default | Description |
|
||||
|---|---|---|
|
||||
| `networking.hostname` | `terdut.example.com` | Hostname the `HTTPRoute` serves |
|
||||
| `networking.listener` | `""` | Gateway listener (`sectionName`) to bind to. Empty attaches to every matching listener, **including plaintext HTTP** — set it to the HTTPS listener's name to serve TLS only |
|
||||
| `networking.servicePort` | `8080` | Port the route forwards to; keep in sync with `service.port` |
|
||||
| `bootstrap.enabled` | `true` | Runs a post-install hook that creates the first user and stores its API key in the `<release>-admin-key` Secret. Already-bootstrapped servers are left alone |
|
||||
| `database.dsn` | `""` | **Required.** Postgres DSN, with no password in it. The chart provisions no database |
|
||||
| `database.passwordSecret.name` | `""` | Secret supplying `PGPASSWORD`. With the Zalando postgres operator, the Secret it generates for the role |
|
||||
| `database.passwordSecret.key` | `password` | Key within that Secret |
|
||||
|
||||
The API key travels in an `Authorization: Bearer` header, so set `networking.listener` whenever the
|
||||
hostname is reachable outside a trusted network.
|
||||
|
||||
### The database
|
||||
|
||||
The chart provisions no database: it takes a DSN and expects a Postgres that already exists. In this
|
||||
cluster the wrapper chart declares an `acid.zalan.do/v1 postgresql` CR; anywhere else, any reachable
|
||||
Postgres 14+ will do.
|
||||
|
||||
The DSN carries no password. pgx falls back to libpq's environment variables for whatever the DSN
|
||||
leaves out, so the password arrives as `PGPASSWORD` from a Secret and never appears in values, in
|
||||
the rendered manifest or in `kubectl describe pod`. With the postgres operator that Secret is the
|
||||
one it generates for the role, so a rebuild mints a new password with nothing to keep in sync —
|
||||
the same wiring miniflux uses.
|
||||
|
||||
The server migrates its own schema on startup, so a new database only has to exist and be writable.
|
||||
|
||||
### Backups
|
||||
|
||||
Postgres is backed up where it runs, not from here. The database pod carries a
|
||||
[k8up](https://k8up.io/) `k8up.io/backupcommand` annotation that streams a `pg_dump`, the same way
|
||||
gitea and immich do in this cluster.
|
||||
|
||||
## On Kubernetes with the operator
|
||||
|
||||
[terdut-operator](https://git.ryuvia.com/niklas/terdut-operator) runs a server for you from a
|
||||
`TerdutServer` object and manages its teams, escalation ladders, dead man's switches and alert
|
||||
sources as Kubernetes objects. It hands the server a generated key through `TERDUT_OPERATOR_KEY`
|
||||
(see [Configuration](./configuration.md) and [`SERVICE-ACCOUNTS.md`](../SERVICE-ACCOUNTS.md)), and
|
||||
the server then treats configuration as operator-managed (`TERDUT_OPERATOR_MODE`), refusing edits
|
||||
made by hand in the web UI. Use the Helm chart above for a plain install, the operator when you
|
||||
want that configuration in gitops.
|
||||
@@ -0,0 +1,93 @@
|
||||
# Development and releasing
|
||||
|
||||
_Building, testing and releasing the server._ Back to the [README](../README.md) and the [documentation index](./README.md).
|
||||
|
||||
## Upgrading
|
||||
|
||||
The schema is a single baseline (`internal/db/migrations/001_schema.sql`) and no
|
||||
release has shipped yet, so there is no upgrade path from earlier development
|
||||
databases: start from an empty one. Changes after the first release arrive as
|
||||
new numbered migrations.
|
||||
|
||||
## Development
|
||||
|
||||
```bash
|
||||
make test-db # start a local Postgres for the tests (podman or docker)
|
||||
make test # run all tests
|
||||
go build ./... # compile all packages
|
||||
go run ./cmd/terdut # run locally (needs TERDUT_DB_DSN)
|
||||
```
|
||||
|
||||
The tests need a real Postgres, because the server does — there is no in-memory Postgres.
|
||||
`TERDUT_TEST_DSN` says where it is, `make test-db` starts
|
||||
one on port 5433 and prints the DSN, and `make test-db-stop` removes it. Each test gets its
|
||||
own schema on that server, so tests cannot see each other's rows. An unset `TERDUT_TEST_DSN`
|
||||
fails the suite rather than skipping it: a run that quietly tests nothing is worse than one
|
||||
that does not run.
|
||||
|
||||
`make fmt lint test helm-lint` is the gate. It mirrors `.gitea/workflows/ci.yaml` step for
|
||||
step, so a green run here means a green pipeline — with one deliberate exception: `make test`
|
||||
adds `-race`, which CI does not. The sweeper, the notifier goroutine and the dead man's switch
|
||||
sweep all run concurrently against the same database, and a race between them would surface as
|
||||
a flaky incident in production rather than as a red build.
|
||||
|
||||
The web UI lives in `internal/web/static/` as plain HTML, CSS and ES modules,
|
||||
embedded into the binary with `go:embed`. It has no build step and no npm, so
|
||||
editing a file and restarting the server is the whole loop.
|
||||
|
||||
## Releasing
|
||||
|
||||
```
|
||||
push or PR → ci.yaml gofmt, go vet, go test -race
|
||||
govulncheck, gitleaks
|
||||
helm lint + render
|
||||
push tag vX.Y.Z → release.yaml the same gate, then publish:
|
||||
git.ryuvia.com/niklas/terdut-server:vX.Y.Z
|
||||
oci://git.ryuvia.com/niklas/terdut-server X.Y.Z
|
||||
then trivy-scan the pushed image
|
||||
PR to Ryuvia/charts → bump the wrapper chart to X.Y.Z; on merge
|
||||
Flux reconciles and the release rolls out
|
||||
```
|
||||
|
||||
Both artifacts go to the **personal** Gitea namespace rather than `ryuvia`, because Gitea
|
||||
scopes package visibility to the owner with no per-package override — so `ryuvia/*` is private
|
||||
because the org is. Publishing to `niklas` keeps them anonymously pullable, which is why no
|
||||
pull secret is needed in the cluster. Same reasoning, and the same choice, as riksdata and
|
||||
rd-web.
|
||||
|
||||
Saying **"Release"** runs all three rows: the `release` skill commits, pushes, tags, waits for
|
||||
the pipeline, and opens the `Ryuvia/charts` PR, stopping before the merge. See
|
||||
`~/.claude/skills/release/`, or `.release.conf` here for this repo's part of it.
|
||||
|
||||
The chart is published **only** from the tag, by the `chart` job. There used to be a second
|
||||
publisher on every `charts/**` push to main, and the two raced for the same chart version with
|
||||
different answers — chart 0.9.0 went out reading `appVersion: "latest"` that way. One
|
||||
publisher, triggered by the tag (`766f439`). The cost is that a chart-only change has no
|
||||
version of its own and rides the next app tag.
|
||||
|
||||
Both workflows are thin drivers over the Makefile: `ci.yaml` runs `make fmt lint test` and
|
||||
`make helm-lint`, `release.yaml` adds `make binaries`, `make push`, `make helm-package` and
|
||||
`make helm-push`. That is deliberate — it is what makes a green local gate and a green
|
||||
pipeline the same code rather than two descriptions of it, and it is how riksdata and rd-web
|
||||
have always worked.
|
||||
|
||||
`make push` builds and pushes in one step, unlike those two, because the image is
|
||||
`linux/amd64,linux/arm64` and buildx cannot load a multi-platform result into the local image
|
||||
store. `make build` stays single-platform and local-only. Both refuse `VERSION=dev`:
|
||||
publishing is one command, so it is also one command to run by accident. Publishing happens
|
||||
by pushing a tag.
|
||||
|
||||
Two things the release process needs to know about this repo:
|
||||
|
||||
- **The image scan runs after publishing**, like riksdata's and rd-web's: trivy cannot read
|
||||
a locally built image on this runner, so it pulls the pushed one. A red `scan-image` means
|
||||
do not bump the wrapper chart to that version — it cannot unpublish anything. The image is
|
||||
`FROM scratch`, so trivy sees exactly one target, the Go binary and its module graph.
|
||||
- **The wrapper chart's `values.yaml` has two `tag:` lines** — the app image and the python
|
||||
backup sidecar — so `chart-bump` is given `--image` to say which one moves. Once the wrapper
|
||||
chart drops the sidecar and declares a `postgresql` CR instead, there is one `tag:` line
|
||||
again, and `--image` becomes belt and braces.
|
||||
|
||||
The wrapper chart must have **its own `version:` bumped in the same commit**. Flux reconciles
|
||||
with `reconcileStrategy: ChartVersion`, so a chart whose version did not change produces no
|
||||
new artifact and the change is never deployed — with no error anywhere.
|
||||
@@ -0,0 +1,45 @@
|
||||
# Escalation
|
||||
|
||||
_Escalation ladders: who is paged next when nobody acknowledges._ Back to the [README](../README.md) and the [documentation index](./README.md).
|
||||
|
||||
Without a ladder, an unacknowledged incident re-pages the same topic every
|
||||
`notify_repeat` forever. That is a louder version of the same silence: if the
|
||||
person on call is asleep, out of signal, or has left, nothing else happens.
|
||||
|
||||
A team can configure an ordered ladder instead. Each level has a timeout and a
|
||||
set of targets, and a target is either a named person or **whoever the team's
|
||||
rota says is on call today** — the target that keeps working when the rota
|
||||
changes and nobody remembers to edit the policy.
|
||||
|
||||
```
|
||||
level 1 5m oncall the rota gets first refusal
|
||||
level 2 5m user:bob then a named second
|
||||
then repeat_count more rounds
|
||||
then the team's fallback topic, once
|
||||
```
|
||||
|
||||
When a level's timeout passes with the incident still `triggered`, the next
|
||||
level is paged. Off the end of the ladder the whole thing runs again
|
||||
`repeat_count` times, and after that the team's `fallback_topic` is paged once
|
||||
as the end of the line. The incident stays open throughout: running out of
|
||||
people to wake is not the same as somebody answering.
|
||||
|
||||
**Acknowledging or resolving stops it**, which is the point — continuing to wake
|
||||
people after somebody has said "I have this" is how a tool teaches people to
|
||||
mute it. **Snoozing pauses it**: a deliberate "not now" holds the ladder where
|
||||
it is, and it resumes when the snooze runs out.
|
||||
|
||||
Every step is on the incident's timeline with the level and the names it woke,
|
||||
so somebody reading it afterwards can tell why their phone rang at 04:00. A
|
||||
level whose targets are all unreachable — no ntfy topic, a disabled account, an
|
||||
empty rota — is recorded as `nobody reachable` and the ladder moves on rather
|
||||
than stalling on a rung that cannot ring.
|
||||
|
||||
**Reminders and escalation never both run.** A team with a ladder gets
|
||||
escalation; a team without keeps the reminder behaviour exactly as it was. Two
|
||||
pages for one silence is the surest way to get a tool muted.
|
||||
|
||||
The ladder's `fallback_topic` is per team, unlike `TERDUT_NTFY_FALLBACK_TOPIC`,
|
||||
which is the install-wide topic used when an incident opens with nobody on call.
|
||||
They answer different questions: one is "nobody was scheduled", the other is
|
||||
"everybody scheduled has been tried".
|
||||
Binary file not shown.
|
After Width: | Height: | Size: 23 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 55 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 88 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 52 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 53 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 100 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 93 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 94 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 42 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 35 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 44 KiB |
@@ -0,0 +1,113 @@
|
||||
# Alerts and incidents
|
||||
|
||||
_How alerts become incidents and how incidents are worked, notified, escalated and expired._ Back to the [README](../README.md) and the [documentation index](./README.md).
|
||||
|
||||
There are two objects, and the difference between them is the whole design.
|
||||
|
||||
**An alert is Alertmanager's record.** It has two states, `firing` and
|
||||
`resolved`, one row per fingerprint, and no human ever writes to it. The API
|
||||
exposes alerts read-only.
|
||||
|
||||
**An incident is the work item.** It goes `triggered → acknowledged → resolved`,
|
||||
carries an assignee, a snooze, notes and a timeline, and is the only thing people
|
||||
act on. Many alerts belong to one incident.
|
||||
|
||||
## Correlation uses Alertmanager's `groupKey`
|
||||
|
||||
Alertmanager has already grouped alerts according to the `group_by` routing tree
|
||||
you configured, and it sends the resulting `groupKey` and `groupLabels` on every
|
||||
webhook. Incidents adopt that answer rather than re-grouping alerts a second
|
||||
time — if you want different correlation, change `group_by` in
|
||||
`alertmanager.yml` and terdut follows.
|
||||
|
||||
At most one incident is open per `groupKey` at a time. Alerts firing in a group
|
||||
that already has an open incident join it. The incident's `severity` is a
|
||||
high-water mark — the highest `severity` label any of its alerts has carried — so
|
||||
an incident that hit `critical` still reads as critical after the critical alert
|
||||
clears.
|
||||
|
||||
## Several clusters, one team
|
||||
|
||||
A team with one Alertmanager per Kubernetes cluster, each posting to its own
|
||||
source, needs two settings or the clusters run together.
|
||||
|
||||
1. Give every alert a `cluster` label at the source. In Prometheus that is
|
||||
`externalLabels: {cluster: prod-eu}` (kube-prometheus-stack:
|
||||
`prometheus.prometheusSpec.externalLabels`).
|
||||
2. Add `cluster` to `group_by` in `alertmanager.yml`.
|
||||
|
||||
The second one is the one that matters. Incidents are matched on the team and
|
||||
Alertmanager's `groupKey`, and the `groupKey` does not include external labels:
|
||||
without `cluster` in `group_by`, the same alert in two clusters has the same
|
||||
key and joins one incident. With it, each cluster gets its own, `cluster` is in
|
||||
the incident's `group_labels`, and the web UI shows it as a coloured chip on the
|
||||
queue, the incident and the alert list, instead of leaving it in the title.
|
||||
An alert that is not grouped by `cluster` still shows the chip on the alert
|
||||
list, which reads the label from the alert itself.
|
||||
|
||||
The queue has a cluster dropdown once there are two or more values to choose
|
||||
between. It filters on the incident's `cluster` group label
|
||||
(`GET /api/incidents?cluster=...`), so it only sees incidents grouped by it.
|
||||
|
||||
## An incident opens only on a new occurrence
|
||||
|
||||
An incident opens when an alert **transitions into firing**: a fingerprint that
|
||||
was never seen, an alert with a newer `startsAt`, or a resolved alert that
|
||||
started again. The unchanged firing notifications Alertmanager re-sends every
|
||||
`repeat_interval` are none of those, and open nothing.
|
||||
|
||||
This is what makes closing an incident by hand mean something. Without the rule,
|
||||
`POST /api/incidents/{id}/resolve` would be undone by the next re-send of an
|
||||
alert that never stopped firing.
|
||||
|
||||
## Leaving the open state
|
||||
|
||||
- **Automatically**, once every alert under the incident has stopped firing —
|
||||
whether by a resolved webhook or by the sweeper's
|
||||
[stale-alert expiry](#stale-alert-expiry). The incident gets
|
||||
`"resolution_source": "alerts"`.
|
||||
- **By hand**, via `POST /api/incidents/{id}/resolve`
|
||||
(`"resolution_source": "manual"`). This is **terminal**: a later occurrence in
|
||||
that group opens a *new* incident rather than reopening this one. If the alert
|
||||
underneath never stops firing, the incident stays closed — that is what
|
||||
resolving by hand asserts.
|
||||
- **On recovery**, for a [dead man's switch](./dead-mans-switch.md) incident whose
|
||||
heartbeat started arriving again (`"resolution_source": "recovered"`). These
|
||||
incidents have no member alerts, so the automatic cascade above cannot reach
|
||||
them.
|
||||
|
||||
To quieten an incident you expect to come back, snooze it instead
|
||||
(`POST /api/incidents/{id}/snooze`). A snooze hides the incident from the default
|
||||
list without closing it, and expires by simply falling into the past.
|
||||
|
||||
## On-call assignment
|
||||
|
||||
A new incident is assigned to whoever holds today's schedule entry at the moment
|
||||
it opens (`GET /api/schedule/current`). If nobody is scheduled it opens
|
||||
unassigned. Reassign with `POST /api/incidents/{id}/assign`.
|
||||
|
||||
One person holds a given day, so `POST /api/schedule` refuses a date somebody
|
||||
already has: taking a shift off the person expecting to be paged for it should
|
||||
not be something a plain call does by accident. Pass `"replace": true` to take
|
||||
them anyway. Either way the whole request is one transaction — a week where some
|
||||
days are free and some are taken moves as a unit, and a failure leaves the rota
|
||||
exactly as it was rather than with a hole in it.
|
||||
|
||||
## Stale alert expiry
|
||||
|
||||
A resolved webhook is the only signal that an alert has stopped firing, so a
|
||||
notification that is dropped, silenced, or lost to a restart would otherwise pin
|
||||
that alert as firing forever. A background sweeper resolves firing alerts that
|
||||
Alertmanager has stopped refreshing, using either signal:
|
||||
|
||||
- the `endsAt` watermark on the last notification has passed, or
|
||||
- no webhook has refreshed the alert within `TERDUT_STALE_AFTER`.
|
||||
|
||||
Alertmanager re-sends firing notifications every `repeat_interval`, which is what
|
||||
keeps a live alert fresh — so `TERDUT_STALE_AFTER` must be comfortably larger
|
||||
than your `repeat_interval` (default 4h), or live alerts will be resolved
|
||||
prematurely. Alerts resolved this way are marked `"resolution_source": "expiry"`
|
||||
to distinguish them from a real Alertmanager resolve (`"alertmanager"`).
|
||||
|
||||
An expiry cascades: once it leaves an incident with nothing firing under it, the
|
||||
incident resolves too, in the same sweep.
|
||||
@@ -0,0 +1,62 @@
|
||||
# Push notifications
|
||||
|
||||
_Pages through ntfy, who gets them and how to acknowledge from the notification._ Back to the [README](../README.md) and the [documentation index](./README.md).
|
||||
|
||||
With `TERDUT_NTFY_URL` set, an incident that opens is pushed to the on-call
|
||||
person's phone through [ntfy](https://ntfy.sh). Everybody sets their own topic
|
||||
under *Account* in the web UI, where a **Send a test push** button proves it
|
||||
before an incident has to; `PUT /api/users/{id}/notify` is the same thing over
|
||||
the API, and an administrator may set somebody else's. A user with no topic
|
||||
falls back to `TERDUT_NTFY_FALLBACK_TOPIC`, as does an incident that opens with
|
||||
nobody on call. If neither yields a topic, nothing is queued.
|
||||
|
||||
The **server** is the install's one ntfy, from `TERDUT_NTFY_URL`, and is not
|
||||
something a user picks. Only the topic is per-person.
|
||||
|
||||
A topic is a shared secret with the ntfy server: anyone who knows it can both
|
||||
read the pages and publish to it, so an unguessable one is worth the trouble.
|
||||
That is also why the topic never appears in an incident's timeline, which every
|
||||
API key can read.
|
||||
|
||||
Three things get pushed:
|
||||
|
||||
- **triggered** — an incident opened. Priority follows severity (`critical` maps
|
||||
to ntfy's max priority, the one that overrides the phone's quiet settings).
|
||||
- **reminder** — the incident is still `triggered` after `TERDUT_NOTIFY_REPEAT`.
|
||||
Repeats until somebody acts. Acknowledging, snoozing, resolving or archiving
|
||||
all stop it — snooze is the mute button.
|
||||
- **resolved** — every alert under the incident stopped firing. Only sent to
|
||||
whoever was paged in the first place, and only for the automatic cascade:
|
||||
resolving by hand pushes nothing, since the person who did it already knows.
|
||||
|
||||
Notifications carry an **Acknowledge** button that acknowledges the incident
|
||||
without opening anything. It POSTs to `/api/notify/ack/{token}`, an
|
||||
unauthenticated route authorised by the 256-bit token in its path — minted fresh
|
||||
per notification, scoped to one incident and one action, and valid for 24 hours.
|
||||
A real API key is never put in a notification, because the message is stored on
|
||||
the ntfy server and cached on the device.
|
||||
|
||||
The token is **not** consumed by use. Acknowledging is idempotent, so a token
|
||||
stays valid for its full 24 hours and a second tap is a no-op that reports the
|
||||
incident's current state rather than an error — which is what you want when a
|
||||
tap is retried on a flaky mobile connection. What bounds it is scope, not a use
|
||||
count: one incident, one action, one day. Expired tokens are purged by the
|
||||
sweeper.
|
||||
|
||||
Two consequences worth planning for:
|
||||
|
||||
- `/api/notify/ack/{token}` **must stay publicly reachable**, or the button will
|
||||
not work when the responder is off your network.
|
||||
- Notifications sent to the fallback topic carry **no** Acknowledge button. The
|
||||
topic is shared, and a button on it would let any subscriber acknowledge as
|
||||
somebody else.
|
||||
|
||||
Delivery is a queue, not an inline call: the webhook writes a row and a
|
||||
background notifier sends it within 30 seconds, retrying with exponential
|
||||
backoff up to 8 attempts. Nothing about ingestion blocks on ntfy being reachable.
|
||||
|
||||
Every delivery is recorded on the incident's timeline: a `notified` event once
|
||||
ntfy accepts the publish, and a `notify_failed` event when a notification
|
||||
exhausts its retries. Written from the result rather than at enqueue, so the
|
||||
timeline says what actually happened — and a page that never landed is visible
|
||||
instead of looking the same as one that did.
|
||||
@@ -0,0 +1,105 @@
|
||||
# Single sign-on (OIDC)
|
||||
|
||||
_Signing in through an OpenID Connect provider, and mapping its groups to teams and administrators._ Back to the [README](../README.md) and the [documentation index](./README.md).
|
||||
|
||||
terdut can sign people in through any OpenID Connect provider; the examples use
|
||||
[Authentik](https://goauthentik.io/). Groups at the provider decide who may sign
|
||||
in, which teams they belong to and whether they administer the install, much as
|
||||
Grafana's OAuth role and org mapping does. Password login keeps working alongside
|
||||
it unless you turn it off.
|
||||
|
||||
**At the provider**, create an OAuth2/OpenID provider and an application for it:
|
||||
a *confidential* client, redirect URI `<TERDUT_PUBLIC_URL>/api/oidc/callback`, and
|
||||
the `openid`, `profile` and `email` scopes. The issuer is the application's, e.g.
|
||||
`https://auth.example.com/application/o/terdut/`. Then set:
|
||||
|
||||
```sh
|
||||
TERDUT_PUBLIC_URL=https://terdut.example.com
|
||||
TERDUT_OIDC_ISSUER=https://auth.example.com/application/o/terdut/
|
||||
TERDUT_OIDC_CLIENT_ID=terdut
|
||||
TERDUT_OIDC_CLIENT_SECRET=...
|
||||
TERDUT_OIDC_ALLOWED_GROUPS=terdut-users,terdut-admins
|
||||
TERDUT_OIDC_ADMIN_GROUP=terdut-admins
|
||||
```
|
||||
|
||||
Which team a group grants is not server-wide config: each team names its own
|
||||
group(s), set by that team's own owner (or an administrator) from its Members
|
||||
tab, or `PUT /api/teams/{teamID}/oidc-groups {"member_group":"sre","owner_group":"sre-leads"}`.
|
||||
A team must already exist before a group can grant access to it — the sync
|
||||
never creates one.
|
||||
|
||||
The web UI's sign-in page shows a "Sign in with <name>" button (a plain link to
|
||||
`/api/oidc/login`) above the password form, or instead of it when
|
||||
`TERDUT_PASSWORD_LOGIN=false`; it asks `GET /api/auth/config` what the server offers
|
||||
(`password_login`, `oidc.enabled`, `oidc.name`). A refused sign-in comes back to that
|
||||
page with the reason spelled out. Access the groups grant is badged **SSO** on the
|
||||
Team, Admin and per-user pages, with its edit and remove controls disabled, and the
|
||||
Account page does not offer to set a password nobody could use.
|
||||
|
||||
**What a sign-in does**
|
||||
|
||||
1. *Who.* The provider's `(issuer, subject)` is the identity. The first time, a
|
||||
user is found by email — only when the provider marks it verified, or
|
||||
`TERDUT_OIDC_TRUST_EMAIL` is set — or created with no password. A username taken
|
||||
by somebody else gets a numeric suffix (`alice-2`). Username and email follow the
|
||||
provider at each sign-in. Authentik reports `email_verified` as false unless
|
||||
configured otherwise, so linking existing users usually needs
|
||||
`TERDUT_OIDC_TRUST_EMAIL=true`.
|
||||
2. *Whether.* With `TERDUT_OIDC_ALLOWED_GROUPS` set, somebody in none of them is
|
||||
refused and nothing is created.
|
||||
3. *What.* The administrator flag follows `TERDUT_OIDC_ADMIN_GROUP`. Team roles
|
||||
follow each team's own `oidc_member_group`/`oidc_owner_group`; where both of a
|
||||
team's groups match, the owner group wins.
|
||||
|
||||
**Managed access.** What the sync grants is marked as managed by single sign-on,
|
||||
and only that is ever changed by it. It is added at sign-in, and removed at the
|
||||
next sign-in after the group is gone, even if that leaves a team without an owner
|
||||
(an administrator can always repair a team) — the provider is the source of truth
|
||||
for what it grants, so the last-owner and last-administrator guards do not apply.
|
||||
Memberships and administrators added by hand are left alone; the exception is a
|
||||
hand-added member whose team's own group grants a *higher* role, who is raised and
|
||||
from then on managed. Editing managed access by hand (`POST` or `DELETE` on a
|
||||
team's members, revoking an SSO-granted administrator) is refused with `409`, since
|
||||
the next sign-in would undo it.
|
||||
|
||||
> **Upgrading past migration 013: reconfigure every team's groups.**
|
||||
> `TERDUT_OIDC_GROUP_MAPPINGS` is gone, and the sync no longer creates a team by
|
||||
> name. Group-to-team-role mapping is now each team's own setting — an owner sets
|
||||
> it from the Members tab, or `PUT /api/teams/{teamID}/oidc-groups`. Until a team's
|
||||
> owner does that, an OIDC-sourced membership in it is dropped at that user's next
|
||||
> SSO sign-in, the same as any other loss of group access. Set every team's groups
|
||||
> before affected users next sign in, to avoid a visible gap in access.
|
||||
|
||||
**How fast changes arrive.** Groups are read only at sign-in. A session made by an
|
||||
SSO sign-in has a hard ceiling (`TERDUT_OIDC_SESSION_MAX_AGE`, default 12h) that
|
||||
sliding never extends, so a change at the provider reaches terdut within that time.
|
||||
Password sessions are unaffected.
|
||||
|
||||
> **API keys are not revoked when somebody is removed at the provider.** terdut
|
||||
> holds no refresh token and never asks the provider again, so a person removed
|
||||
> from every allowed group loses their sessions within `TERDUT_OIDC_SESSION_MAX_AGE`
|
||||
> and cannot sign in again, but keeps any API key they made (the TUI and scripts use
|
||||
> them) until an administrator disables the user in terdut.
|
||||
|
||||
**Signing in from a terminal.** A client with no browser of its own, such as the
|
||||
TUI over SSH, signs in with a device code, run by terdut itself so the terminal
|
||||
never talks to the provider:
|
||||
|
||||
1. The terminal calls `POST /api/oidc/device` and shows the person a link
|
||||
(`<TERDUT_PUBLIC_URL>/device?code=XXXX-XXXX`) and the code.
|
||||
2. On any device the person opens the link, signs in (by the provider or by
|
||||
password, whatever the login page offers), sees the code and the account, and
|
||||
presses **Approve**. Only a browser session can approve; an API key cannot.
|
||||
3. The terminal polls `POST /api/oidc/device/token` every 5 seconds and is given the
|
||||
ordinary `terdut_session` cookie once. A person who signs in through the provider
|
||||
gets the same `TERDUT_OIDC_SESSION_MAX_AGE` ceiling on the terminal's session as
|
||||
on their browser's.
|
||||
|
||||
A login expires after 10 minutes. `GET /api/auth/config` reports `device_login`.
|
||||
|
||||
**If the provider is down**, terdut still starts (discovery is fetched on first
|
||||
use) and password login is the way in. With `TERDUT_PASSWORD_LOGIN=false` that way
|
||||
is closed: set it back to `true`. The first administrator comes from the bootstrap
|
||||
endpoint, and stays a manual administrator that no group can revoke; on an SSO-only
|
||||
install set `bootstrap.enabled: false` in the chart if you don't want that account,
|
||||
or keep it and never give it a password.
|
||||
@@ -0,0 +1,90 @@
|
||||
# The web UI
|
||||
|
||||
_What the web UI offers, how sign-in and sessions work, and the Team and Admin tabs._ Back to the [README](../README.md) and the [documentation index](./README.md).
|
||||
|
||||
The server serves a web UI at `/`: the incident queue, each incident's alerts
|
||||
and timeline with every action (acknowledge, assign, snooze, note, resolve,
|
||||
archive), who is on call, the alert feed, and an *Account* tab for your own
|
||||
password and the ntfy topic your pages go to. It is built for a phone first. On a phone
|
||||
it navigates through a hamburger menu and has a sticky action bar, it follows the
|
||||
system's dark mode, and it can be added to the home screen. From 900px wide it switches
|
||||
to a sidebar with the queue and the incident side by side. The Stats page shows
|
||||
incident counts, MTTA and MTTR, and alert frequency by name, hour and day over a
|
||||
chosen range.
|
||||
|
||||
You sign in with a username and password. Users have no password until one is
|
||||
set, and a user without one can only use API keys:
|
||||
|
||||
```bash
|
||||
# an admin sets someone's first password with their API key
|
||||
curl -X PUT http://localhost:8080/api/users/2/password \
|
||||
-H "Authorization: Bearer $KEY" -H "Content-Type: application/json" \
|
||||
-d '{"password": "<at least 10 characters>"}'
|
||||
```
|
||||
|
||||
After that, users change it themselves under *Account*. Changing your own
|
||||
password requires the current one.
|
||||
|
||||
How a browser stays signed in:
|
||||
|
||||
- A successful login sets an `HttpOnly`, `SameSite=Lax` session cookie. It lasts
|
||||
30 days and slides forward while it is used, so an on-call phone stays signed
|
||||
in.
|
||||
- The cookie is marked `Secure` when `TERDUT_PUBLIC_URL` starts with `https://`,
|
||||
so set it to the HTTPS address. TLS terminates at the gateway and the server
|
||||
itself only ever sees plain HTTP.
|
||||
- Requests authenticated by the cookie are checked for cross-origin use (Go's
|
||||
`http.CrossOriginProtection`). That is the CSRF guard. Bearer-key clients are
|
||||
not affected.
|
||||
- Setting a password signs that user out everywhere else.
|
||||
- Ten failed logins for one username within 15 minutes lock that username for
|
||||
the rest of the window.
|
||||
|
||||
With `TERDUT_PUBLIC_URL` set, tapping a push notification opens the incident in
|
||||
the web UI (`/incidents/{id}`).
|
||||
|
||||
A **Team** tab holds everything a team owns, in five sub-sections with a URL
|
||||
each and a strip across the top to move between them: the on-call rota
|
||||
(`/team/rota`), the membership (`/team/members`), the escalation ladder
|
||||
(`/team/escalation`), the alert sources with their keys (`/team/sources`) and
|
||||
the dead man's switches (`/team/deadman`). `/team` itself is an overview — who
|
||||
is on call today, how many members and owners, how many ladder levels, how many
|
||||
keys and how many switches — so a page fetches only what it shows. An owner
|
||||
edits it; a member sees the same pages read-only, because the server refuses
|
||||
their writes anyway. Somebody in more than one team picks between them above
|
||||
the strip, since the choice changes the subject of all five.
|
||||
|
||||
The rota is a month at a time, one coloured initial per day with a legend
|
||||
underneath, and it says how many days are left uncovered — the question a rota
|
||||
is read for is who holds which stretch, and a run of one colour answers it
|
||||
where a list of dates does not. An owner taps a day to hand it to somebody or
|
||||
empty it, and fills a whole shift from the range form folded in below.
|
||||
|
||||
The **Admin** tab appears only for a system administrator, and holds what
|
||||
belongs to the whole server rather than to one team. It has three sub-sections,
|
||||
each with a URL of its own and a strip across the top to move between them:
|
||||
every team (`/admin/teams`), every user (`/admin/users`), and the settings that
|
||||
used to be environment variables (`/admin/settings`). `/admin` itself is an
|
||||
overview — how many of each, and what each section is for. Adding somebody is
|
||||
minting them an invite link into a team, rather than creating a bare account:
|
||||
the person who accepts it picks their own password, so one never passes through
|
||||
an administrator, and the link carries the team, so they land somewhere with a
|
||||
queue in it. That happens on the team's own page, since an invite is a fact
|
||||
about a team; the user list points there rather than asking which team beside a
|
||||
form.
|
||||
|
||||
A name in the team list opens **that team's page**, at `/admin/teams/{id}`: when it
|
||||
was created, how many are in it and how much is open, a field to rename it, the
|
||||
members with their roles, the invites into it, and deletion. The member list is the
|
||||
one thing there that needed a new endpoint — `GET /api/teams/{id}/members` is
|
||||
member-only and answers `404` to an administrator who is not in the team, which is
|
||||
the rule and not an oversight, so the page reads `GET /api/admin/teams/{id}` instead.
|
||||
An administrator still sees none of that team's incidents, alerts or rota.
|
||||
|
||||
A name in the user list opens **that person's page**, at `/admin/users/{id}`: their
|
||||
email and when they joined, where their notifications go, whether they are an
|
||||
administrator, whether the account is disabled, the teams they are in with their
|
||||
role in each, a password field for a first or forgotten one, and deletion. It is
|
||||
the one place membership is edited from the person's side — the Team tab answers
|
||||
"who is in this team", and answering "which teams is this person in" there means
|
||||
visiting each team in turn.
|
||||
Reference in New Issue
Block a user