Set the chart's placeholder version to 0.13.0

Cosmetic, as in 4c85e76 and 041e159. `make helm-package` passes --version
and --app-version from the tag, so neither field decides anything about
what release.yaml publishes.

Done anyway because a tree heading for v0.13.0 that still says 0.12.0
tells a reader something false. appVersion keeps the v, per
APPVERSION_PREFIX in .release.conf.

README.md

@@ -155,12 +155,25 @@ somewhere to exec. The sidecar, the PVC and the `backupSidecar` values are all g
 
 ## Configuration
 
+Two kinds of setting, split by who changes them and how often.
+
+**Where the server is plugged in** stays in the environment: the listen address,
+the database DSN, the ntfy URL and token, the public URL. They are needed before
+the database is open, and two of them are credentials.
+
+**How the server behaves** lives in the database and is edited by an
+administrator in the web UI or through `PUT /api/admin/settings`, taking effect
+on the next sweep rather than at the next restart. The variables below marked
+**seed** are the value each of those starts from: written once, on first start,
+and never overwritten afterwards — a redeploy cannot put a chart's default back
+over an administrator's edit.
+
 | Variable | Default | Description |
 |---|---|---|
 | `TERDUT_ADDR` | `:8080` | TCP address to listen on |
 | `TERDUT_DB_DSN` | — | **Required.** Postgres connection string, e.g. `postgres://terdut:secret@localhost:5432/terdut?sslmode=require` |
-| `TERDUT_ARCHIVE_AFTER` | `168h` (7d) | How long a resolved alert or incident stays in the default list before being auto-archived |
+| `TERDUT_ARCHIVE_AFTER` | `168h` (7d) | **seed.** How long a resolved alert or incident stays in the default list before being auto-archived |
-| `TERDUT_STALE_AFTER` | `6h` | How long a firing alert may go without a refreshing webhook before it is treated as resolved — **must exceed your Alertmanager `repeat_interval`** |
+| `TERDUT_STALE_AFTER` | `6h` | **seed.** How long a firing alert may go without a refreshing webhook before it is treated as resolved — **must exceed your Alertmanager `repeat_interval`** |
 | `TERDUT_DEADMAN_MATCHERS` | `alertname=Watchdog` | The **default** matchers a team starts with — switches are per team now, and this seeds teams that have no configuration of their own. `;` separates matchers, `,` the label conditions within one, `=` is exact equality. Every matcher must name an `alertname` |
 | `TERDUT_DEADMAN_TIMEOUT` | `15m` | How long a heartbeat may go unheard before its switch is declared dead — **must be shorter than the `repeat_interval` of the route carrying it**. `0` disables dead man's switch handling |
 | `TERDUT_DEADMAN_SEVERITY` | `critical` | Severity a dead man's switch incident opens at |
@@ -168,7 +181,7 @@ somewhere to exec. The sidecar, the PVC and the `backupSidecar` values are all g
 | `TERDUT_NTFY_TOKEN` | — | Bearer token for an access-controlled ntfy |
 | `TERDUT_NTFY_FALLBACK_TOPIC` | — | Topic used when nobody is on call |
 | `TERDUT_PUBLIC_URL` | — | Base URL a phone uses to reach this server: the notification's link into the web UI, its Acknowledge button, and whether the session cookie is `Secure` |
-| `TERDUT_NOTIFY_REPEAT` | `15m` | How long an incident may sit unacknowledged before it is paged again. `0` notifies once and never repeats |
+| `TERDUT_NOTIFY_REPEAT` | `15m` | **seed.** How long an incident may sit unacknowledged before it is paged again. `0` notifies once and never repeats |
 
 Durations use Go syntax (`30m`, `12h`, `168h`). An unparseable value falls back to the default.
 
@@ -182,19 +195,39 @@ In the Helm chart the two sweeper durations are set via `sweeper.staleAfter` and
 
 ## Alertmanager configuration
 
-Add terdut-server as a webhook receiver in your `alertmanager.yml`:
+Alerts arrive on a team's **integration key**, which says both that the sender
+may post and which team the alerts belong to. Mint one as an owner of the team:
+
+```bash
+curl -X POST https://terdut.example.com/api/teams/1/integrations \
+  -H "Authorization: Bearer $TERDUT_API_KEY" \
+  -H 'Content-Type: application/json' \
+  -d '{"name":"prod alertmanager"}'
+```
+
+The response carries the key and the full URL **once**; only a SHA-256 hash is
+stored. Put it in your `alertmanager.yml`:
 
 ```yaml
 receivers:
   - name: terdut
     webhook_configs:
-      - url: http://terdut-server:8080/api/alertmanager/webhook
+      - url: http://terdut-server:8080/api/integrations/<key>/alertmanager
         send_resolved: true
 
 route:
   receiver: terdut
 ```
 
+The whole URL is a credential, so treat it like one. Alertmanager 0.26 and
+later can read it from a file with `url_file:` instead, which keeps it out of
+your configuration repository:
+
+```yaml
+      - url_file: /etc/alertmanager/secrets/terdut-webhook-url/url
+        send_resolved: true
+```
+
 The webhook endpoint requires no authentication.
 
 If you use the [dead man's switch](#dead-mans-switch) — and the default configuration does — give
@@ -344,6 +377,50 @@ exhausts its retries. Written from the result rather than at enqueue, so the
 timeline says what actually happened — and a page that never landed is visible
 instead of looking the same as one that did.
 
+### Escalation
+
+Without a ladder, an unacknowledged incident re-pages the same topic every
+`notify_repeat` forever. That is a louder version of the same silence: if the
+person on call is asleep, out of signal, or has left, nothing else happens.
+
+A team can configure an ordered ladder instead. Each level has a timeout and a
+set of targets, and a target is either a named person or **whoever the team's
+rota says is on call today** — the target that keeps working when the rota
+changes and nobody remembers to edit the policy.
+
+```
+level 1   5m    oncall            the rota gets first refusal
+level 2   5m    user:bob          then a named second
+                                  then repeat_count more rounds
+                                  then the team's fallback topic, once
+```
+
+When a level's timeout passes with the incident still `triggered`, the next
+level is paged. Off the end of the ladder the whole thing runs again
+`repeat_count` times, and after that the team's `fallback_topic` is paged once
+as the end of the line. The incident stays open throughout: running out of
+people to wake is not the same as somebody answering.
+
+**Acknowledging or resolving stops it**, which is the point — continuing to wake
+people after somebody has said "I have this" is how a tool teaches people to
+mute it. **Snoozing pauses it**: a deliberate "not now" holds the ladder where
+it is, and it resumes when the snooze runs out.
+
+Every step is on the incident's timeline with the level and the names it woke,
+so somebody reading it afterwards can tell why their phone rang at 04:00. A
+level whose targets are all unreachable — no ntfy topic, a disabled account, an
+empty rota — is recorded as `nobody reachable` and the ladder moves on rather
+than stalling on a rung that cannot ring.
+
+**Reminders and escalation never both run.** A team with a ladder gets
+escalation; a team without keeps the reminder behaviour exactly as it was. Two
+pages for one silence is the surest way to get a tool muted.
+
+The ladder's `fallback_topic` is per team, unlike `TERDUT_NTFY_FALLBACK_TOPIC`,
+which is the install-wide topic used when an incident opens with nobody on call.
+They answer different questions: one is "nobody was scheduled", the other is
+"everybody scheduled has been tried".
+
 ### Stale alert expiry
 
 A resolved webhook is the only signal that an alert has stopped firing, so a
@@ -447,7 +524,7 @@ of the last heartbeat, and the heartbeat's labels are on the incident's
 
 ### Authentication
 
-All endpoints except `/api/bootstrap`, `/api/alertmanager/webhook`,
+All endpoints except `/api/bootstrap`, `/api/integrations/{key}/alertmanager`,
 `/api/notify/ack/{token}`, `/api/login` and `/api/logout` require either an API key:
 
 ```
@@ -502,11 +579,20 @@ on anybody's.
 | `POST` | `/api/users` | **admin** | Create user `{"username","email"}`. Not an administrator |
 | `DELETE` | `/api/users/{id}` | **admin** | Delete user (cascades to keys). `409` for yourself or the last administrator |
 | `PUT` | `/api/users/{id}/admin` | **admin** | Grant or revoke the administrator flag `{"is_admin"}`. `409` for yourself or the last administrator |
+| `PUT` | `/api/users/{id}/disabled` | **admin** | Take an account out of use, or put it back `{"disabled"}`. `409` for yourself or the last administrator |
 | `PUT` | `/api/users/{id}/notify` | self or admin | Set push notification target `{"ntfy_topic"}` — empty string clears it |
 | `PUT` | `/api/users/{id}/password` | self or admin | Set web UI password `{"password","current_password"}`. `current_password` is required only when changing your own existing password. Ends the user's other sessions |
 | `POST` | `/api/users/{id}/api-keys` | self or admin | Issue API key `{"name"}` — key shown once |
 | `DELETE` | `/api/users/{id}/api-keys/{keyID}` | self or admin | Revoke API key |
 
+### Administration
+
+| Method | Path | Who | Description |
+|---|---|---|---|
+| `GET` | `/api/admin/teams` | **admin** | Every team on the server, with its member and open-incident counts. `/api/teams` answers "what am I in"; this answers "what is there" |
+| `GET` | `/api/admin/settings` | **admin** | The editable settings with their bounds, plus the environment-configured ones, read-only. Never credentials |
+| `PUT` | `/api/admin/settings` | **admin** | Change one or more `{"key": seconds}`. `400` for an unknown key or a value outside its bounds |
+
 ### Alert ingestion
 
 Alerts arrive on a team's integration key. The key is both the credential and the
@@ -517,10 +603,10 @@ and the full URL once and stores only a SHA-256 hash.
 | Method | Path | Description |
 |---|---|---|
 | `POST` | `/api/integrations/{key}/alertmanager` | Alertmanager v4 webhook receiver for the key's team. `401` for an unknown key |
-| `POST` | `/api/alertmanager/webhook` | **Deprecated, unauthenticated.** The pre-teams receiver, kept for one release so an upgrade does not stop delivering while the Alertmanager config is edited. Routes everything to the oldest team |
 
-The deprecated path is why anything that can reach the port can still open an
+This is the only way in. The pre-teams `POST /api/alertmanager/webhook` took no
-incident. Move senders to a key and it goes away.
+credential at all — anything able to reach the port could open an incident —
+and was removed in v0.13.0 once senders had moved onto keys.
 
 ### Teams
 
@@ -528,6 +614,7 @@ incident. Move senders to a key and it goes away.
 |---|---|---|---|
 | `GET` | `/api/teams` | any | The caller's own teams, each with their role |
 | `POST` | `/api/teams` | any | Create a team `{"name"}`; the creator becomes its first owner |
+| `PUT` | `/api/teams/{teamID}` | **owner** | Rename it `{"name"}`. `409` if the name is taken |
 | `DELETE` | `/api/teams/{teamID}` | **owner** | Delete a team and everything under it. `409` while it has open incidents |
 | `GET` | `/api/teams/{teamID}/members` | member | Who is in the team |
 | `POST` | `/api/teams/{teamID}/members` | **owner** | Add a member, or change their role `{"user_id","role"}` |
@@ -535,6 +622,8 @@ incident. Move senders to a key and it goes away.
 | `GET` | `/api/teams/{teamID}/integrations` | member | List integrations. Never returns keys |
 | `POST` | `/api/teams/{teamID}/integrations` | **owner** | Mint an integration `{"name","kind"}` — key and URL shown once |
 | `DELETE` | `/api/teams/{teamID}/integrations/{integrationID}` | **owner** | Revoke an integration |
+| `GET` | `/api/teams/{teamID}/escalation` | member | The team's [escalation ladder](#escalation) `{repeat_count, fallback_topic, levels[]}`. Empty levels means the team has none |
+| `PUT` | `/api/teams/{teamID}/escalation` | **owner** | Replace it wholesale. `400` for a level with no targets or no timeout — a rung that pages nobody is a silence with a number on it |
 | `GET` | `/api/teams/{teamID}/deadman` | member | The team's [dead man's switch](#dead-mans-switch) configuration `{matchers, timeout_seconds, severity}` |
 | `PUT` | `/api/teams/{teamID}/deadman` | **owner** | Replace it. `400` when no matcher names an `alertname`, because a switch that silently watches nothing is the failure this feature exists to prevent |
 
@@ -765,10 +854,11 @@ name on them.
 
 What changes, and will need attention:
 
-- **Alert ingestion moved.** `POST /api/alertmanager/webhook` still works but is
+- **Alert ingestion moved.** Mint a key with
-  deprecated and unauthenticated, and routes everything to the oldest team. Mint
+  `POST /api/teams/{teamID}/integrations` and point Alertmanager at the URL it
-  a key with `POST /api/teams/{teamID}/integrations` and point Alertmanager at
+  returns. In v0.12.0 the old `POST /api/alertmanager/webhook` still worked,
-  the URL it returns. The old path goes away in a later release.
+  deprecated, routing everything to the oldest team; **v0.13.0 removes it**, so
+  upgrade straight from v0.11.x to v0.13.0 only after the senders are moved.
 - **The schedule endpoints moved** under `/api/teams/{teamID}/schedule`, and
   editing the rota is now an owner's job. `GET /api/schedule/current` stayed
   where it was but now returns an **array** — one entry per team with somebody
@@ -810,30 +900,25 @@ non-administrator now gets `403` where a `200` used to come back.
 
 ## Upgrading from SQLite
 
-Versions up to v0.10.2 stored everything in a SQLite file. From the Postgres release onwards
+Versions up to v0.10.2 stored everything in a SQLite file. From v0.11.1 the server needs
-the server needs `TERDUT_DB_DSN` and keeps nothing on disk.
+`TERDUT_DB_DSN` and keeps nothing on disk.
 
-The cutover is ordered — the server must not be running while the copy happens:
+The copy was done by `scripts/sqlite-to-postgres.go`, which **was deleted in v0.13.0** along
+with the SQLite driver it was the last user of. It is still in the history — check out the
+`v0.12.0` tag to get it:
 
 ```bash
-# 1. Stop the old server, keeping its database file.
+git show v0.12.0:scripts/sqlite-to-postgres.go > sqlite-to-postgres.go
-# 2. Create an empty Postgres database, then let the new binary build the schema:
-TERDUT_DB_DSN='postgres://terdut:secret@localhost:5432/terdut?sslmode=disable' ./terdut &
-#    ...watch for "listening on", then stop it again.
-# 3. Copy the data across:
-go run -tags migrate ./scripts/sqlite-to-postgres.go \
-    -sqlite /data/terdut.db \
-    -dsn 'postgres://terdut:secret@localhost:5432/terdut?sslmode=disable'
-# 4. Start the new server for good.
 ```
 
+The cutover is ordered, and the server must not be running while the copy happens: stop the
+old version, let the new binary build the schema against an empty Postgres, run the script
+with `-sqlite` and `-dsn`, then start the new version for good. On Kubernetes step three runs
+as a Job with the same image against the PVC before it is removed.
+
 The copy preserves every id, so incidents keep their numbers and the timeline, alert
 membership, outbox and ack tokens all still point where they did. It refuses a target that
-already has rows, so a second run cannot double-insert. On Kubernetes, step 3 runs as a Job
+already has rows, so a second run cannot double-insert.
-with the same image against the PVC before it is removed.
-
-The script is deliberately temporary: it is the only thing left that needs the SQLite driver,
-and both should be deleted once the installs that need them have migrated.
 
 ## Upgrading to incidents
 

charts/terdut-server/Chart.yaml

@@ -15,5 +15,5 @@ type: application
 # appVersion and image.tag in values.yaml no longer agree, and that is not an oversight:
 # image.tag stays "latest", which is what a local install actually pulls. appVersion is
 # metadata and drives nothing.
-version: 0.12.0
+version: 0.13.0
-appVersion: "v0.12.0"
+appVersion: "v0.13.0"

cmd/terdut/main.go

@@ -45,7 +45,13 @@ func main() {
 		log.Fatalf("seed dead man's switch defaults: %v", err)
 	}
 
-	router := api.NewRouter(database, notify)
+	// The behaviour knobs move into the database on first start, after which an
+	// administrator owns them and a redeploy leaves them alone.
+	if err := api.SeedSettings(context.Background(), database, cfg); err != nil {
+		log.Fatalf("seed settings: %v", err)
+	}
+
+	router := api.NewRouter(database, notify, cfg)
 
 	srv := &http.Server{
 		Addr:         cfg.Addr,

go.mod

@@ -7,22 +7,12 @@ require (
 	github.com/jackc/pgerrcode v0.0.0-20250907135507-afb5586c32a6
 	github.com/jackc/pgx/v5 v5.11.0
 	golang.org/x/crypto v0.55.0
-	modernc.org/sqlite v1.50.1
 )
 
 require (
-	github.com/dustin/go-humanize v1.0.1 // indirect
-	github.com/google/uuid v1.6.0 // indirect
 	github.com/jackc/pgpassfile v1.0.0 // indirect
 	github.com/jackc/pgservicefile v0.0.0-20240606120523-5a60cdf6a761 // indirect
 	github.com/jackc/puddle/v2 v2.2.2 // indirect
-	github.com/mattn/go-isatty v0.0.20 // indirect
-	github.com/ncruces/go-strftime v1.0.0 // indirect
-	github.com/remyoudompheng/bigfft v0.0.0-20230129092748-24d4a6f8daec // indirect
 	golang.org/x/sync v0.22.0 // indirect
-	golang.org/x/sys v0.47.0 // indirect
 	golang.org/x/text v0.41.0 // indirect
-	modernc.org/libc v1.72.3 // indirect
-	modernc.org/mathutil v1.7.1 // indirect
-	modernc.org/memory v1.11.0 // indirect
 )

go.sum

@@ -1,16 +1,8 @@
 github.com/davecgh/go-spew v1.1.0/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
 github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c=
 github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
-github.com/dustin/go-humanize v1.0.1 h1:GzkhY7T5VNhEkwH0PVJgjz+fX1rhBrR7pRT3mDkpeCY=
-github.com/dustin/go-humanize v1.0.1/go.mod h1:Mu1zIs6XwVuF/gI1OepvI0qD18qycQx+mFykh5fBlto=
 github.com/go-chi/chi/v5 v5.2.5 h1:Eg4myHZBjyvJmAFjFvWgrqDTXFyOzjj7YIm3L3mu6Ug=
 github.com/go-chi/chi/v5 v5.2.5/go.mod h1:X7Gx4mteadT3eDOMTsXzmI4/rwUpOwBHLpAfupzFJP0=
-github.com/google/pprof v0.0.0-20250317173921-a4b03ec1a45e h1:ijClszYn+mADRFY17kjQEVQ1XRhq2/JR1M3sGqeJoxs=
-github.com/google/pprof v0.0.0-20250317173921-a4b03ec1a45e/go.mod h1:boTsfXsheKC2y+lKOCMpSfarhxDeIzfZG1jqGcPl3cA=
-github.com/google/uuid v1.6.0 h1:NIvaJDMOsjHA8n1jAhLSgzrAzy1Hgr+hNrb57e+94F0=
-github.com/google/uuid v1.6.0/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo=
-github.com/hashicorp/golang-lru/v2 v2.0.7 h1:a+bsQ5rvGLjzHuww6tVxozPZFVghXaHOwFs4luLUK2k=
-github.com/hashicorp/golang-lru/v2 v2.0.7/go.mod h1:QeFd9opnmA6QUJc5vARoKUSoFhyfM2/ZepoAG6RGpeM=
 github.com/jackc/pgerrcode v0.0.0-20250907135507-afb5586c32a6 h1:D/V0gu4zQ3cL2WKeVNVM4r2gLxGGf6McLwgXzRTo2RQ=
 github.com/jackc/pgerrcode v0.0.0-20250907135507-afb5586c32a6/go.mod h1:a/s9Lp5W7n/DD0VrVoyJ00FbP2ytTPDVOivvn2bMlds=
 github.com/jackc/pgpassfile v1.0.0 h1:/6Hmqy13Ss2zCq62VdNG8tM1wchn8zjSGOBJ6icpsIM=
@@ -21,14 +13,8 @@ github.com/jackc/pgx/v5 v5.11.0 h1:IzBBtyK9AHqf98cctWFifYSci2hgQR/cd56wB4p+ogg=
 github.com/jackc/pgx/v5 v5.11.0/go.mod h1:mal1tBGAFfLHvZzaYh77YS/eC6IX9OWbRV1QIIM0Jn4=
 github.com/jackc/puddle/v2 v2.2.2 h1:PR8nw+E/1w0GLuRFSmiioY6UooMp6KJv0/61nB7icHo=
 github.com/jackc/puddle/v2 v2.2.2/go.mod h1:vriiEXHvEE654aYKXXjOvZM39qJ0q+azkZFrfEOc3H4=
-github.com/mattn/go-isatty v0.0.20 h1:xfD0iDuEKnDkl03q4limB+vH+GxLEtL/jb4xVJSWWEY=
-github.com/mattn/go-isatty v0.0.20/go.mod h1:W+V8PltTTMOvKvAeJH7IuucS94S2C6jfK/D7dTCTo3Y=
-github.com/ncruces/go-strftime v1.0.0 h1:HMFp8mLCTPp341M/ZnA4qaf7ZlsbTc+miZjCLOFAw7w=
-github.com/ncruces/go-strftime v1.0.0/go.mod h1:Fwc5htZGVVkseilnfgOVb9mKy6w1naJmn9CehxcKcls=
 github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM=
 github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4=
-github.com/remyoudompheng/bigfft v0.0.0-20230129092748-24d4a6f8daec h1:W09IVJc94icq4NjY3clb7Lk8O1qJ8BdBEF8z0ibU0rE=
-github.com/remyoudompheng/bigfft v0.0.0-20230129092748-24d4a6f8daec/go.mod h1:qqbHyh8v60DhA7CoWK5oRCqLrMHRGoxYCSS9EjAz6Eo=
 github.com/stretchr/objx v0.1.0/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME=
 github.com/stretchr/testify v1.3.0/go.mod h1:M5WIy9Dh21IEIfnGCwXGc5bZfKNJtfHm1UVUgZn+9EI=
 github.com/stretchr/testify v1.7.0/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/h/Wwjteg=
@@ -36,46 +22,11 @@ github.com/stretchr/testify v1.11.1 h1:7s2iGBzp5EwR7/aIZr8ao5+dra3wiQyKjjFuvgVKu
 github.com/stretchr/testify v1.11.1/go.mod h1:wZwfW3scLgRK+23gO65QZefKpKQRnfz6sD981Nm4B6U=
 golang.org/x/crypto v0.55.0 h1:+KWHjbgOaAQ66dh/YlkZKHlz9ZUlq61AFirAR9ntP8M=
 golang.org/x/crypto v0.55.0/go.mod h1:uq0V9dE/fzQuJtbnL+2EhWOE63vo164FY8xqEnV9xis=
-golang.org/x/mod v0.38.0 h1:MECBjubtXD7yj4HrhIUcywNaGeNVUdfVnxmPajOk4yk=
-golang.org/x/mod v0.38.0/go.mod h1:V6Xz0pq8TQ3dGqVQ1FVHuelZpAL0uNhSkk9ogYP3c40=
 golang.org/x/sync v0.22.0 h1:SZjpbeLmrCk4xhRSZFNZW5gFUeCeFgjekvI/+gfScek=
 golang.org/x/sync v0.22.0/go.mod h1:9xrNwdLfx4jkKbNva9FpL6vEN7evnE43NNNJQ2LF3+0=
-golang.org/x/sys v0.6.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
-golang.org/x/sys v0.47.0 h1:o7XGOvZQCADBQQ4Y7VNq2dRWQR7JmOUW8Kxx4ZsNgWs=
-golang.org/x/sys v0.47.0/go.mod h1:4GL1E5IUh+htKOUEOaiffhrAeqysfVGipDYzABqnCmw=
 golang.org/x/text v0.41.0 h1:vz/seA0lnX87Othu2f/0L24RcgrXD9/YFTSuGjj3rH8=
 golang.org/x/text v0.41.0/go.mod h1:jvf1O8ajNzZqhSrQBPbutR/EB83Cc0CFrezNQIwbb5M=
-golang.org/x/tools v0.48.0 h1:3+hClM1aLL5mjMKm5ovokw9epgRXPuu2tILgismM6RE=
-golang.org/x/tools v0.48.0/go.mod h1:08xX0orndb/F7jJxGDicx061tyd5pcMto75YMAXr6lk=
 gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
 gopkg.in/yaml.v3 v3.0.0-20200313102051-9f266ea9e77c/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=
 gopkg.in/yaml.v3 v3.0.1 h1:fxVm/GzAzEWqLHuvctI91KS9hhNmmWOoWu0XTYJS7CA=
 gopkg.in/yaml.v3 v3.0.1/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=
-modernc.org/cc/v4 v4.28.2 h1:3tQ0lf2ADtoby2EtSP+J7IE2SHwEJdP8ioR59wx7XpY=
-modernc.org/cc/v4 v4.28.2/go.mod h1:OnovgIhbbMXMu1aISnJ0wvVD1KnW+cAUJkIrAWh+kVI=
-modernc.org/ccgo/v4 v4.34.0 h1:yRLPFZieg532OT4rp4JFNIVcquwalMX26G95WQDqwCQ=
-modernc.org/ccgo/v4 v4.34.0/go.mod h1:AS5WYMyBakQ+fhsHhtP8mWB82KTGPkNNJDGfGQCe0/A=
-modernc.org/fileutil v1.4.0 h1:j6ZzNTftVS054gi281TyLjHPp6CPHr2KCxEXjEbD6SM=
-modernc.org/fileutil v1.4.0/go.mod h1:EqdKFDxiByqxLk8ozOxObDSfcVOv/54xDs/DUHdvCUU=
-modernc.org/gc/v2 v2.6.5 h1:nyqdV8q46KvTpZlsw66kWqwXRHdjIlJOhG6kxiV/9xI=
-modernc.org/gc/v2 v2.6.5/go.mod h1:YgIahr1ypgfe7chRuJi2gD7DBQiKSLMPgBQe9oIiito=
-modernc.org/gc/v3 v3.1.2 h1:ZtDCnhonXSZexk/AYsegNRV1lJGgaNZJuKjJSWKyEqo=
-modernc.org/gc/v3 v3.1.2/go.mod h1:HFK/6AGESC7Ex+EZJhJ2Gni6cTaYpSMmU/cT9RmlfYY=
-modernc.org/goabi0 v0.2.0 h1:HvEowk7LxcPd0eq6mVOAEMai46V+i7Jrj13t4AzuNks=
-modernc.org/goabi0 v0.2.0/go.mod h1:CEFRnnJhKvWT1c1JTI3Avm+tgOWbkOu5oPA8eH8LnMI=
-modernc.org/libc v1.72.3 h1:ZnDF4tXn4NBXFutMMQC4vtbTFSXhhKzR73fv0beZEAU=
-modernc.org/libc v1.72.3/go.mod h1:dn0dZNnnn1clLyvRxLxYExxiKRZIRENOfqQ8XEeg4Qs=
-modernc.org/mathutil v1.7.1 h1:GCZVGXdaN8gTqB1Mf/usp1Y/hSqgI2vAGGP4jZMCxOU=
-modernc.org/mathutil v1.7.1/go.mod h1:4p5IwJITfppl0G4sUEDtCr4DthTaT47/N3aT6MhfgJg=
-modernc.org/memory v1.11.0 h1:o4QC8aMQzmcwCK3t3Ux/ZHmwFPzE6hf2Y5LbkRs+hbI=
-modernc.org/memory v1.11.0/go.mod h1:/JP4VbVC+K5sU2wZi9bHoq2MAkCnrt2r98UGeSK7Mjw=
-modernc.org/opt v0.2.0 h1:tGyef5ApycA7FSEOMraay9SaTk5zmbx7Tu+cJs4QKZg=
-modernc.org/opt v0.2.0/go.mod h1:03fq9lsNfvkYSfxrfUhZCWPk1lm4cq4N+Bh//bEtgns=
-modernc.org/sortutil v1.2.1 h1:+xyoGf15mM3NMlPDnFqrteY07klSFxLElE2PVuWIJ7w=
-modernc.org/sortutil v1.2.1/go.mod h1:7ZI3a3REbai7gzCLcotuw9AC4VZVpYMjDzETGsSMqJE=
-modernc.org/sqlite v1.50.1 h1:l+cQvn0sd0zJJtfygGHuQJ5AjlrwXmWPw4KP3ZMwr9w=
-modernc.org/sqlite v1.50.1/go.mod h1:tcNzv5p84E0skkmJn038y+hWJbLQXQqEnQfeh5r2JLM=
-modernc.org/strutil v1.2.1 h1:UneZBkQA+DX2Rp35KcM69cSsNES9ly8mQWD71HKlOA0=
-modernc.org/strutil v1.2.1/go.mod h1:EHkiggD70koQxjVdSBM3JKM7k6L0FbGE5eymy9i3B9A=
-modernc.org/token v1.1.0 h1:Xl7Ap9dKaEs5kLoOQeQmPWevfnk/DM5qcLcYlA8ys6Y=
-modernc.org/token v1.1.0/go.mod h1:UGzOrNV1mAFSEB63lOFHIpNRUVMvYTc6yu1SMY/XTDM=

internal/api/alertmanager.go

@@ -94,27 +94,6 @@ func handleIntegrationWebhook(db *sql.DB, notify NotifyConfig) http.HandlerFunc
 	}
 }
 
-// handleLegacyWebhook is the pre-teams unauthenticated endpoint, kept for one
-// release so an upgrade does not silently stop delivering while somebody edits
-// the Alertmanager config. It routes to the oldest team, which on an upgraded
-// install is the Default team everything was moved into.
-//
-// It is deprecated and unauthenticated — anything that can reach the port can
-// open an incident. Move senders to an integration key and this goes away.
-func handleLegacyWebhook(db *sql.DB, notify NotifyConfig) http.HandlerFunc {
-	return func(w http.ResponseWriter, r *http.Request) {
-		teamID, err := defaultTeamID(r.Context(), db)
-		if err != nil {
-			log.Printf("legacy webhook: no team to route to: %v", err)
-			w.WriteHeader(http.StatusOK)
-			return
-		}
-		log.Printf("legacy webhook: unauthenticated payload routed to team %d; "+
-			"move this sender to an integration key", teamID)
-		receiveWebhook(w, r, db, notify, teamID)
-	}
-}
-
 func receiveWebhook(w http.ResponseWriter, r *http.Request, db *sql.DB, notify NotifyConfig, teamID int64) {
 	var payload amPayload
 	if err := decodeJSON(r, &payload); err != nil {
@@ -424,12 +403,18 @@ func openIncident(ctx context.Context, q querier, notify NotifyConfig, teamID in
 		}
 	}
 
-	// Queue the page, but do not send it here: this runs inside a transaction on
+	// Queue the page, but do not send it here: this runs inside the webhook's
-	// a single-connection pool, so an HTTP call would hold up every other
+	// transaction, and an HTTP call would hold a connection open across a
-	// request. The notifier picks the row up within a tick.
+	// network round trip. The notifier picks the row up within a tick.
 	if err := enqueueOpened(ctx, q, notify, id, onCall); err != nil {
 		return 0, err
 	}
+
+	// And start the escalation clock, if the team keeps one. In the same
+	// transaction, so an incident is never briefly open with nobody counting.
+	if err := startEscalation(ctx, q, id, teamID); err != nil {
+		return 0, err
+	}
 	return id, nil
 }
 

internal/api/api_test.go

@@ -22,6 +22,10 @@ import (
 type ts struct {
 	*httptest.Server
 	key string
+	// ingestKey is an integration key for the default team: the only way in
+	// since the unauthenticated webhook was removed, so the tests exercise the
+	// same path production does.
+	ingestKey string
 	db        *sql.DB
 	notify    api.NotifyConfig
 	deadman   api.DeadmanConfig
@@ -48,7 +52,7 @@ func newDeadmanTS(t *testing.T, deadman api.DeadmanConfig, notify ...api.NotifyC
 	}
 
 	database := newTestDB(t)
-	srv := httptest.NewServer(api.NewRouter(database, cfg))
+	srv := httptest.NewServer(api.NewRouter(database, cfg, testConfig()))
 	t.Cleanup(srv.Close)
 
 	body, _ := json.Marshal(map[string]string{"username": "admin", "email": "admin@test.com"})
@@ -66,6 +70,16 @@ func newDeadmanTS(t *testing.T, deadman api.DeadmanConfig, notify ...api.NotifyC
 
 	s := &ts{Server: srv, key: key, db: database, notify: cfg, deadman: deadman}
 
+	var integration struct {
+		Key string `json:"key"`
+	}
+	decode(t, s.req(t, http.MethodPost, "/api/teams/"+defaultTeam+"/integrations",
+		map[string]string{"name": "test"}), &integration)
+	if integration.Key == "" {
+		t.Fatal("no integration key was returned")
+	}
+	s.ingestKey = integration.Key
+
 	// Dead man's switches belong to a team now, so a test that wants them
 	// configures the default team the way an owner would.
 	if deadman.Timeout > 0 {
@@ -234,7 +248,8 @@ func postWebhook(t *testing.T, s *ts, alerts []map[string]any, groupKey ...strin
 		}
 	}
 	data, _ := json.Marshal(payload)
-	resp, err := http.Post(s.URL+"/api/alertmanager/webhook", "application/json", bytes.NewReader(data))
+	resp, err := http.Post(s.URL+"/api/integrations/"+s.ingestKey+"/alertmanager",
+		"application/json", bytes.NewReader(data))
 	if err != nil {
 		t.Fatalf("post webhook: %v", err)
 	}

internal/api/archiver.go

@@ -19,6 +19,10 @@ const (
 
 // StartArchiver runs the alert sweeper until ctx is cancelled, starting with an
 // immediate pass so a restart reconciles state right away.
+// archiveAfter and staleAfter are the values the server started with. They are
+// the fallback, not the setting: each pass reads the current value from the
+// settings table, so an administrator's change takes effect on the next tick
+// instead of at the next restart.
 func StartArchiver(ctx context.Context, db *sql.DB, archiveAfter, staleAfter time.Duration, notify NotifyConfig) {
 	ticker := time.NewTicker(sweepInterval)
 	defer ticker.Stop()
@@ -45,6 +49,10 @@ func StartArchiver(ctx context.Context, db *sql.DB, archiveAfter, staleAfter tim
 // staleness rules would otherwise resolve it as 'expiry' long before that.
 // Exported so tests can drive a pass without waiting on the ticker.
 func Sweep(ctx context.Context, db *sql.DB, archiveAfter, staleAfter time.Duration, notify NotifyConfig) {
+	settings := NewSettings(db)
+	staleAfter = settings.Duration(ctx, SettingStaleAfter, staleAfter)
+	archiveAfter = settings.Duration(ctx, SettingArchiveAfter, archiveAfter)
+
 	heartbeats := sweepDeadman(ctx, db, notify)
 	expireStale(ctx, db, staleAfter, heartbeats)
 	resolveSettledIncidents(ctx, db)

internal/api/auth_test.go

@@ -301,7 +301,7 @@ func TestSetPassword_EndsOtherSessionsButNotThisOne(t *testing.T) {
 
 func TestBootstrap_WithPassword(t *testing.T) {
 	database := newTestDB(t)
-	srv := httptest.NewServer(api.NewRouter(database, api.NotifyConfig{}))
+	srv := httptest.NewServer(api.NewRouter(database, api.NotifyConfig{}, testConfig()))
 	t.Cleanup(srv.Close)
 
 	body := `{"username":"admin","email":"a@test.com","password":"` + adminPassword + `"}`

internal/api/escalation.go

@@ -0,0 +1,506 @@
+package api
+
+import (
+	"context"
+	"database/sql"
+	"log"
+	"net/http"
+	"strconv"
+	"strings"
+	"time"
+)
+
+// evEscalated records a rung of the ladder on the incident's timeline: which
+// level, and who it woke.
+const evEscalated = "escalated"
+
+// escalationPolicy is a team's ladder, loaded whole. It is small — a handful of
+// levels with a few targets each — and every use needs all of it, so there is
+// no point reading it a level at a time.
+type escalationPolicy struct {
+	teamID        int64
+	repeatCount   int64
+	fallbackTopic string
+	levels        []escalationLevel
+}
+
+type escalationLevel struct {
+	id       int64
+	position int64
+	timeout  time.Duration
+	targets  []escalationTarget
+}
+
+type escalationTarget struct {
+	kind   string // "user" or "oncall"
+	userID *int64
+}
+
+// configured reports whether this team has anything to escalate through. A
+// policy row with no levels is the same as no policy: the team gets the
+// pre-escalation behaviour, which is reminders on the assignee's topic.
+func (p *escalationPolicy) configured() bool { return p != nil && len(p.levels) > 0 }
+
+// level returns the level at a 1-based position.
+func (p *escalationPolicy) level(pos int64) (escalationLevel, bool) {
+	for _, l := range p.levels {
+		if l.position == pos {
+			return l, true
+		}
+	}
+	return escalationLevel{}, false
+}
+
+// loadEscalationPolicy reads one team's ladder. A team with no policy row
+// returns nil, which every caller treats as "not configured" rather than as an
+// error: most teams will never set one up.
+func loadEscalationPolicy(ctx context.Context, q querier, teamID int64) (*escalationPolicy, error) {
+	p := &escalationPolicy{teamID: teamID}
+	err := q.QueryRowContext(ctx,
+		"SELECT repeat_count, fallback_topic FROM escalation_policies WHERE team_id = $1",
+		teamID).Scan(&p.repeatCount, &p.fallbackTopic)
+	if err == sql.ErrNoRows {
+		return nil, nil
+	}
+	if err != nil {
+		return nil, err
+	}
+
+	rows, err := q.QueryContext(ctx, `
+		SELECT l.id, l.position, l.timeout_seconds, t.kind, t.user_id
+		FROM   escalation_levels l
+		LEFT   JOIN escalation_targets t ON t.level_id = l.id
+		WHERE  l.team_id = $1
+		ORDER  BY l.position, t.id`, teamID)
+	if err != nil {
+		return nil, err
+	}
+	defer rows.Close()
+
+	byPosition := map[int64]int{} // position -> index in p.levels
+	for rows.Next() {
+		var id, position, timeout int64
+		var kind *string
+		var userID *int64
+		if err := rows.Scan(&id, &position, &timeout, &kind, &userID); err != nil {
+			return nil, err
+		}
+		idx, seen := byPosition[position]
+		if !seen {
+			p.levels = append(p.levels, escalationLevel{
+				id:       id,
+				position: position,
+				timeout:  time.Duration(timeout) * time.Second,
+			})
+			idx = len(p.levels) - 1
+			byPosition[position] = idx
+		}
+		// LEFT JOIN: a level with no targets yet still produces a row, with a
+		// NULL kind. It is a rung that pages nobody, which the API refuses to
+		// store but an older row could still hold.
+		if kind != nil {
+			p.levels[idx].targets = append(p.levels[idx].targets,
+				escalationTarget{kind: *kind, userID: userID})
+		}
+	}
+	return p, rows.Err()
+}
+
+// escalate advances every incident whose current level has run out of time.
+//
+// Runs on the notifier's tick, beside the reminder pass, because it is the same
+// question asked differently: reminders ask "has this been ignored long
+// enough to say it again", escalation asks "long enough to say it to somebody
+// else". Sharing the tick means one query cadence and one outbox.
+func escalate(ctx context.Context, db *sql.DB, cfg NotifyConfig) {
+	rows, err := db.QueryContext(ctx, `
+		SELECT i.id, i.team_id, i.escalation_level, i.escalation_level_at, i.escalation_round
+		FROM   incidents i
+		JOIN   escalation_policies p ON p.team_id = i.team_id
+		WHERE  i.resolved_at IS NULL
+		  AND  i.archived_at IS NULL
+		  AND  i.status = 'triggered'
+		  AND  (i.snoozed_until IS NULL OR i.snoozed_until <= $1)
+		  AND  i.escalation_level > 0`, time.Now().Unix())
+	if err != nil {
+		log.Printf("escalation: find due: %v", err)
+		return
+	}
+
+	type pending struct {
+		incidentID, teamID, level, round int64
+		levelAt                          int64
+	}
+	var due []pending
+	for rows.Next() {
+		var p pending
+		var levelAt *int64
+		if err := rows.Scan(&p.incidentID, &p.teamID, &p.level, &levelAt, &p.round); err != nil {
+			rows.Close()
+			log.Printf("escalation: scan: %v", err)
+			return
+		}
+		if levelAt == nil {
+			continue
+		}
+		p.levelAt = *levelAt
+		due = append(due, p)
+	}
+	rows.Close()
+	if err := rows.Err(); err != nil {
+		log.Printf("escalation: iterate: %v", err)
+		return
+	}
+
+	now := time.Now()
+	for _, d := range due {
+		policy, err := loadEscalationPolicy(ctx, db, d.teamID)
+		if err != nil {
+			log.Printf("escalation: load policy for team %d: %v", d.teamID, err)
+			continue
+		}
+		if !policy.configured() {
+			continue
+		}
+		current, ok := policy.level(d.level)
+		if !ok {
+			continue
+		}
+		if now.Sub(time.Unix(d.levelAt, 0)) < current.timeout {
+			continue
+		}
+		if err := advanceEscalation(ctx, db, cfg, policy, d.incidentID, d.level, d.round, now); err != nil {
+			log.Printf("escalation: advance incident %d: %v", d.incidentID, err)
+		}
+	}
+}
+
+// advanceEscalation moves one incident to its next rung, or off the end of the
+// ladder.
+//
+// The whole move is one transaction: the level, the page and the timeline entry
+// are one event, and an incident recorded as being at level 3 that nobody at
+// level 3 was told about is the worst of the possible half-states.
+func advanceEscalation(ctx context.Context, db *sql.DB, cfg NotifyConfig, policy *escalationPolicy, incidentID, level, round int64, now time.Time) error {
+	tx, err := db.BeginTx(ctx, nil)
+	if err != nil {
+		return err
+	}
+	defer tx.Rollback() //nolint:errcheck
+
+	next := level + 1
+	nextRound := round
+	if _, ok := policy.level(next); !ok {
+		// Off the end. Either start the chain again, or make the last call.
+		if round < policy.repeatCount {
+			next, nextRound = 1, round+1
+		} else {
+			if err := escalationExhausted(ctx, tx, policy, incidentID, now); err != nil {
+				return err
+			}
+			return tx.Commit()
+		}
+	}
+
+	target, ok := policy.level(next)
+	if !ok {
+		return nil
+	}
+	paged, err := pageLevel(ctx, tx, cfg, policy, incidentID, target)
+	if err != nil {
+		return err
+	}
+
+	if _, err := tx.ExecContext(ctx, `
+		UPDATE incidents
+		   SET escalation_level = $1, escalation_level_at = $2, escalation_round = $3
+		 WHERE id = $4`, next, now.Unix(), nextRound, incidentID); err != nil {
+		return err
+	}
+
+	detail := "level " + strconv.FormatInt(next, 10)
+	if nextRound > round {
+		detail += " (round " + strconv.FormatInt(nextRound+1, 10) + ")"
+	}
+	if len(paged) > 0 {
+		detail += ": " + strings.Join(paged, ", ")
+	} else {
+		// Worth recording loudly: the rung exists, its turn came, and it woke
+		// nobody. That is a policy that looks configured and is not.
+		detail += ": nobody reachable"
+	}
+	if err := logEvent(ctx, tx, incidentID, evEscalated, nil, nil, &detail); err != nil {
+		return err
+	}
+	return tx.Commit()
+}
+
+// escalationExhausted is the end of the line: the fallback topic, once, and a
+// timeline entry saying the ladder is finished. The incident stays triggered —
+// escalation running out is not the same as somebody answering.
+func escalationExhausted(ctx context.Context, tx *sql.Tx, policy *escalationPolicy, incidentID int64, now time.Time) error {
+	detail := "escalation exhausted"
+	if policy.fallbackTopic != "" {
+		if err := enqueueNotification(ctx, tx, incidentID, nil, policy.fallbackTopic, notifyEscalated); err != nil {
+			return err
+		}
+		detail += ": paged " + policy.fallbackTopic
+	} else {
+		detail += ": no fallback topic configured"
+	}
+
+	// Level 0 again, so the sweep stops considering it. The round counter is
+	// left where it is, as the record of how far it got.
+	if _, err := tx.ExecContext(ctx,
+		"UPDATE incidents SET escalation_level = 0, escalation_level_at = NULL WHERE id = $1",
+		incidentID); err != nil {
+		return err
+	}
+	return logEvent(ctx, tx, incidentID, evEscalated, nil, nil, &detail)
+}
+
+// pageLevel notifies every target of one level and reports who was woken.
+//
+// Each target gets its own outbox row, so each gets its own Acknowledge token:
+// the button in a notification must acknowledge as the person holding the
+// phone, not as whoever was paged first.
+func pageLevel(ctx context.Context, tx *sql.Tx, cfg NotifyConfig, policy *escalationPolicy, incidentID int64, level escalationLevel) ([]string, error) {
+	var paged []string
+	seen := map[int64]bool{}
+
+	for _, t := range level.targets {
+		userID := t.userID
+		if t.kind == "oncall" {
+			onCall, err := currentOnCall(ctx, tx, policy.teamID)
+			if err != nil {
+				return nil, err
+			}
+			if onCall == nil {
+				continue
+			}
+			userID = onCall
+		}
+		if userID == nil || seen[*userID] {
+			continue
+		}
+		seen[*userID] = true
+
+		var topic *string
+		var username string
+		if err := tx.QueryRowContext(ctx,
+			"SELECT ntfy_topic, username FROM users WHERE id = $1 AND disabled_at IS NULL",
+			*userID).Scan(&topic, &username); err != nil {
+			// A disabled or deleted account is not an error in the middle of an
+			// escalation: it is a target that cannot be woken, and the next
+			// level is the answer to that.
+			continue
+		}
+		if topic == nil || *topic == "" {
+			continue
+		}
+		if err := enqueueNotification(ctx, tx, incidentID, userID, *topic, notifyEscalated); err != nil {
+			return nil, err
+		}
+		paged = append(paged, username)
+	}
+	return paged, nil
+}
+
+// startEscalation puts a newly opened incident on the first rung, when its team
+// has a ladder. Called from openIncident, inside the same transaction, so an
+// incident is never briefly open with no escalation clock running.
+func startEscalation(ctx context.Context, q querier, incidentID, teamID int64) error {
+	policy, err := loadEscalationPolicy(ctx, q, teamID)
+	if err != nil || !policy.configured() {
+		return err
+	}
+	_, err = q.ExecContext(ctx,
+		"UPDATE incidents SET escalation_level = 1, escalation_level_at = $1 WHERE id = $2",
+		time.Now().Unix(), incidentID)
+	return err
+}
+
+// stopEscalation takes an incident off the ladder. Acknowledging or resolving
+// is somebody saying "I have this", and continuing to wake people after that is
+// the behaviour that teaches people to ignore the tool.
+func stopEscalation(ctx context.Context, q querier, incidentID int64) error {
+	_, err := q.ExecContext(ctx,
+		"UPDATE incidents SET escalation_level = 0, escalation_level_at = NULL WHERE id = $1",
+		incidentID)
+	return err
+}
+
+// handleGetEscalation returns a team's ladder.
+func handleGetEscalation(db *sql.DB) http.HandlerFunc {
+	return func(w http.ResponseWriter, r *http.Request) {
+		teamID, ok := teamParam(w, r)
+		if !ok {
+			return
+		}
+		if !requireTeamMember(w, r, teamID) {
+			return
+		}
+
+		policy, err := loadEscalationPolicy(r.Context(), db, teamID)
+		if err != nil {
+			respond(w, http.StatusInternalServerError, errResp("internal error"))
+			return
+		}
+		respond(w, http.StatusOK, escalationResponse(policy, teamID))
+	}
+}
+
+type escalationLevelJSON struct {
+	Position       int64                  `json:"position"`
+	TimeoutSeconds int64                  `json:"timeout_seconds"`
+	Targets        []escalationTargetJSON `json:"targets"`
+}
+
+type escalationTargetJSON struct {
+	Kind   string `json:"kind"`
+	UserID *int64 `json:"user_id,omitempty"`
+}
+
+type escalationJSON struct {
+	TeamID        int64                 `json:"team_id"`
+	RepeatCount   int64                 `json:"repeat_count"`
+	FallbackTopic string                `json:"fallback_topic"`
+	Levels        []escalationLevelJSON `json:"levels"`
+}
+
+func escalationResponse(p *escalationPolicy, teamID int64) escalationJSON {
+	out := escalationJSON{TeamID: teamID, Levels: []escalationLevelJSON{}}
+	if p == nil {
+		return out
+	}
+	out.RepeatCount = p.repeatCount
+	out.FallbackTopic = p.fallbackTopic
+	for _, l := range p.levels {
+		level := escalationLevelJSON{
+			Position:       l.position,
+			TimeoutSeconds: int64(l.timeout.Seconds()),
+			Targets:        []escalationTargetJSON{},
+		}
+		for _, t := range l.targets {
+			level.Targets = append(level.Targets, escalationTargetJSON{Kind: t.kind, UserID: t.userID})
+		}
+		out.Levels = append(out.Levels, level)
+	}
+	return out
+}
+
+// handleSetEscalation replaces a team's ladder wholesale.
+//
+// Replace rather than patch: the levels are an order, and an API that edits one
+// rung has to answer what happens to the numbering of the others. Sending the
+// whole ladder makes the order the client's to decide and the server's to
+// store, and makes an edit atomic — there is no moment where level 2 exists
+// twice.
+func handleSetEscalation(db *sql.DB) http.HandlerFunc {
+	return func(w http.ResponseWriter, r *http.Request) {
+		teamID, ok := teamParam(w, r)
+		if !ok {
+			return
+		}
+		if !requireTeamOwner(w, r, teamID) {
+			return
+		}
+
+		var req escalationJSON
+		if err := decodeJSON(r, &req); err != nil {
+			respond(w, http.StatusBadRequest, errResp("invalid request body"))
+			return
+		}
+		if req.RepeatCount < 0 || req.RepeatCount > 10 {
+			respond(w, http.StatusBadRequest, errResp("repeat_count must be between 0 and 10"))
+			return
+		}
+		for i, l := range req.Levels {
+			if l.TimeoutSeconds <= 0 {
+				respond(w, http.StatusBadRequest, errResp("every level needs a timeout"))
+				return
+			}
+			if len(l.Targets) == 0 {
+				// A rung that pages nobody is not a delay, it is a silence with
+				// a number on it.
+				respond(w, http.StatusBadRequest,
+					errResp("level "+strconv.FormatInt(int64(i+1), 10)+" has no targets"))
+				return
+			}
+			for _, t := range l.Targets {
+				switch t.Kind {
+				case "oncall":
+					if t.UserID != nil {
+						respond(w, http.StatusBadRequest, errResp("an oncall target takes no user_id"))
+						return
+					}
+				case "user":
+					if t.UserID == nil {
+						respond(w, http.StatusBadRequest, errResp("a user target needs a user_id"))
+						return
+					}
+				default:
+					respond(w, http.StatusBadRequest, errResp("target kind must be user or oncall"))
+					return
+				}
+			}
+		}
+
+		tx, err := db.BeginTx(r.Context(), nil)
+		if err != nil {
+			respond(w, http.StatusInternalServerError, errResp("internal error"))
+			return
+		}
+		defer tx.Rollback() //nolint:errcheck
+
+		if _, err := tx.ExecContext(r.Context(), `
+			INSERT INTO escalation_policies (team_id, repeat_count, fallback_topic, updated_at)
+			VALUES ($1, $2, $3, `+nowEpoch+`)
+			ON CONFLICT (team_id) DO UPDATE SET
+				repeat_count = excluded.repeat_count,
+				fallback_topic = excluded.fallback_topic,
+				updated_at = excluded.updated_at`,
+			teamID, req.RepeatCount, req.FallbackTopic); err != nil {
+			respond(w, http.StatusInternalServerError, errResp("internal error"))
+			return
+		}
+		// The levels are replaced, not merged; the cascade takes the targets.
+		if _, err := tx.ExecContext(r.Context(),
+			"DELETE FROM escalation_levels WHERE team_id = $1", teamID); err != nil {
+			respond(w, http.StatusInternalServerError, errResp("internal error"))
+			return
+		}
+
+		for i, l := range req.Levels {
+			var levelID int64
+			if err := tx.QueryRowContext(r.Context(), `
+				INSERT INTO escalation_levels (team_id, position, timeout_seconds)
+				VALUES ($1, $2, $3) RETURNING id`,
+				teamID, int64(i+1), l.TimeoutSeconds).Scan(&levelID); err != nil {
+				respond(w, http.StatusInternalServerError, errResp("internal error"))
+				return
+			}
+			for _, t := range l.Targets {
+				if _, err := tx.ExecContext(r.Context(), `
+					INSERT INTO escalation_targets (level_id, kind, user_id)
+					VALUES ($1, $2, $3)`, levelID, t.Kind, t.UserID); err != nil {
+					// The only foreign key here is the user.
+					respond(w, http.StatusBadRequest, errResp("unknown user in targets"))
+					return
+				}
+			}
+		}
+
+		if err := tx.Commit(); err != nil {
+			respond(w, http.StatusInternalServerError, errResp("internal error"))
+			return
+		}
+
+		policy, err := loadEscalationPolicy(r.Context(), db, teamID)
+		if err != nil {
+			respond(w, http.StatusInternalServerError, errResp("internal error"))
+			return
+		}
+		respond(w, http.StatusOK, escalationResponse(policy, teamID))
+	}
+}

internal/api/escalation_test.go

@@ -0,0 +1,385 @@
+package api_test
+
+import (
+	"net/http"
+	"strings"
+	"testing"
+	"time"
+
+	"git.ryuvia.com/niklas/terdut-server/internal/api"
+)
+
+// teamUser creates a user in the default team with an ntfy topic, so they can
+// actually be paged.
+func teamUser(t *testing.T, s *ts, username, topic string) int64 {
+	t.Helper()
+	var user struct {
+		ID int64 `json:"id"`
+	}
+	decode(t, s.req(t, http.MethodPost, "/api/users",
+		map[string]string{"username": username, "email": username + "@test.com"}), &user)
+	resp := s.req(t, http.MethodPost, "/api/teams/"+defaultTeam+"/members",
+		map[string]any{"user_id": user.ID, "role": "member"})
+	resp.Body.Close()
+	setTopic(t, s, int(user.ID), topic)
+	return user.ID
+}
+
+// Escalation is all timeouts, and there is no fake clock in this package. The
+// tests back-date escalation_level_at instead, which is the same trick the dead
+// man's switch tests use on received_at: the sweeper reads a stored timestamp,
+// so moving the timestamp is moving the clock.
+
+// ladder configures the default team with two levels: the rota first, then a
+// named person, then the fallback topic.
+func ladder(t *testing.T, s *ts, secondUserID int64, repeat int64, fallback string) {
+	t.Helper()
+	resp := s.req(t, http.MethodPut, "/api/teams/"+defaultTeam+"/escalation", map[string]any{
+		"repeat_count":   repeat,
+		"fallback_topic": fallback,
+		"levels": []map[string]any{
+			{"timeout_seconds": 300, "targets": []map[string]any{{"kind": "oncall"}}},
+			{"timeout_seconds": 300, "targets": []map[string]any{{"kind": "user", "user_id": secondUserID}}},
+		},
+	})
+	defer resp.Body.Close()
+	if resp.StatusCode != http.StatusOK {
+		t.Fatalf("configure the ladder: %d", resp.StatusCode)
+	}
+}
+
+// overdue back-dates an incident's current level so its timeout has passed.
+func overdue(t *testing.T, s *ts, incidentID int64) {
+	t.Helper()
+	s.exec(t, "UPDATE incidents SET escalation_level_at = $1 WHERE id = $2",
+		time.Now().Add(-time.Hour).Unix(), incidentID)
+}
+
+func escalationLevel(t *testing.T, s *ts, incidentID int64) (level, round int64) {
+	t.Helper()
+	if err := s.db.QueryRow(
+		"SELECT escalation_level, escalation_round FROM incidents WHERE id = $1",
+		incidentID).Scan(&level, &round); err != nil {
+		t.Fatalf("read escalation state: %v", err)
+	}
+	return level, round
+}
+
+// The whole point: nobody answers, so somebody else is woken.
+func TestEscalation_PagesTheNextLevel(t *testing.T) {
+	s, f := notifyTS(t, api.NotifyConfig{PublicURL: "https://terdut.example.com", RepeatEvery: 15 * time.Minute})
+	second := teamUser(t, s, "second", "terdut-second")
+	ladder(t, s, second, 0, "terdut-fallback")
+
+	postWebhook(t, s, []map[string]any{
+		amAlert("fp-esc", "DiskFull", "firing", "2026-05-20T10:00:00Z", zeroTime, nil),
+	})
+	s.sweepNotify(t)
+
+	// Level 1 is the rota, so the first page went to the admin.
+	if level, _ := escalationLevel(t, s, 1); level != 1 {
+		t.Fatalf("a new incident should start at level 1, got %d", level)
+	}
+	if got := f.topicsSince(t); len(got) == 0 || got[0] != "terdut-admin" {
+		t.Fatalf("the first page should go to the on-call user, went to %v", got)
+	}
+
+	// Time passes with no acknowledgement.
+	f.forget()
+	overdue(t, s, 1)
+	s.sweepNotify(t)
+
+	if level, _ := escalationLevel(t, s, 1); level != 2 {
+		t.Errorf("expected level 2, got %d", level)
+	}
+	if got := f.topicsSince(t); len(got) != 1 || got[0] != "terdut-second" {
+		t.Errorf("level 2 should page the named user, paged %v", got)
+	}
+
+	// And the timeline says so, which is what somebody reads afterwards to
+	// understand why their phone rang at 04:00.
+	timeline := list(t, s.req(t, http.MethodGet, "/api/incidents/1/timeline", nil))
+	found := ""
+	for _, e := range timeline {
+		if e["type"] == "escalated" {
+			found, _ = e["detail"].(string)
+		}
+	}
+	if found == "" {
+		t.Error("the timeline should record the escalation")
+	} else if !strings.HasPrefix(found, "level 2") || !strings.Contains(found, "second") {
+		t.Errorf("the escalation entry should say which level and who: %q", found)
+	}
+}
+
+// Acknowledging is somebody saying "I have this". Nobody else should be woken.
+func TestEscalation_AcknowledgementStopsIt(t *testing.T) {
+	s, f := notifyTS(t, api.NotifyConfig{PublicURL: "https://terdut.example.com", RepeatEvery: 15 * time.Minute})
+	second := teamUser(t, s, "second", "terdut-second")
+	ladder(t, s, second, 0, "terdut-fallback")
+
+	postWebhook(t, s, []map[string]any{
+		amAlert("fp-ack", "DiskFull", "firing", "2026-05-20T10:00:00Z", zeroTime, nil),
+	})
+	s.sweepNotify(t)
+
+	s.req(t, http.MethodPost, "/api/incidents/1/acknowledge", nil).Body.Close()
+	if level, _ := escalationLevel(t, s, 1); level != 0 {
+		t.Errorf("acknowledging should take the incident off the ladder, level is %d", level)
+	}
+
+	f.forget()
+	overdue(t, s, 1) // no-op: level is 0, so there is nothing due
+	s.sweepNotify(t)
+	if got := f.topicsSince(t); len(got) != 0 {
+		t.Errorf("an acknowledged incident should page nobody, paged %v", got)
+	}
+}
+
+// Resolving stops it too, and by the same mechanism.
+func TestEscalation_ResolutionStopsIt(t *testing.T) {
+	s, f := notifyTS(t, api.NotifyConfig{PublicURL: "https://terdut.example.com", RepeatEvery: 15 * time.Minute})
+	second := teamUser(t, s, "second", "terdut-second")
+	ladder(t, s, second, 0, "terdut-fallback")
+
+	postWebhook(t, s, []map[string]any{
+		amAlert("fp-res", "DiskFull", "firing", "2026-05-20T10:00:00Z", zeroTime, nil),
+	})
+	s.sweepNotify(t)
+	s.req(t, http.MethodPost, "/api/incidents/1/resolve", nil).Body.Close()
+
+	f.forget()
+	overdue(t, s, 1)
+	s.sweepNotify(t)
+	if level, _ := escalationLevel(t, s, 1); level != 0 {
+		t.Errorf("a resolved incident should be off the ladder, level is %d", level)
+	}
+}
+
+// Snoozing is a deliberate "not now", so the ladder waits rather than carrying
+// on without the person who asked for quiet.
+func TestEscalation_SnoozePausesIt(t *testing.T) {
+	s, f := notifyTS(t, api.NotifyConfig{PublicURL: "https://terdut.example.com", RepeatEvery: 15 * time.Minute})
+	second := teamUser(t, s, "second", "terdut-second")
+	ladder(t, s, second, 0, "terdut-fallback")
+
+	postWebhook(t, s, []map[string]any{
+		amAlert("fp-snooze", "DiskFull", "firing", "2026-05-20T10:00:00Z", zeroTime, nil),
+	})
+	s.sweepNotify(t)
+
+	resp := s.req(t, http.MethodPost, "/api/incidents/1/snooze", map[string]any{"duration": "1h"})
+	resp.Body.Close()
+
+	f.forget()
+	overdue(t, s, 1)
+	s.sweepNotify(t)
+
+	if level, _ := escalationLevel(t, s, 1); level != 1 {
+		t.Errorf("a snoozed incident should stay where it is, level is %d", level)
+	}
+	if got := f.topicsSince(t); len(got) != 0 {
+		t.Errorf("a snoozed incident should page nobody, paged %v", got)
+	}
+
+	// When the snooze ends, the ladder picks up where it left off.
+	s.exec(t, "UPDATE incidents SET snoozed_until = $1 WHERE id = 1", time.Now().Add(-time.Minute).Unix())
+	s.sweepNotify(t)
+	if level, _ := escalationLevel(t, s, 1); level != 2 {
+		t.Errorf("after the snooze the ladder should resume, level is %d", level)
+	}
+}
+
+// Running out of ladder pages the team's fallback topic once, and says so.
+func TestEscalation_ExhaustionPagesTheFallback(t *testing.T) {
+	s, f := notifyTS(t, api.NotifyConfig{PublicURL: "https://terdut.example.com", RepeatEvery: 15 * time.Minute})
+	second := teamUser(t, s, "second", "terdut-second")
+	ladder(t, s, second, 0, "terdut-fallback")
+
+	postWebhook(t, s, []map[string]any{
+		amAlert("fp-end", "DiskFull", "firing", "2026-05-20T10:00:00Z", zeroTime, nil),
+	})
+	s.sweepNotify(t)
+
+	overdue(t, s, 1)
+	s.sweepNotify(t) // level 2
+	f.forget()
+	overdue(t, s, 1)
+	s.sweepNotify(t) // off the end
+
+	if got := f.topicsSince(t); len(got) != 1 || got[0] != "terdut-fallback" {
+		t.Errorf("exhaustion should page the fallback topic once, paged %v", got)
+	}
+	level, _ := escalationLevel(t, s, 1)
+	if level != 0 {
+		t.Errorf("an exhausted ladder should stop asking, level is %d", level)
+	}
+
+	// The incident is still open: running out of people is not an answer.
+	var status string
+	if err := s.db.QueryRow("SELECT status FROM incidents WHERE id = 1").Scan(&status); err != nil {
+		t.Fatal(err)
+	}
+	if status != "triggered" {
+		t.Errorf("exhaustion must not resolve the incident, status is %q", status)
+	}
+}
+
+// repeat_count walks the whole ladder again before giving up.
+func TestEscalation_RepeatsTheChain(t *testing.T) {
+	s, f := notifyTS(t, api.NotifyConfig{PublicURL: "https://terdut.example.com", RepeatEvery: 15 * time.Minute})
+	second := teamUser(t, s, "second", "terdut-second")
+	ladder(t, s, second, 1, "terdut-fallback") // one extra round
+
+	postWebhook(t, s, []map[string]any{
+		amAlert("fp-repeat", "DiskFull", "firing", "2026-05-20T10:00:00Z", zeroTime, nil),
+	})
+	s.sweepNotify(t)
+
+	overdue(t, s, 1)
+	s.sweepNotify(t) // level 2
+	f.forget()
+	overdue(t, s, 1)
+	s.sweepNotify(t) // back to level 1, round 2
+
+	level, round := escalationLevel(t, s, 1)
+	if level != 1 || round != 1 {
+		t.Errorf("expected level 1 round 1, got level %d round %d", level, round)
+	}
+	if got := f.topicsSince(t); len(got) != 1 || got[0] != "terdut-admin" {
+		t.Errorf("the second round should start at the top again, paged %v", got)
+	}
+}
+
+// A team without a ladder keeps exactly the behaviour it had, and never gets
+// both a reminder and an escalation for the same silence.
+func TestEscalation_WithoutAPolicyRemindersStillRun(t *testing.T) {
+	s, f := notifyTS(t, api.NotifyConfig{PublicURL: "https://terdut.example.com", RepeatEvery: 15 * time.Minute})
+	postWebhook(t, s, []map[string]any{
+		amAlert("fp-noesc", "DiskFull", "firing", "2026-05-20T10:00:00Z", zeroTime, nil),
+	})
+	s.sweepNotify(t)
+
+	// Age the first notification past the repeat interval.
+	f.forget()
+	s.exec(t, "UPDATE notifications SET created_at = $1, sent_at = $1",
+		time.Now().Add(-time.Hour).Unix())
+	s.sweepNotify(t)
+
+	if got := f.topicsSince(t); len(got) != 1 || got[0] != "terdut-admin" {
+		t.Errorf("without a ladder the reminder should still fire, paged %v", got)
+	}
+	if level, _ := escalationLevel(t, s, 1); level != 0 {
+		t.Errorf("an incident in a team with no ladder should not be on one, level is %d", level)
+	}
+}
+
+// With a ladder, reminders stop: two pages for one silence is how people learn
+// to mute the tool.
+func TestEscalation_WithAPolicyRemindersDoNotAlsoFire(t *testing.T) {
+	s, f := notifyTS(t, api.NotifyConfig{PublicURL: "https://terdut.example.com", RepeatEvery: 15 * time.Minute})
+	second := teamUser(t, s, "second", "terdut-second")
+	ladder(t, s, second, 0, "terdut-fallback")
+
+	postWebhook(t, s, []map[string]any{
+		amAlert("fp-both", "DiskFull", "firing", "2026-05-20T10:00:00Z", zeroTime, nil),
+	})
+	s.sweepNotify(t)
+
+	f.forget()
+	// Old enough for a reminder, but not yet due for escalation.
+	s.exec(t, "UPDATE notifications SET created_at = $1, sent_at = $1",
+		time.Now().Add(-time.Hour).Unix())
+	s.sweepNotify(t)
+
+	if got := f.topicsSince(t); len(got) != 0 {
+		t.Errorf("a team with a ladder should not also get reminders, paged %v", got)
+	}
+}
+
+// The API refuses a ladder that cannot page anybody.
+func TestEscalation_RejectsAnUnusablePolicy(t *testing.T) {
+	s := newTS(t)
+
+	for _, c := range []struct {
+		name string
+		body map[string]any
+	}{
+		{"a level with no targets", map[string]any{
+			"levels": []map[string]any{{"timeout_seconds": 300, "targets": []map[string]any{}}},
+		}},
+		{"a level with no timeout", map[string]any{
+			"levels": []map[string]any{{"timeout_seconds": 0, "targets": []map[string]any{{"kind": "oncall"}}}},
+		}},
+		{"a user target with no user", map[string]any{
+			"levels": []map[string]any{{"timeout_seconds": 300, "targets": []map[string]any{{"kind": "user"}}}},
+		}},
+		{"an unknown target kind", map[string]any{
+			"levels": []map[string]any{{"timeout_seconds": 300, "targets": []map[string]any{{"kind": "everybody"}}}},
+		}},
+		{"an absurd repeat count", map[string]any{
+			"repeat_count": 99,
+			"levels":       []map[string]any{{"timeout_seconds": 300, "targets": []map[string]any{{"kind": "oncall"}}}},
+		}},
+	} {
+		resp := s.req(t, http.MethodPut, "/api/teams/"+defaultTeam+"/escalation", c.body)
+		resp.Body.Close()
+		if resp.StatusCode != http.StatusBadRequest {
+			t.Errorf("%s: expected 400, got %d", c.name, resp.StatusCode)
+		}
+	}
+}
+
+// Editing the ladder is an owner's job; reading it is any member's.
+func TestEscalation_OwnerOnlyToEdit(t *testing.T) {
+	s := newTS(t)
+	_, call := member(t, s, "plain")
+
+	resp := call(http.MethodPut, "/api/teams/"+defaultTeam+"/escalation", map[string]any{
+		"levels": []map[string]any{{"timeout_seconds": 300, "targets": []map[string]any{{"kind": "oncall"}}}},
+	})
+	resp.Body.Close()
+	if resp.StatusCode != http.StatusForbidden {
+		t.Errorf("a member editing the ladder: expected 403, got %d", resp.StatusCode)
+	}
+
+	resp = call(http.MethodGet, "/api/teams/"+defaultTeam+"/escalation", nil)
+	resp.Body.Close()
+	if resp.StatusCode != http.StatusOK {
+		t.Errorf("a member reading the ladder: expected 200, got %d", resp.StatusCode)
+	}
+}
+
+// A target who cannot be woken is not a reason to stop: the next level is the
+// answer to an unreachable one.
+func TestEscalation_SkipsUnreachableTargets(t *testing.T) {
+	s, f := notifyTS(t, api.NotifyConfig{PublicURL: "https://terdut.example.com", RepeatEvery: 15 * time.Minute})
+	// Second user has no ntfy topic at all.
+	var user struct {
+		ID int64 `json:"id"`
+	}
+	decode(t, s.req(t, http.MethodPost, "/api/users",
+		map[string]string{"username": "silent", "email": "silent@test.com"}), &user)
+	s.req(t, http.MethodPost, "/api/teams/"+defaultTeam+"/members",
+		map[string]any{"user_id": user.ID, "role": "member"}).Body.Close()
+
+	ladder(t, s, user.ID, 0, "terdut-fallback")
+
+	postWebhook(t, s, []map[string]any{
+		amAlert("fp-silent", "DiskFull", "firing", "2026-05-20T10:00:00Z", zeroTime, nil),
+	})
+	s.sweepNotify(t)
+
+	f.forget()
+	overdue(t, s, 1)
+	s.sweepNotify(t)
+
+	// Level 2 was entered even though it woke nobody, so the ladder keeps
+	// moving toward the fallback rather than stalling on a silent rung.
+	if level, _ := escalationLevel(t, s, 1); level != 2 {
+		t.Errorf("expected the ladder to advance past an unreachable target, level is %d", level)
+	}
+	if got := f.topicsSince(t); len(got) != 0 {
+		t.Errorf("a target with no topic should page nothing, paged %v", got)
+	}
+}

internal/api/incident_store.go

@@ -235,6 +235,9 @@ func resolveIfSettled(ctx context.Context, q querier, incidentID int64) (bool, e
 	if n == 0 {
 		return false, nil
 	}
+	if err := stopEscalation(ctx, q, incidentID); err != nil {
+		return false, err
+	}
 	if err := logEvent(ctx, q, incidentID, evResolved, nil, nil, nil); err != nil {
 		return false, err
 	}
@@ -260,6 +263,10 @@ func acknowledgeIncident(ctx context.Context, q querier, incidentID, userID int6
 	if n, _ := res.RowsAffected(); n == 0 {
 		return false, nil
 	}
+	// Somebody has it: stop waking anybody else.
+	if err := stopEscalation(ctx, q, incidentID); err != nil {
+		return false, err
+	}
 	return true, logEvent(ctx, q, incidentID, evAcknowledged, &userID, nil, nil)
 }
 

internal/api/incidents.go

@@ -243,6 +243,11 @@ func handleIncidentResolve(db *sql.DB) http.HandlerFunc {
 			time.Now().Unix(), incidentResolutionManual, id) {
 			return
 		}
+		// A person closing an incident is the clearest possible "I have this".
+		if err := stopEscalation(r.Context(), db, id); err != nil {
+			respond(w, http.StatusInternalServerError, errResp("internal error"))
+			return
+		}
 		if err := logEvent(r.Context(), db, id, evResolved, &user.ID, nil, nil); err != nil {
 			respond(w, http.StatusInternalServerError, errResp("internal error"))
 			return

internal/api/middleware.go

@@ -144,8 +144,12 @@ func sessionUser(ctx context.Context, db *sql.DB, token string) (sessionID, user
 func serveAs(w http.ResponseWriter, r *http.Request, next http.Handler, db *sql.DB, userID, sessionID int64) {
 	var u models.User
 	var createdUnix int64
+	// disabled_at IS NULL is part of the lookup rather than a check afterwards:
+	// a disabled account is one that cannot authenticate, by either credential,
+	// and the way to be sure of that is for there to be no path where the row
+	// is loaded and the flag is then forgotten.
 	if err := db.QueryRowContext(r.Context(),
-		"SELECT id, username, email, created_at, is_admin FROM users WHERE id = $1", userID,
+		"SELECT id, username, email, created_at, is_admin FROM users WHERE id = $1 AND disabled_at IS NULL", userID,
 	).Scan(&u.ID, &u.Username, &u.Email, &createdUnix, &u.IsAdmin); err != nil {
 		respond(w, http.StatusUnauthorized, errResp("unauthorized"))
 		return

internal/api/notifier.go

@@ -43,6 +43,10 @@ const (
 	notifyTriggered = "triggered"
 	notifyReminder  = "reminder"
 	notifyResolved  = "resolved"
+
+	// notifyEscalated is a page that went out because nobody answered the last
+	// one. Told apart from a reminder because it goes to somebody else.
+	notifyEscalated = "escalated"
 )
 
 // Timeline event types the notifier writes, so an incident's history says who
@@ -120,6 +124,9 @@ func StartNotifier(ctx context.Context, db *sql.DB, cfg NotifyConfig) {
 // Exported so tests can drive a pass without waiting on the ticker.
 func NotifySweep(ctx context.Context, db *sql.DB, cfg NotifyConfig) {
 	enqueueReminders(ctx, db, cfg)
+	// Escalation before delivery, so a level that comes due on this tick is
+	// paged on this tick rather than waiting for the next one.
+	escalate(ctx, db, cfg)
 	deliverPending(ctx, db, cfg)
 }
 
@@ -134,7 +141,11 @@ func NotifySweep(ctx context.Context, db *sql.DB, cfg NotifyConfig) {
 // queued, so an ntfy outage produces a retry backlog rather than a reminder
 // backlog that all lands at once when it comes back.
 func enqueueReminders(ctx context.Context, db *sql.DB, cfg NotifyConfig) {
-	if cfg.RepeatEvery <= 0 {
+	// cfg.RepeatEvery is what the server started with; the settings table is
+	// what it runs on. Read per tick, so an administrator lengthening the
+	// interval at 02:00 is obeyed at 02:00 and not at the next restart.
+	repeat := NewSettings(db).Duration(ctx, SettingNotifyRepeat, cfg.RepeatEvery)
+	if repeat <= 0 {
 		return
 	}
 	now := time.Now()
@@ -155,8 +166,13 @@ func enqueueReminders(ctx context.Context, db *sql.DB, cfg NotifyConfig) {
 		  AND  i.resolved_at IS NULL
 		  AND  i.archived_at IS NULL
 		  AND  i.status = 'triggered'
-		  AND  (i.snoozed_until IS NULL OR i.snoozed_until <= $2)`,
+		  AND  (i.snoozed_until IS NULL OR i.snoozed_until <= $2)
-		now.Add(-cfg.RepeatEvery).Unix(), now.Unix())
+		  -- A team with an escalation ladder gets escalation instead. Both
+		  -- would mean two pages for one silence, which is how people learn to
+		  -- mute a tool.
+		  AND  NOT EXISTS (
+		           SELECT 1 FROM escalation_levels el WHERE el.team_id = i.team_id)`,
+		now.Add(-repeat).Unix(), now.Unix())
 	if err != nil {
 		log.Printf("notifier: find reminders: %v", err)
 		return

internal/api/notify_test.go

@@ -69,6 +69,27 @@ func (f *fakeNtfy) messages() []pushed {
 	return append([]pushed(nil), f.got...)
 }
 
+// topicsSince lists the topics published to since the last forget, which is how
+// the escalation tests ask "who did this tick wake".
+func (f *fakeNtfy) topicsSince(t *testing.T) []string {
+	t.Helper()
+	f.mu.Lock()
+	defer f.mu.Unlock()
+	out := make([]string, 0, len(f.got))
+	for _, m := range f.got {
+		out = append(out, m.Topic)
+	}
+	return out
+}
+
+// forget drops what has been published so far, so the next assertion is about
+// this tick rather than the whole test.
+func (f *fakeNtfy) forget() {
+	f.mu.Lock()
+	defer f.mu.Unlock()
+	f.got = nil
+}
+
 func (f *fakeNtfy) failWith(status int) {
 	f.mu.Lock()
 	defer f.mu.Unlock()

internal/api/router.go

@@ -4,6 +4,7 @@ import (
 	"database/sql"
 	"net/http"
 
+	"git.ryuvia.com/niklas/terdut-server/internal/config"
 	"git.ryuvia.com/niklas/terdut-server/internal/web"
 	"github.com/go-chi/chi/v5"
 	"github.com/go-chi/chi/v5/middleware"
@@ -13,7 +14,7 @@ import (
 // the only handler that has to decide where a new incident's page goes; a zero
 // notify disables notifications. Dead man's switches are per team and read from
 // the database, so nothing about them is wired in here.
-func NewRouter(db *sql.DB, notify NotifyConfig) http.Handler {
+func NewRouter(db *sql.DB, notify NotifyConfig, cfg config.Config) http.Handler {
 	r := chi.NewRouter()
 	r.Use(middleware.Logger)
 	r.Use(middleware.Recoverer)
@@ -31,14 +32,13 @@ func NewRouter(db *sql.DB, notify NotifyConfig) http.Handler {
 
 	// Alert ingestion. The key in the path says both that the sender may post
 	// and which team the alerts belong to, which is why it needs no session.
+	//
+	// This is the only way in. The pre-teams /api/alertmanager/webhook, which
+	// took no credential at all, was removed in v0.13.0 once the cluster's
+	// Alertmanager had moved onto a key; a sender still posting there gets the
+	// JSON 404 every unknown /api path gets.
 	r.Post("/api/integrations/{key}/alertmanager", handleIntegrationWebhook(db, notify))
 
-	// DEPRECATED, and unauthenticated: anything that can reach the port can
-	// open an incident here. Kept for one release so an upgrade does not stop
-	// delivering while the Alertmanager config is edited; it routes everything
-	// to the oldest team. Remove it once senders carry a key.
-	r.Post("/api/alertmanager/webhook", handleLegacyWebhook(db, notify))
-
 	// Signing in to the web UI. Login trades a password for a session cookie,
 	// which AuthMiddleware accepts in place of an API key.
 	r.Post("/api/login", handleLogin(db, newLoginLimiter(), notify.PublicURL))
@@ -71,6 +71,13 @@ func NewRouter(db *sql.DB, notify NotifyConfig) http.Handler {
 			r.Post("/api/users", handleCreateUser(db))
 			r.Delete("/api/users/{id}", handleDeleteUser(db))
 			r.Put("/api/users/{id}/admin", handleSetAdmin(db))
+			r.Put("/api/users/{id}/disabled", handleSetUserDisabled(db))
+
+			// What exists on this server, and how it behaves. /api/teams
+			// answers "what am I in"; this one answers "what is there".
+			r.Get("/api/admin/teams", handleAdminListTeams(db))
+			r.Get("/api/admin/settings", handleGetSettings(db, cfg))
+			r.Put("/api/admin/settings", handleSetSettings(db))
 		})
 
 		// Alerts are read-only: they are Alertmanager's record, not a work
@@ -96,11 +103,16 @@ func NewRouter(db *sql.DB, notify NotifyConfig) http.Handler {
 		// Teams. A user sees the teams they belong to; an owner configures one.
 		r.Get("/api/teams", handleListTeams(db))
 		r.Post("/api/teams", handleCreateTeam(db))
+		r.Put("/api/teams/{teamID}", handleRenameTeam(db))
 		r.Delete("/api/teams/{teamID}", handleDeleteTeam(db))
 		r.Get("/api/teams/{teamID}/members", handleListTeamMembers(db))
 		r.Post("/api/teams/{teamID}/members", handleAddTeamMember(db))
 		r.Delete("/api/teams/{teamID}/members/{userID}", handleRemoveTeamMember(db))
 
+		// A team's escalation ladder: who is paged when nobody answers.
+		r.Get("/api/teams/{teamID}/escalation", handleGetEscalation(db))
+		r.Put("/api/teams/{teamID}/escalation", handleSetEscalation(db))
+
 		// A team's own dead man's switches: which of its alerts are heartbeats,
 		// and how long a silence has to last before somebody is paged.
 		r.Get("/api/teams/{teamID}/deadman", handleGetTeamDeadman(db))

internal/api/settings.go

@@ -0,0 +1,346 @@
+package api
+
+import (
+	"context"
+	"database/sql"
+	"errors"
+	"net/http"
+	"strconv"
+	"time"
+
+	"git.ryuvia.com/niklas/terdut-server/internal/config"
+	"github.com/go-chi/chi/v5"
+)
+
+// The settings an administrator can change at runtime. Each is behaviour rather
+// than infrastructure: what the server does, not where it is plugged in.
+//
+// The values are seconds, stored as text. A duration string would be friendlier
+// to read in psql and worse everywhere else — it can be stored unparseable, and
+// then the question is what a background loop should do at 02:00 with a
+// tuning knob it cannot understand.
+const (
+	SettingNotifyRepeat = "notify_repeat_seconds"
+	SettingStaleAfter   = "stale_after_seconds"
+	SettingArchiveAfter = "archive_after_seconds"
+)
+
+// settingBounds keeps an edit from producing a server that cannot work. The
+// ceilings are loose — they exist to catch a slipped decimal point, not to have
+// an opinion about anybody's rota.
+var settingBounds = map[string]struct {
+	min, max time.Duration
+	label    string
+}{
+	SettingNotifyRepeat: {0, 24 * time.Hour, "how long an incident may sit unacknowledged before it is paged again; 0 disables reminders"},
+	SettingStaleAfter:   {5 * time.Minute, 30 * 24 * time.Hour, "how long a firing alert may go without a refreshing webhook before the sweeper resolves it"},
+	SettingArchiveAfter: {time.Minute, 365 * 24 * time.Hour, "how long a resolved alert or incident stays in the default list"},
+}
+
+// Settings reads the runtime configuration. It holds no cache: the readers are
+// two background loops that tick every 30 seconds and 15 minutes, and handlers
+// that run once per request, so a query each time costs nothing measurable and
+// means an administrator's change takes effect on the next tick rather than at
+// the next restart.
+type Settings struct{ db *sql.DB }
+
+// NewSettings returns a reader over db.
+func NewSettings(db *sql.DB) *Settings { return &Settings{db: db} }
+
+// Duration reads one setting, falling back to def when the row is missing or
+// unreadable. A tuning knob is never worth failing a sweep over: the fallback
+// is the value the server started with.
+func (s *Settings) Duration(ctx context.Context, key string, def time.Duration) time.Duration {
+	var raw string
+	err := s.db.QueryRowContext(ctx, "SELECT value FROM settings WHERE key = $1", key).Scan(&raw)
+	if err != nil {
+		return def
+	}
+	secs, err := strconv.ParseInt(raw, 10, 64)
+	if err != nil {
+		return def
+	}
+	return time.Duration(secs) * time.Second
+}
+
+// SeedSettings writes each key from the server's environment configuration,
+// once. Never overwrites: after the first start the database owns these, and a
+// redeploy must not put a chart's default back over an administrator's edit —
+// the same rule as the per-team dead man's switches.
+func SeedSettings(ctx context.Context, db *sql.DB, cfg config.Config) error {
+	seeds := map[string]time.Duration{
+		SettingNotifyRepeat: cfg.NotifyRepeat,
+		SettingStaleAfter:   cfg.StaleAfter,
+		SettingArchiveAfter: cfg.ArchiveAfter,
+	}
+	for key, d := range seeds {
+		if _, err := db.ExecContext(ctx, `
+			INSERT INTO settings (key, value) VALUES ($1, $2)
+			ON CONFLICT (key) DO NOTHING`,
+			key, strconv.FormatInt(int64(d.Seconds()), 10)); err != nil {
+			return err
+		}
+	}
+	return nil
+}
+
+// settingsResponse is what the admin page renders. The environment half is
+// included and marked read-only, so somebody looking for the ntfy URL finds out
+// where it lives rather than concluding the server does not have one.
+type settingsResponse struct {
+	Editable map[string]settingValue `json:"editable"`
+	FromEnv  map[string]string       `json:"from_env"`
+}
+
+type settingValue struct {
+	Seconds     int64  `json:"seconds"`
+	Description string `json:"description"`
+	MinSeconds  int64  `json:"min_seconds"`
+	MaxSeconds  int64  `json:"max_seconds"`
+}
+
+func handleGetSettings(db *sql.DB, cfg config.Config) http.HandlerFunc {
+	settings := NewSettings(db)
+	return func(w http.ResponseWriter, r *http.Request) {
+		out := settingsResponse{
+			Editable: map[string]settingValue{},
+			FromEnv: map[string]string{
+				// Never the ntfy token or the DSN: both are credentials, and an
+				// admin page that renders them turns a browser tab into a place
+				// they leak from.
+				"ntfy_url":        cfg.NtfyURL,
+				"ntfy_configured": strconv.FormatBool(cfg.NtfyURL != ""),
+				"ntfy_token_set":  strconv.FormatBool(cfg.NtfyToken != ""),
+				"public_url":      cfg.PublicURL,
+				"listen_address":  cfg.Addr,
+			},
+		}
+		for key, b := range settingBounds {
+			def := map[string]time.Duration{
+				SettingNotifyRepeat: cfg.NotifyRepeat,
+				SettingStaleAfter:   cfg.StaleAfter,
+				SettingArchiveAfter: cfg.ArchiveAfter,
+			}[key]
+			out.Editable[key] = settingValue{
+				Seconds:     int64(settings.Duration(r.Context(), key, def).Seconds()),
+				Description: b.label,
+				MinSeconds:  int64(b.min.Seconds()),
+				MaxSeconds:  int64(b.max.Seconds()),
+			}
+		}
+		respond(w, http.StatusOK, out)
+	}
+}
+
+// handleSetSettings changes one or more settings. Unknown keys are refused
+// rather than stored: a typo that writes notify_repeat_second would otherwise
+// sit in the table looking like configuration and doing nothing.
+func handleSetSettings(db *sql.DB) http.HandlerFunc {
+	return func(w http.ResponseWriter, r *http.Request) {
+		var req map[string]int64
+		if err := decodeJSON(r, &req); err != nil {
+			respond(w, http.StatusBadRequest, errResp("invalid request body"))
+			return
+		}
+		if len(req) == 0 {
+			respond(w, http.StatusBadRequest, errResp("no settings given"))
+			return
+		}
+
+		for key, secs := range req {
+			b, known := settingBounds[key]
+			if !known {
+				respond(w, http.StatusBadRequest, errResp("unknown setting: "+key))
+				return
+			}
+			d := time.Duration(secs) * time.Second
+			if d < b.min || d > b.max {
+				respond(w, http.StatusBadRequest, errResp(
+					key+" must be between "+b.min.String()+" and "+b.max.String()))
+				return
+			}
+		}
+
+		tx, err := db.BeginTx(r.Context(), nil)
+		if err != nil {
+			respond(w, http.StatusInternalServerError, errResp("internal error"))
+			return
+		}
+		defer tx.Rollback() //nolint:errcheck
+
+		for key, secs := range req {
+			if _, err := tx.ExecContext(r.Context(), `
+				INSERT INTO settings (key, value, updated_at)
+				VALUES ($1, $2, `+nowEpoch+`)
+				ON CONFLICT (key) DO UPDATE SET
+					value = excluded.value, updated_at = excluded.updated_at`,
+				key, strconv.FormatInt(secs, 10)); err != nil {
+				respond(w, http.StatusInternalServerError, errResp("internal error"))
+				return
+			}
+		}
+		if err := tx.Commit(); err != nil {
+			respond(w, http.StatusInternalServerError, errResp("internal error"))
+			return
+		}
+
+		w.WriteHeader(http.StatusNoContent)
+	}
+}
+
+// handleAdminListTeams lists every team on the server, with its size. The
+// ordinary /api/teams answers "what am I in"; this one answers "what exists",
+// which only an administrator may ask.
+func handleAdminListTeams(db *sql.DB) http.HandlerFunc {
+	return func(w http.ResponseWriter, r *http.Request) {
+		rows, err := db.QueryContext(r.Context(), `
+			SELECT t.id, t.name, t.created_at,
+			       (SELECT COUNT(*) FROM team_members m WHERE m.team_id = t.id),
+			       (SELECT COUNT(*) FROM incidents i
+			         WHERE i.team_id = t.id AND i.resolved_at IS NULL)
+			FROM   teams t
+			ORDER  BY t.name`)
+		if err != nil {
+			respond(w, http.StatusInternalServerError, errResp("internal error"))
+			return
+		}
+		defer rows.Close()
+
+		type adminTeam struct {
+			ID            int64     `json:"id"`
+			Name          string    `json:"name"`
+			CreatedAt     time.Time `json:"created_at"`
+			Members       int64     `json:"members"`
+			OpenIncidents int64     `json:"open_incidents"`
+		}
+		teams := []adminTeam{}
+		for rows.Next() {
+			var t adminTeam
+			var created int64
+			if err := rows.Scan(&t.ID, &t.Name, &created, &t.Members, &t.OpenIncidents); err != nil {
+				respond(w, http.StatusInternalServerError, errResp("internal error"))
+				return
+			}
+			t.CreatedAt = time.Unix(created, 0).UTC()
+			teams = append(teams, t)
+		}
+		if err := rows.Err(); err != nil {
+			respond(w, http.StatusInternalServerError, errResp("internal error"))
+			return
+		}
+		respond(w, http.StatusOK, teams)
+	}
+}
+
+// handleRenameTeam renames a team. An owner's job, and an administrator's when
+// a team has nobody left to do it.
+func handleRenameTeam(db *sql.DB) http.HandlerFunc {
+	return func(w http.ResponseWriter, r *http.Request) {
+		teamID, ok := teamParam(w, r)
+		if !ok {
+			return
+		}
+		if !requireTeamOwner(w, r, teamID) {
+			return
+		}
+
+		var req struct {
+			Name string `json:"name"`
+		}
+		if err := decodeJSON(r, &req); err != nil || req.Name == "" {
+			respond(w, http.StatusBadRequest, errResp("name is required"))
+			return
+		}
+
+		res, err := db.ExecContext(r.Context(),
+			"UPDATE teams SET name = $1 WHERE id = $2", req.Name, teamID)
+		if err != nil {
+			if isUniqueViolation(err) {
+				respond(w, http.StatusConflict, errResp("a team with that name already exists"))
+				return
+			}
+			respond(w, http.StatusInternalServerError, errResp("internal error"))
+			return
+		}
+		if n, _ := res.RowsAffected(); n == 0 {
+			respond(w, http.StatusNotFound, errResp("not found"))
+			return
+		}
+		w.WriteHeader(http.StatusNoContent)
+	}
+}
+
+// handleSetUserDisabled takes an account out of use, or puts it back.
+//
+// Not a delete: the person's acknowledgements, assignments and timeline entries
+// stay attached to them. Deleting a user nulls those columns, which rewrites
+// what happened during an incident months after the fact.
+func handleSetUserDisabled(db *sql.DB) http.HandlerFunc {
+	return func(w http.ResponseWriter, r *http.Request) {
+		id, err := strconv.ParseInt(chi.URLParam(r, "id"), 10, 64)
+		if err != nil {
+			respond(w, http.StatusBadRequest, errResp("invalid user id"))
+			return
+		}
+		var req struct {
+			Disabled *bool `json:"disabled"`
+		}
+		if err := decodeJSON(r, &req); err != nil || req.Disabled == nil {
+			respond(w, http.StatusBadRequest, errResp("disabled is required"))
+			return
+		}
+
+		if *req.Disabled {
+			caller, _ := userFromContext(r.Context())
+			if caller.ID == id {
+				respond(w, http.StatusConflict, errResp("cannot disable your own account"))
+				return
+			}
+			last, err := isLastAdmin(r.Context(), db, id)
+			if err != nil {
+				respond(w, http.StatusInternalServerError, errResp("internal error"))
+				return
+			}
+			if last {
+				respond(w, http.StatusConflict, errResp("cannot disable the last administrator"))
+				return
+			}
+		}
+
+		var res sql.Result
+		if *req.Disabled {
+			res, err = db.ExecContext(r.Context(),
+				"UPDATE users SET disabled_at = "+nowEpoch+" WHERE id = $1 AND disabled_at IS NULL", id)
+		} else {
+			res, err = db.ExecContext(r.Context(),
+				"UPDATE users SET disabled_at = NULL WHERE id = $1", id)
+		}
+		if err != nil {
+			respond(w, http.StatusInternalServerError, errResp("internal error"))
+			return
+		}
+		if n, _ := res.RowsAffected(); n == 0 {
+			// Either no such user, or already in the state asked for. The
+			// second is not a failure, so check which before answering.
+			var exists int
+			if err := db.QueryRowContext(r.Context(),
+				"SELECT 1 FROM users WHERE id = $1", id).Scan(&exists); errors.Is(err, sql.ErrNoRows) {
+				respond(w, http.StatusNotFound, errResp("user not found"))
+				return
+			}
+		}
+
+		// Signing back in is the only way to use a re-enabled account, and a
+		// disabled one must not keep a live session.
+		if *req.Disabled {
+			db.ExecContext(r.Context(), "DELETE FROM sessions WHERE user_id = $1", id) //nolint:errcheck
+		}
+
+		user, err := fetchUser(r.Context(), db, id)
+		if err != nil {
+			respond(w, http.StatusInternalServerError, errResp("internal error"))
+			return
+		}
+		respond(w, http.StatusOK, user)
+	}
+}

internal/api/settings_test.go

@@ -0,0 +1,275 @@
+package api_test
+
+import (
+	"net/http"
+	"testing"
+	"time"
+
+	"git.ryuvia.com/niklas/terdut-server/internal/api"
+)
+
+// The settings an administrator can change, and the ones they cannot.
+func TestSettings_EditableAndReadOnly(t *testing.T) {
+	s := newTS(t)
+
+	var got struct {
+		Editable map[string]struct {
+			Seconds     int64  `json:"seconds"`
+			Description string `json:"description"`
+			MinSeconds  int64  `json:"min_seconds"`
+			MaxSeconds  int64  `json:"max_seconds"`
+		} `json:"editable"`
+		FromEnv map[string]string `json:"from_env"`
+	}
+	decode(t, s.req(t, http.MethodGet, "/api/admin/settings", nil), &got)
+
+	// Seeded from the environment the server started with, not from zero.
+	if v := got.Editable["notify_repeat_seconds"].Seconds; v != 900 {
+		t.Errorf("notify_repeat_seconds seeded as %d, want 900", v)
+	}
+	if v := got.Editable["stale_after_seconds"].Seconds; v != 21600 {
+		t.Errorf("stale_after_seconds seeded as %d, want 21600", v)
+	}
+	if got.Editable["archive_after_seconds"].Description == "" {
+		t.Error("a setting without a description is a number nobody can act on")
+	}
+
+	// The environment half is visible so somebody can see where it lives, but
+	// never the credentials themselves.
+	if _, ok := got.FromEnv["public_url"]; !ok {
+		t.Error("public_url should be reported as environment-configured")
+	}
+	for _, leak := range []string{"ntfy_token", "dsn", "database_dsn", "password"} {
+		if v, ok := got.FromEnv[leak]; ok {
+			t.Errorf("%s must not be in the settings response (got %q)", leak, v)
+		}
+	}
+}
+
+// Changing a setting takes effect on the next tick, without a restart. This is
+// the whole point of moving them out of the environment.
+func TestSettings_ChangeTakesEffectOnTheNextSweep(t *testing.T) {
+	s := newTS(t)
+
+	// An alert whose last webhook was two hours ago. Under the seeded
+	// stale_after of six hours the sweeper leaves it alone.
+	postWebhook(t, s, []map[string]any{
+		amAlert("fp-settings", "Stale", "firing", "2026-05-20T10:00:00Z", zeroTime, nil),
+	})
+	s.exec(t, "UPDATE alerts SET received_at = $1 WHERE fingerprint = $2",
+		time.Now().Add(-2*time.Hour).Unix(), "fp-settings")
+
+	sweep(t, s, noArchive)
+	if status, _, _ := s.alertRow(t, "fp-settings"); status != "firing" {
+		t.Fatalf("before the change the alert should still be firing, got %q", status)
+	}
+
+	// Shorten it to an hour. Nothing restarts.
+	resp := s.req(t, http.MethodPut, "/api/admin/settings",
+		map[string]int64{"stale_after_seconds": 3600})
+	resp.Body.Close()
+	if resp.StatusCode != http.StatusNoContent {
+		t.Fatalf("change setting: %d", resp.StatusCode)
+	}
+
+	sweep(t, s, noArchive)
+	status, source, _ := s.alertRow(t, "fp-settings")
+	if status != "resolved" {
+		t.Errorf("after the change the alert should have expired, got %q", status)
+	}
+	if source == nil || *source != "expiry" {
+		t.Errorf("expected resolution_source expiry, got %v", source)
+	}
+}
+
+// A typo must not look like configuration, and a slipped decimal point must not
+// produce a server that sweeps every second.
+func TestSettings_RejectsUnknownKeysAndSillyValues(t *testing.T) {
+	s := newTS(t)
+
+	for _, c := range []struct {
+		name string
+		body map[string]int64
+	}{
+		{"unknown key", map[string]int64{"notify_repeat_second": 60}},
+		{"below the floor", map[string]int64{"stale_after_seconds": 30}},
+		{"above the ceiling", map[string]int64{"archive_after_seconds": 400 * 24 * 3600}},
+		{"nothing at all", map[string]int64{}},
+	} {
+		resp := s.req(t, http.MethodPut, "/api/admin/settings", c.body)
+		resp.Body.Close()
+		if resp.StatusCode != http.StatusBadRequest {
+			t.Errorf("%s: expected 400, got %d", c.name, resp.StatusCode)
+		}
+	}
+}
+
+// Settings are the server's behaviour, so only an administrator may change
+// them — or see where the rest of the configuration comes from.
+func TestSettings_AreAdminOnly(t *testing.T) {
+	s := newTS(t)
+	_, call := member(t, s, "member")
+
+	for _, c := range []struct {
+		method string
+		body   any
+	}{
+		{http.MethodGet, nil},
+		{http.MethodPut, map[string]int64{"notify_repeat_seconds": 60}},
+	} {
+		resp := call(c.method, "/api/admin/settings", c.body)
+		resp.Body.Close()
+		if resp.StatusCode != http.StatusForbidden {
+			t.Errorf("%s /api/admin/settings: expected 403, got %d", c.method, resp.StatusCode)
+		}
+	}
+
+	resp := call(http.MethodGet, "/api/admin/teams", nil)
+	resp.Body.Close()
+	if resp.StatusCode != http.StatusForbidden {
+		t.Errorf("GET /api/admin/teams: expected 403, got %d", resp.StatusCode)
+	}
+}
+
+// An administrator sees every team, including ones they are not in — which is
+// exactly what /api/teams must not show them.
+func TestSettings_AdminSeesEveryTeam(t *testing.T) {
+	s := newTS(t)
+	newTeam(t, s, "red")
+	newTeam(t, s, "blue")
+
+	all := list(t, s.req(t, http.MethodGet, "/api/admin/teams", nil))
+	if len(all) != 3 { // Default, red, blue
+		t.Fatalf("admin should see all 3 teams, saw %d", len(all))
+	}
+	for _, team := range all {
+		if _, ok := team["members"]; !ok {
+			t.Error("the admin listing should say how big each team is")
+		}
+	}
+
+	// The admin created them, so they own them — but they are not a member of
+	// a team somebody else makes, and /api/teams still answers "what am I in".
+	mine := list(t, s.req(t, http.MethodGet, "/api/teams", nil))
+	if len(mine) != 3 {
+		t.Errorf("the creator is an owner of what they created, saw %d", len(mine))
+	}
+}
+
+// Disabling is not deleting: the account stops working and the history stays.
+func TestSettings_DisablingAnAccountKeepsItsHistory(t *testing.T) {
+	s := newTS(t)
+	memberID, call := member(t, s, "leaver")
+
+	// They acknowledge an incident, so there is history to preserve.
+	postWebhook(t, s, []map[string]any{
+		amAlert("fp-leaver", "DiskFull", "firing", "2026-05-20T10:00:00Z", zeroTime, nil),
+	})
+	resp := call(http.MethodPost, "/api/incidents/1/acknowledge", nil)
+	resp.Body.Close()
+	if resp.StatusCode != http.StatusOK {
+		t.Fatalf("acknowledge: %d", resp.StatusCode)
+	}
+
+	resp = s.req(t, http.MethodPut, "/api/users/"+id64(memberID)+"/disabled",
+		map[string]bool{"disabled": true})
+	resp.Body.Close()
+	if resp.StatusCode != http.StatusOK {
+		t.Fatalf("disable: %d", resp.StatusCode)
+	}
+
+	// Their API key stops working.
+	resp = call(http.MethodGet, "/api/incidents", nil)
+	resp.Body.Close()
+	if resp.StatusCode != http.StatusUnauthorized {
+		t.Errorf("a disabled user's key: expected 401, got %d", resp.StatusCode)
+	}
+
+	// The acknowledgement still names them.
+	var incident map[string]any
+	decode(t, s.req(t, http.MethodGet, "/api/incidents/1", nil), &incident)
+	if incident["acknowledged_by"] != "leaver" {
+		t.Errorf("the acknowledgement should still name leaver, got %v", incident["acknowledged_by"])
+	}
+	if incident["status"] != "acknowledged" {
+		t.Errorf("the incident should still be acknowledged, got %v", incident["status"])
+	}
+
+	// And re-enabling gives the account back.
+	resp = s.req(t, http.MethodPut, "/api/users/"+id64(memberID)+"/disabled",
+		map[string]bool{"disabled": false})
+	resp.Body.Close()
+	resp = call(http.MethodGet, "/api/incidents", nil)
+	resp.Body.Close()
+	if resp.StatusCode != http.StatusOK {
+		t.Errorf("after re-enabling: expected 200, got %d", resp.StatusCode)
+	}
+}
+
+// The same two guards as deleting and demoting: an install must keep somebody
+// who can administer it.
+func TestSettings_CannotDisableYourselfOrTheLastAdmin(t *testing.T) {
+	s := newTS(t)
+
+	resp := s.req(t, http.MethodPut, "/api/users/1/disabled", map[string]bool{"disabled": true})
+	resp.Body.Close()
+	if resp.StatusCode != http.StatusConflict {
+		t.Errorf("disabling yourself: expected 409, got %d", resp.StatusCode)
+	}
+}
+
+// Renaming a team is an owner's job, and the name stays unique.
+func TestSettings_TeamRename(t *testing.T) {
+	s := newTS(t)
+	team := newTeam(t, s, "red")
+
+	resp := s.req(t, http.MethodPut, "/api/teams/"+id64(team.id), map[string]string{"name": "Platform"})
+	resp.Body.Close()
+	if resp.StatusCode != http.StatusNoContent {
+		t.Fatalf("rename: %d", resp.StatusCode)
+	}
+
+	teams := list(t, s.req(t, http.MethodGet, "/api/admin/teams", nil))
+	found := false
+	for _, x := range teams {
+		if x["name"] == "Platform" {
+			found = true
+		}
+	}
+	if !found {
+		t.Error("the renamed team should be listed under its new name")
+	}
+
+	// Taking a name that exists is a conflict, not a silent second team with
+	// the same label.
+	resp = s.req(t, http.MethodPut, "/api/teams/"+id64(team.id), map[string]string{"name": "Default"})
+	resp.Body.Close()
+	if resp.StatusCode != http.StatusConflict {
+		t.Errorf("renaming onto an existing name: expected 409, got %d", resp.StatusCode)
+	}
+}
+
+// The seed runs once. A redeploy must not put the chart's default back over an
+// administrator's edit — the rule the dead man's switches already follow.
+func TestSettings_SeedDoesNotOverwrite(t *testing.T) {
+	s := newTS(t)
+
+	resp := s.req(t, http.MethodPut, "/api/admin/settings",
+		map[string]int64{"notify_repeat_seconds": 60})
+	resp.Body.Close()
+
+	// A second start, with the environment still saying 15 minutes.
+	if err := api.SeedSettings(t.Context(), s.db, testConfig()); err != nil {
+		t.Fatalf("re-seed: %v", err)
+	}
+
+	var got struct {
+		Editable map[string]struct {
+			Seconds int64 `json:"seconds"`
+		} `json:"editable"`
+	}
+	decode(t, s.req(t, http.MethodGet, "/api/admin/settings", nil), &got)
+	if v := got.Editable["notify_repeat_seconds"].Seconds; v != 60 {
+		t.Errorf("the edit should survive a restart, got %d", v)
+	}
+}

internal/api/teams.go

@@ -463,9 +463,10 @@ func teamParam(w http.ResponseWriter, r *http.Request) (int64, bool) {
 	return id, true
 }
 
-// defaultTeamID is the team the deprecated unauthenticated webhook routes to:
+// defaultTeamID is the oldest team, which on an upgraded install is the
-// the oldest one, which on an upgraded install is the "Default" team every
+// "Default" team every pre-teams row was moved into and on a fresh one is the
-// pre-teams row was moved into.
+// team migration 003 creates. Bootstrap puts the first user in it, so somebody
+// signing in to a new server lands somewhere rather than in no team at all.
 func defaultTeamID(ctx context.Context, db *sql.DB) (int64, error) {
 	var id int64
 	err := db.QueryRowContext(ctx, "SELECT id FROM teams ORDER BY id LIMIT 1").Scan(&id)

internal/api/testdb_test.go

@@ -7,7 +7,9 @@ import (
 	"os"
 	"strings"
 	"testing"
+	"time"
 
+	"git.ryuvia.com/niklas/terdut-server/internal/config"
 	"git.ryuvia.com/niklas/terdut-server/internal/db"
 )
 
@@ -30,6 +32,19 @@ import (
 // tests nothing is worse than one that does not run.
 const testDSNEnv = "TERDUT_TEST_DSN"
 
+// testConfig is the environment half of the server's configuration, which the
+// admin settings page renders read-only and SeedSettings seeds the editable
+// half from. The durations match the defaults config.Load would produce, so a
+// test that never touches the settings table behaves as a fresh install does.
+func testConfig() config.Config {
+	return config.Config{
+		Addr:         ":8080",
+		ArchiveAfter: 7 * 24 * time.Hour,
+		StaleAfter:   6 * time.Hour,
+		NotifyRepeat: 15 * time.Minute,
+	}
+}
+
 // defaultTeam is the team migration 003 creates and the bootstrap user owns, as
 // a path segment. Every test that does not say otherwise works inside it.
 const defaultTeam = "1"

internal/api/users.go

@@ -96,7 +96,7 @@ func handleBootstrap(db *sql.DB) http.HandlerFunc {
 func handleListUsers(db *sql.DB) http.HandlerFunc {
 	return func(w http.ResponseWriter, r *http.Request) {
 		rows, err := db.QueryContext(r.Context(),
-			"SELECT id, username, email, created_at, ntfy_topic, is_admin FROM users ORDER BY id")
+			"SELECT id, username, email, created_at, ntfy_topic, is_admin, disabled_at FROM users ORDER BY id")
 		if err != nil {
 			respond(w, http.StatusInternalServerError, errResp("internal error"))
 			return
@@ -107,11 +107,13 @@ func handleListUsers(db *sql.DB) http.HandlerFunc {
 		for rows.Next() {
 			var u models.User
 			var ts int64
-			if err := rows.Scan(&u.ID, &u.Username, &u.Email, &ts, &u.NtfyTopic, &u.IsAdmin); err != nil {
+			var disabled *int64
+			if err := rows.Scan(&u.ID, &u.Username, &u.Email, &ts, &u.NtfyTopic, &u.IsAdmin, &disabled); err != nil {
 				respond(w, http.StatusInternalServerError, errResp("internal error"))
 				return
 			}
 			u.CreatedAt = time.Unix(ts, 0).UTC()
+			u.DisabledAt = unixPtr(disabled)
 			users = append(users, u)
 		}
 		respond(w, http.StatusOK, users)
@@ -325,13 +327,15 @@ func randomToken() (raw, hash string, err error) {
 func fetchUser(ctx context.Context, db *sql.DB, id int64) (models.User, error) {
 	var u models.User
 	var ts int64
+	var disabled *int64
 	err := db.QueryRowContext(ctx,
-		"SELECT id, username, email, created_at, ntfy_topic, is_admin FROM users WHERE id = $1", id).
+		"SELECT id, username, email, created_at, ntfy_topic, is_admin, disabled_at FROM users WHERE id = $1", id).
-		Scan(&u.ID, &u.Username, &u.Email, &ts, &u.NtfyTopic, &u.IsAdmin)
+		Scan(&u.ID, &u.Username, &u.Email, &ts, &u.NtfyTopic, &u.IsAdmin, &disabled)
 	if err != nil {
 		return u, err
 	}
 	u.CreatedAt = time.Unix(ts, 0).UTC()
+	u.DisabledAt = unixPtr(disabled)
 	return u, nil
 }
 

internal/db/migrations/005_settings.sql

@@ -0,0 +1,35 @@
+-- Settings that an administrator can change without a redeploy, and the flag
+-- that takes an account out of use without deleting it.
+--
+-- Three of the server's tunables were environment variables, which meant
+-- changing how long an incident waits before it is paged again required editing
+-- a chart, merging it, and waiting for a reconcile. They are behaviour, not
+-- infrastructure, and the difference is who needs to change them and how often.
+--
+-- What stays in the environment: the ntfy URL and token, the database DSN, the
+-- listen address and the public URL. Those are where the server is plugged in
+-- rather than how it behaves, they are needed before the database is open, and
+-- two of them are credentials.
+--
+-- Key/value rather than a column per setting. A settings table with one row and
+-- a column per knob needs a migration for every new knob, and #6 and #7 will
+-- both add some. The cost is that values are text and the accessor has to say
+-- what type it wanted; settings.go does that in one place.
+--
+-- No rows are seeded here: a migration cannot read the environment. The server
+-- inserts each key from its own configuration at startup, once, so an install
+-- that upgrades keeps exactly the behaviour it had. See SeedSettings.
+CREATE TABLE settings (
+    key        TEXT   PRIMARY KEY,
+    value      TEXT   NOT NULL,
+    updated_at BIGINT NOT NULL DEFAULT FLOOR(EXTRACT(EPOCH FROM now()))::bigint
+);
+
+-- Disabling an account rather than deleting it: the person has left, or the
+-- credential is suspect, and their incidents, acknowledgements and timeline
+-- entries must stay exactly where they are. Deleting a user nulls their
+-- acknowledged_by and assigned_to, which quietly rewrites history.
+--
+-- A disabled user cannot sign in and their API keys stop working, but they are
+-- still a name the timeline can show and still a member of their teams.
+ALTER TABLE users ADD COLUMN disabled_at BIGINT;

internal/db/migrations/006_escalation.sql

@@ -0,0 +1,95 @@
+-- Escalation: page somebody else when the first person does not answer.
+--
+-- This is the gap the whole multi-tenancy line of work was opened to close.
+-- Until now an unacknowledged incident re-paged the same topic every
+-- notify_repeat forever, which is a louder version of the same silence: if the
+-- person on call is asleep, has no signal, or has left, nothing else happens.
+--
+-- Shape: one policy per team, an ordered list of levels, each level with a
+-- timeout and a set of targets. When a level's timeout passes and the incident
+-- is still triggered, the next level is paged. When the last level passes, the
+-- chain repeats repeat_count times, and then the team's fallback topic is paged
+-- once as the end of the line.
+--
+-- A team WITHOUT a policy keeps exactly today's behaviour: page the assignee,
+-- then remind on the same topic. Escalation is opt-in per team, and the two
+-- never both run for one incident -- see enqueueReminders.
+CREATE TABLE escalation_policies (
+    -- One per team for now, hence the team as the key rather than an id with a
+    -- unique index: routing different alerts to different chains needs the
+    -- alert to carry something to route ON, which is a separate question.
+    team_id BIGINT PRIMARY KEY REFERENCES teams(id) ON DELETE CASCADE,
+
+    -- How many extra times to run the whole chain after it has been walked
+    -- once. 0 means walk it once and stop at the fallback.
+    repeat_count BIGINT NOT NULL DEFAULT 0 CHECK (repeat_count >= 0 AND repeat_count <= 10),
+
+    -- Where the last page goes when every level has been tried. Per team now:
+    -- TERDUT_NTFY_FALLBACK_TOPIC was one topic for the whole install, which in
+    -- a multi-team server pages the wrong people. Empty means the chain simply
+    -- ends.
+    fallback_topic TEXT NOT NULL DEFAULT '',
+
+    updated_at BIGINT NOT NULL DEFAULT FLOOR(EXTRACT(EPOCH FROM now()))::bigint
+);
+
+CREATE TABLE escalation_levels (
+    id              BIGINT GENERATED BY DEFAULT AS IDENTITY PRIMARY KEY,
+    team_id         BIGINT NOT NULL REFERENCES escalation_policies(team_id) ON DELETE CASCADE,
+    -- 1-based, dense. The API rewrites the whole ladder on every edit rather
+    -- than patching one rung, so there is no way to leave a gap.
+    position        BIGINT NOT NULL,
+    -- How long this level has to produce an acknowledgement before the next one
+    -- is paged. Seconds, like every other duration in this schema.
+    timeout_seconds BIGINT NOT NULL CHECK (timeout_seconds > 0),
+
+    UNIQUE (team_id, position)
+);
+
+-- Who a level pages. Either a named person, or whoever the team's rota says is
+-- on call today -- which is the target that keeps working when the rota
+-- changes and nobody remembers to edit the policy.
+CREATE TABLE escalation_targets (
+    id       BIGINT GENERATED BY DEFAULT AS IDENTITY PRIMARY KEY,
+    level_id BIGINT NOT NULL REFERENCES escalation_levels(id) ON DELETE CASCADE,
+    kind     TEXT   NOT NULL CHECK (kind IN ('user', 'oncall')),
+    -- Set for kind='user', NULL for kind='oncall'.
+    user_id  BIGINT REFERENCES users(id) ON DELETE CASCADE,
+
+    CHECK ((kind = 'user' AND user_id IS NOT NULL) OR (kind = 'oncall' AND user_id IS NULL))
+);
+
+CREATE INDEX escalation_targets_level_idx ON escalation_targets(level_id);
+
+-- ---------------------------------------------------------------------------
+-- Where an incident is in its chain.
+--
+-- On the incident rather than in a side table: it is read on every notifier
+-- tick alongside the incident's status, and one row per incident is exactly
+-- what the state is.
+-- ---------------------------------------------------------------------------
+
+-- 0 means no level has been paged yet, which is the state of every incident
+-- that existed before escalation and of every incident in a team with no
+-- policy. 1 is the first level.
+ALTER TABLE incidents ADD COLUMN escalation_level BIGINT NOT NULL DEFAULT 0;
+
+-- When the current level was entered, and therefore what its timeout is
+-- measured from. NULL while escalation_level is 0.
+ALTER TABLE incidents ADD COLUMN escalation_level_at BIGINT;
+
+-- How many times the chain has been walked in full. Compared against the
+-- policy's repeat_count.
+ALTER TABLE incidents ADD COLUMN escalation_round BIGINT NOT NULL DEFAULT 0;
+
+-- The notifier's escalation query: incidents still waiting, oldest level first.
+CREATE INDEX incidents_escalation_idx
+    ON incidents(escalation_level_at)
+ WHERE resolved_at IS NULL AND status = 'triggered';
+
+-- 'escalated' joins the outbox kinds: a page that went out because nobody
+-- answered the last one, which is worth telling apart from the first page and
+-- from a reminder when reading the timeline or debugging a delivery.
+ALTER TABLE notifications DROP CONSTRAINT notifications_kind_check;
+ALTER TABLE notifications ADD CONSTRAINT notifications_kind_check
+    CHECK (kind IN ('triggered', 'reminder', 'resolved', 'escalated'));

internal/models/user.go

@@ -13,6 +13,11 @@ type User struct {
 	// fallback topic instead.
 	NtfyTopic *string `json:"ntfy_topic,omitempty"`
 
+	// DisabledAt is when the account was taken out of use, or nil. A disabled
+	// user cannot authenticate by either credential, and keeps their name on
+	// every acknowledgement and timeline entry they made.
+	DisabledAt *time.Time `json:"disabled_at,omitempty"`
+
 	// IsAdmin is the system administrator flag: managing users and API keys.
 	// Not omitempty — a client has to be able to tell "false" from "this server
 	// is too old to have the field", and the web UI decides what to show from

internal/web/static/app.css

@@ -630,3 +630,30 @@ kbd {
   .toast, .app.detail-open ~ .toast { bottom: 24px; }
   .only-desktop { display: block; }
 }
+
+/* --- admin ---------------------------------------------------------------
+   The admin page is three tables of things you act on, so it needs table
+   styling the rest of the app never did: the queue is a list of links and the
+   account page is a form. */
+.admin-table { width: 100%; border-collapse: collapse; font-size: 14px; }
+.admin-table th {
+  text-align: left; font-weight: 600; color: var(--muted); font-size: 12px;
+  text-transform: uppercase; letter-spacing: 0.04em;
+  padding: 4px 8px 4px 0; border-bottom: 1px solid var(--border);
+}
+.admin-table td { padding: 8px 8px 8px 0; border-bottom: 1px solid var(--border); vertical-align: middle; }
+.admin-table tr:last-child td { border-bottom: none; }
+.admin-table .num { text-align: right; font-variant-numeric: tabular-nums; }
+.admin-table td .btn-sm + .btn-sm { margin-left: 6px; }
+/* A disabled account stays readable — it is still the name on old
+   acknowledgements — but should not look like a working one. */
+.disabled-row td { opacity: 0.55; }
+.btn-sm.danger { color: var(--crit); border-color: var(--crit-soft); }
+
+.inline-form { display: flex; gap: 8px; margin-top: 12px; }
+.inline-form input { flex: 1; min-width: 0; }
+
+.admin-settings .setting-value { width: 5.5em; margin-right: 6px; }
+.admin-settings .setting-unit { max-width: 8em; }
+.admin-settings button[type="submit"] { margin-top: 12px; }
+.small { font-size: 13px; }

internal/web/static/index.html

@@ -59,6 +59,13 @@
         <svg viewBox="0 0 24 24" aria-hidden="true"><path d="M6 16V11a6 6 0 0 1 12 0v5l1.5 2h-15z"/><path d="M10 20.5a2 2 0 0 0 4 0"/></svg>
         <span class="nav-label">Alerts</span>
       </a>
+      <!-- Hidden unless the signed-in user is a system administrator; app.js
+           unhides it once /api/me says so. The server refuses every admin
+           endpoint regardless, so this is a courtesy and not a gate. -->
+      <a class="nav-link" href="/admin" data-section="admin" id="nav-admin" hidden>
+        <svg viewBox="0 0 24 24" aria-hidden="true"><path d="M12 3l7 3v6c0 4-3 7-7 9-4-2-7-5-7-9V6z"/></svg>
+        <span class="nav-label">Admin</span>
+      </a>
       <a class="nav-link" href="/more" data-section="more">
         <svg viewBox="0 0 24 24" aria-hidden="true"><circle cx="12" cy="8" r="3.5"/><path d="M5 20a7 7 0 0 1 14 0"/></svg>
         <span class="nav-label">Account</span>
@@ -80,6 +87,7 @@
 
     <section id="view-oncall" class="view view-page" data-view="oncall" hidden></section>
     <section id="view-alerts" class="view view-page" data-view="alerts" hidden></section>
+    <section id="view-admin" class="view view-page" data-view="admin" hidden></section>
     <section id="view-more" class="view view-page" data-view="more" hidden></section>
   </div>
 

internal/web/static/js/admin.js

@@ -0,0 +1,308 @@
+// Administration: the teams on this server, the people who can sign in, and
+// the settings that change how the server behaves.
+//
+// Only rendered for a system administrator. The server enforces that on every
+// endpoint regardless — hiding a section is a courtesy to the reader, not a
+// permission — so this view simply says so rather than pretending to be a
+// gate.
+
+import * as api from './api.js';
+import { h, clear, spinner, confirm } from './ui.js';
+import { state, myID } from './state.js';
+
+const view = () => document.getElementById('view-admin');
+
+let data = null; // { teams, users, settings }
+let error = null;
+let busy = false;
+
+export function show() {
+  if (!data) clear(view(), spinner());
+  refresh();
+}
+
+export async function refresh() {
+  if (!state.me?.user?.is_admin) {
+    data = null;
+    render();
+    return;
+  }
+  try {
+    const [teams, users, settings] = await Promise.all([
+      api.adminTeams(),
+      api.users(),
+      api.adminSettings(),
+    ]);
+    data = { teams, users, settings };
+    error = null;
+  } catch (err) {
+    error = err.message;
+  }
+  render();
+}
+
+function render() {
+  if (!state.me?.user?.is_admin) {
+    clear(view(), h('div', { class: 'card' },
+      h('p', { class: 'muted', text: 'Administration is for system administrators. Ask one for access.' })));
+    return;
+  }
+  if (!data) {
+    clear(view(), error ? h('div', { class: 'load-error', text: error }) : spinner());
+    return;
+  }
+  clear(view(),
+    error && h('div', { class: 'load-error', text: `Showing older data: ${error}` }),
+    teamsCard(),
+    usersCard(),
+    settingsCard(),
+  );
+}
+
+// --- teams -----------------------------------------------------------------
+
+function teamsCard() {
+  const rows = data.teams.map((t) =>
+    h('tr', {},
+      h('td', {}, h('strong', { text: t.name })),
+      h('td', { class: 'num', text: String(t.members) }),
+      h('td', { class: 'num', text: String(t.open_incidents) }),
+      h('td', {},
+        h('button', {
+          class: 'btn-sm',
+          type: 'button',
+          text: 'Rename',
+          onclick: () => renameTeam(t),
+        }),
+        // A team with open incidents cannot be deleted, and saying so before
+        // the click is kinder than a 409 afterwards.
+        h('button', {
+          class: 'btn-sm danger',
+          type: 'button',
+          text: 'Delete',
+          disabled: t.open_incidents > 0,
+          title: t.open_incidents > 0 ? 'Resolve its open incidents first' : '',
+          onclick: () => deleteTeam(t),
+        }),
+      ),
+    ));
+
+  return h('div', { class: 'card' },
+    h('h2', { text: 'Teams' }),
+    h('table', { class: 'admin-table' },
+      h('thead', {}, h('tr', {},
+        h('th', { text: 'Name' }),
+        h('th', { class: 'num', text: 'Members' }),
+        h('th', { class: 'num', text: 'Open' }),
+        h('th', { text: '' }))),
+      h('tbody', {}, rows)),
+    newTeamForm(),
+  );
+}
+
+function newTeamForm() {
+  const name = h('input', { name: 'name', type: 'text', placeholder: 'New team name', required: true });
+  const form = h('form', { class: 'inline-form' }, name,
+    h('button', { class: 'btn', type: 'submit', text: 'Create' }));
+  form.addEventListener('submit', async (e) => {
+    e.preventDefault();
+    if (busy) return;
+    busy = true;
+    try {
+      await api.createTeam(name.value.trim());
+      name.value = '';
+      await refresh();
+    } catch (err) {
+      error = err.message;
+      render();
+    } finally {
+      busy = false;
+    }
+  });
+  return form;
+}
+
+async function renameTeam(team) {
+  const next = window.prompt(`Rename ${team.name} to:`, team.name);
+  if (!next || next === team.name) return;
+  try {
+    await api.renameTeam(team.id, next);
+  } catch (err) {
+    error = err.message;
+  }
+  refresh();
+}
+
+async function deleteTeam(team) {
+  if (!(await confirm({
+    title: `Delete ${team.name}?`,
+    text: 'Its alerts, incidents, schedule and integrations go with it. This cannot be undone.',
+    confirmLabel: 'Delete',
+    danger: true,
+  }))) return;
+  try {
+    await api.deleteTeam(team.id);
+  } catch (err) {
+    error = err.message;
+  }
+  refresh();
+}
+
+// --- users -----------------------------------------------------------------
+
+function usersCard() {
+  const rows = data.users.map((u) => {
+    const self = u.id === myID();
+    return h('tr', { class: u.disabled_at ? 'disabled-row' : '' },
+      h('td', {},
+        h('strong', { text: u.username }),
+        u.disabled_at && h('span', { class: 'row-team', text: 'disabled' }),
+        self && h('span', { class: 'you', text: 'you' })),
+      h('td', { class: 'muted', text: u.email }),
+      h('td', {}, u.is_admin ? h('span', { class: 'row-team', text: 'admin' }) : null),
+      h('td', {},
+        // Neither action is offered for your own account: the server refuses
+        // both, and an enabled-looking button that always fails is worse than
+        // no button.
+        !self && h('button', {
+          class: 'btn-sm',
+          type: 'button',
+          text: u.is_admin ? 'Revoke admin' : 'Make admin',
+          onclick: () => setAdmin(u, !u.is_admin),
+        }),
+        !self && h('button', {
+          class: 'btn-sm danger',
+          type: 'button',
+          text: u.disabled_at ? 'Enable' : 'Disable',
+          onclick: () => setDisabled(u, !u.disabled_at),
+        }),
+      ),
+    );
+  });
+
+  return h('div', { class: 'card' },
+    h('h2', { text: 'Users' }),
+    h('p', { class: 'muted small' },
+      'Disabling an account stops it signing in and stops its API keys, and keeps ',
+      'its acknowledgements and timeline entries. Deleting a user erases those.'),
+    h('table', { class: 'admin-table' },
+      h('thead', {}, h('tr', {},
+        h('th', { text: 'User' }),
+        h('th', { text: 'Email' }),
+        h('th', { text: '' }),
+        h('th', { text: '' }))),
+      h('tbody', {}, rows)),
+  );
+}
+
+async function setAdmin(user, next) {
+  if (next && !(await confirm({
+    title: `Make ${user.username} an administrator?`,
+    text: 'They will be able to create and delete users, and grant this to others.',
+    confirmLabel: 'Make admin',
+  }))) return;
+  try {
+    await api.setUserAdmin(user.id, next);
+  } catch (err) {
+    error = err.message;
+  }
+  refresh();
+}
+
+async function setDisabled(user, next) {
+  if (next && !(await confirm({
+    title: `Disable ${user.username}?`,
+    text: 'They cannot sign in and their API keys stop working. Their history stays.',
+    confirmLabel: 'Disable',
+    danger: true,
+  }))) return;
+  try {
+    await api.setUserDisabled(user.id, next);
+  } catch (err) {
+    error = err.message;
+  }
+  refresh();
+}
+
+// --- settings --------------------------------------------------------------
+
+// Seconds are what the API speaks; people think in minutes and hours. The two
+// are converted here rather than in the server, which should keep exactly one
+// unit.
+const UNITS = [
+  { label: 'minutes', seconds: 60 },
+  { label: 'hours', seconds: 3600 },
+  { label: 'days', seconds: 86400 },
+];
+
+function bestUnit(seconds) {
+  for (const u of [...UNITS].reverse()) {
+    if (seconds > 0 && seconds % u.seconds === 0) return u;
+  }
+  return UNITS[0];
+}
+
+function settingsCard() {
+  const editable = data.settings.editable || {};
+  const inputs = new Map();
+
+  const rows = Object.entries(editable).map(([key, s]) => {
+    const unit = bestUnit(s.seconds);
+    const value = h('input', {
+      type: 'number',
+      min: '0',
+      value: String(Math.round(s.seconds / unit.seconds)),
+      class: 'setting-value',
+    });
+    const select = h('select', { class: 'setting-unit' },
+      ...UNITS.map((u) => h('option', {
+        value: String(u.seconds),
+        text: u.label,
+        selected: u.seconds === unit.seconds,
+      })));
+    inputs.set(key, () => Number(value.value) * Number(select.value));
+
+    return h('tr', {},
+      h('td', {}, h('strong', { text: key.replace(/_seconds$/, '').replace(/_/g, ' ') })),
+      h('td', { class: 'muted small', text: s.description }),
+      h('td', {}, value, select),
+    );
+  });
+
+  const form = h('form', { class: 'admin-settings' },
+    h('table', { class: 'admin-table' }, h('tbody', {}, rows)),
+    h('button', { class: 'btn', type: 'submit', text: 'Save settings' }));
+
+  form.addEventListener('submit', async (e) => {
+    e.preventDefault();
+    if (busy) return;
+    busy = true;
+    const body = {};
+    for (const [key, read] of inputs) body[key] = read();
+    try {
+      await api.setAdminSettings(body);
+      await refresh();
+    } catch (err) {
+      error = err.message;
+      render();
+    } finally {
+      busy = false;
+    }
+  });
+
+  const env = Object.entries(data.settings.from_env || {}).map(([k, v]) =>
+    h('tr', {},
+      h('td', {}, h('code', { text: k })),
+      h('td', { class: 'muted', text: v === '' ? '(unset)' : v })));
+
+  return h('div', { class: 'card' },
+    h('h2', { text: 'Settings' }),
+    h('p', { class: 'muted small', text: 'Saved changes take effect on the next sweep — no restart.' }),
+    form,
+    h('h3', { text: 'From the environment' }),
+    h('p', { class: 'muted small' },
+      'Where the server is plugged in, rather than how it behaves. These are set ',
+      'in the deployment and are read-only here. Credentials are never shown.'),
+    h('table', { class: 'admin-table' }, h('tbody', {}, env)),
+  );
+}

internal/web/static/js/api.js

@@ -88,6 +88,19 @@ export const alerts = (query, opts) => call('GET', '/alerts', { query, ...opts }
 
 // schedule
 export const teams = () => call('GET', '/teams');
+export const createTeam = (name) => call('POST', '/teams', { body: { name } });
+export const renameTeam = (id, name) => call('PUT', `/teams/${id}`, { body: { name } });
+export const deleteTeam = (id) => call('DELETE', `/teams/${id}`);
+
+// Administration. Every one of these is refused with 403 for anybody without
+// the flag, so the UI hides the section rather than guarding it.
+export const adminTeams = () => call('GET', '/admin/teams');
+export const adminSettings = () => call('GET', '/admin/settings');
+export const setAdminSettings = (body) => call('PUT', '/admin/settings', { body });
+export const setUserAdmin = (id, isAdmin) =>
+  call('PUT', `/users/${id}/admin`, { body: { is_admin: isAdmin } });
+export const setUserDisabled = (id, disabled) =>
+  call('PUT', `/users/${id}/disabled`, { body: { disabled } });
 export const schedule = (teamID, from, to) =>
   call('GET', `/teams/${teamID}/schedule`, { query: { from, to } });
 

internal/web/static/js/app.js

@@ -9,6 +9,7 @@ import * as incident from './incident.js';
 import * as oncall from './oncall.js';
 import * as alerts from './alerts.js';
 import * as account from './account.js';
+import * as admin from './admin.js';
 
 const $ = (id) => document.getElementById(id);
 
@@ -17,6 +18,7 @@ const SECTIONS = {
   queue: { title: 'Queue', view: queue },
   oncall: { title: 'On-call', view: oncall },
   alerts: { title: 'Alerts', view: alerts },
+  admin: { title: 'Admin', view: admin },
   more: { title: 'Account', view: account },
 };
 
@@ -24,7 +26,7 @@ function parseRoute(pathname) {
   const m = pathname.match(/^\/incidents\/(\d+)\/?$/);
   if (m) return { section: 'queue', incident: Number(m[1]) };
   const name = pathname.replace(/^\/|\/$/g, '');
-  if (name === 'oncall' || name === 'alerts' || name === 'more') return { section: name };
+  if (name === 'oncall' || name === 'alerts' || name === 'admin' || name === 'more') return { section: name };
   return { section: 'queue', incident: null };
 }
 
@@ -142,6 +144,9 @@ async function boot() {
   try {
     state.me = await api.me();
     await loadTeams();
+    // The Admin tab exists only for an administrator. Somebody who types /admin
+    // anyway gets the view's own "ask an administrator" card, not a blank page.
+    $('nav-admin').hidden = !state.me?.user?.is_admin;
     showApp();
   } catch (err) {
     if (err.status === 401) showLogin();

scripts/sqlite-to-postgres.go

@@ -1,198 +0,0 @@
-//go:build migrate
-
-// Command sqlite-to-postgres copies a terdut SQLite database into a freshly
-// migrated Postgres one. It exists for exactly one upgrade — the one that moved
-// this server off SQLite — and should be deleted once the installs that need it
-// have run it. The modernc.org/sqlite dependency goes with it.
-//
-// Build-tagged so the dependency stays out of the server binary and out of a
-// plain `go build ./...`:
-//
-//	go run -tags migrate ./scripts/sqlite-to-postgres.go \
-//	    -sqlite /path/to/terdut.db \
-//	    -dsn 'postgres://terdut:secret@localhost:5432/terdut?sslmode=disable'
-//
-// The Postgres side must already have the schema: start the new server once
-// against an empty database, let it migrate, stop it, then run this. The copy
-// refuses to touch a database that already has rows, so a second run cannot
-// double-insert.
-//
-// Ids are preserved, which is what keeps every foreign key — incident_alerts,
-// incident_events, notifications, the ack tokens — pointing at the same rows it
-// pointed at before. The identity sequences are moved past the copied ids at the
-// end, so the first row the server writes afterwards does not collide.
-package main
-
-import (
-	"database/sql"
-	"flag"
-	"fmt"
-	"log"
-	"strings"
-
-	_ "github.com/jackc/pgx/v5/stdlib"
-	_ "modernc.org/sqlite"
-)
-
-// tables are copied parents first: every foreign key points at a table earlier
-// in this list.
-var tables = []struct {
-	name    string
-	columns []string
-	// jsonb marks columns that were TEXT in SQLite and are jsonb in Postgres,
-	// so the insert can cast them.
-	jsonb []string
-	// sequence is the identity sequence to advance afterwards, empty when the
-	// table has no generated id.
-	sequence string
-}{
-	{name: "users", columns: []string{"id", "username", "email", "created_at", "ntfy_topic", "password_hash"}, sequence: "users_id_seq"},
-	{name: "api_keys", columns: []string{"id", "user_id", "key_hash", "name", "created_at", "last_used_at"}, sequence: "api_keys_id_seq"},
-	{name: "sessions", columns: []string{"id", "token_hash", "user_id", "created_at", "last_seen_at", "expires_at", "user_agent"}, sequence: "sessions_id_seq"},
-	{name: "alerts", columns: []string{"id", "fingerprint", "name", "status", "labels", "annotations", "starts_at", "ends_at", "generator_url", "received_at", "archived_at", "resolution_source"}, jsonb: []string{"labels", "annotations"}, sequence: "alerts_id_seq"},
-	{name: "schedule_entries", columns: []string{"id", "user_id", "date", "created_at"}, sequence: "schedule_entries_id_seq"},
-	{name: "incidents", columns: []string{"id", "group_key", "title", "group_labels", "status", "severity", "triggered_at", "acknowledged_by", "acknowledged_at", "assigned_to", "snoozed_until", "resolved_at", "resolution_source", "archived_at"}, jsonb: []string{"group_labels"}, sequence: "incidents_id_seq"},
-	{name: "incident_alerts", columns: []string{"incident_id", "alert_id", "added_at"}},
-	{name: "incident_events", columns: []string{"id", "incident_id", "type", "user_id", "alert_id", "detail", "created_at"}, sequence: "incident_events_id_seq"},
-	{name: "notifications", columns: []string{"id", "incident_id", "user_id", "topic", "kind", "created_at", "send_after", "attempts", "sent_at", "last_error"}, sequence: "notifications_id_seq"},
-	{name: "incident_ack_tokens", columns: []string{"token_hash", "incident_id", "user_id", "created_at", "expires_at"}},
-}
-
-func main() {
-	sqlitePath := flag.String("sqlite", "", "path to the existing terdut SQLite database")
-	dsn := flag.String("dsn", "", "Postgres DSN of the migrated, empty database")
-	flag.Parse()
-
-	if *sqlitePath == "" || *dsn == "" {
-		log.Fatal("both -sqlite and -dsn are required")
-	}
-
-	src, err := sql.Open("sqlite", *sqlitePath)
-	if err != nil {
-		log.Fatalf("open sqlite: %v", err)
-	}
-	defer src.Close()
-
-	dst, err := sql.Open("pgx", *dsn)
-	if err != nil {
-		log.Fatalf("open postgres: %v", err)
-	}
-	defer dst.Close()
-
-	if err := dst.Ping(); err != nil {
-		log.Fatalf("ping postgres: %v", err)
-	}
-	if err := assertEmpty(dst); err != nil {
-		log.Fatalf("%v", err)
-	}
-
-	// One transaction for the whole copy: a run that dies half way leaves the
-	// target as it found it, rather than a partial database someone has to
-	// recognise as partial.
-	tx, err := dst.Begin()
-	if err != nil {
-		log.Fatalf("begin: %v", err)
-	}
-	defer tx.Rollback() //nolint:errcheck
-
-	for _, t := range tables {
-		n, err := copyTable(src, tx, t.name, t.columns, t.jsonb)
-		if err != nil {
-			log.Fatalf("copy %s: %v", t.name, err)
-		}
-		log.Printf("%-20s %d row(s)", t.name, n)
-	}
-
-	for _, t := range tables {
-		if t.sequence == "" {
-			continue
-		}
-		if err := advanceSequence(tx, t.sequence, t.name); err != nil {
-			log.Fatalf("advance %s: %v", t.sequence, err)
-		}
-	}
-
-	if err := tx.Commit(); err != nil {
-		log.Fatalf("commit: %v", err)
-	}
-	log.Print("done")
-}
-
-// assertEmpty refuses a target that already holds data, so running this twice
-// cannot duplicate anything.
-func assertEmpty(dst *sql.DB) error {
-	for _, t := range tables {
-		var n int64
-		if err := dst.QueryRow("SELECT COUNT(*) FROM " + t.name).Scan(&n); err != nil {
-			return fmt.Errorf("count %s (has the new server migrated this database?): %w", t.name, err)
-		}
-		if n > 0 {
-			return fmt.Errorf("%s already has %d row(s): the target must be empty", t.name, n)
-		}
-	}
-	return nil
-}
-
-func copyTable(src *sql.DB, tx *sql.Tx, table string, columns, jsonb []string) (int64, error) {
-	rows, err := src.Query("SELECT " + strings.Join(columns, ", ") + " FROM " + table)
-	if err != nil {
-		return 0, err
-	}
-	defer rows.Close()
-
-	insert := "INSERT INTO " + table + " (" + strings.Join(columns, ", ") + ") VALUES (" +
-		strings.Join(valuePlaceholders(columns, jsonb), ", ") + ")"
-
-	stmt, err := tx.Prepare(insert)
-	if err != nil {
-		return 0, err
-	}
-	defer stmt.Close()
-
-	var copied int64
-	for rows.Next() {
-		// Scanning into any lets the SQLite driver decide each column's Go type
-		// and hands it straight back to pgx, which is all this needs: the column
-		// types match on both sides, apart from the JSON casts above.
-		values := make([]any, len(columns))
-		targets := make([]any, len(columns))
-		for i := range values {
-			targets[i] = &values[i]
-		}
-		if err := rows.Scan(targets...); err != nil {
-			return copied, err
-		}
-		if _, err := stmt.Exec(values...); err != nil {
-			return copied, fmt.Errorf("insert row %d: %w", copied+1, err)
-		}
-		copied++
-	}
-	return copied, rows.Err()
-}
-
-// valuePlaceholders numbers the placeholders, casting the columns that became
-// jsonb: pgx sends a Go string as text, and Postgres will not assign text to a
-// jsonb column without being told.
-func valuePlaceholders(columns, jsonb []string) []string {
-	isJSON := make(map[string]bool, len(jsonb))
-	for _, c := range jsonb {
-		isJSON[c] = true
-	}
-	out := make([]string, len(columns))
-	for i, c := range columns {
-		out[i] = fmt.Sprintf("$%d", i+1)
-		if isJSON[c] {
-			out[i] += "::jsonb"
-		}
-	}
-	return out
-}
-
-// advanceSequence puts an identity sequence past the largest copied id. Without
-// it the first insert after the migration would reuse id 1.
-func advanceSequence(tx *sql.Tx, sequence, table string) error {
-	_, err := tx.Exec(fmt.Sprintf(
-		`SELECT setval('%s', COALESCE((SELECT MAX(id) FROM %s), 0) + 1, false)`,
-		sequence, table))
-	return err
-}