niklas/terdut-server
1
0
Fork 0
Code Issues 9 Pull Requests Actions Packages Projects Releases 61 Wiki Activity

Add notification channels and a provider abstraction

Browse Source 
Users get a notification_channels table (replacing users.ntfy_topic); pages fan
out to every enabled channel through a notify.Provider registry, with ntfy as
the first provider. Fallbacks are typed {type, address}. Schema edited in place:
existing databases must be recreated.

Claude-Session: https://claude.ai/code/session_016mBLURvJoMuUEr9cB2RpUN
This commit is contained in:
Niklas Ye
2026-10-12 21:00:40 +02:00
parent 6292ad6708
commit b114251f45
42 changed files with 2200 additions and 611 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/README.md
+2 -2
View File
@@ -11,8 +11,8 @@ The [README](../README.md) is the short tour. These pages hold the detail.
- [The web UI](./web-ui.md): sessions, the Team and Admin tabs.
- [Alertmanager configuration](./alertmanager.md): routes, integration keys and webhooks.
- [Alerts and incidents](./incidents.md): correlation, lifecycle, on-call assignment, stale-alert expiry.
- [Push notifications](./notifications.md): ntfy pages and acknowledging from them.
- [Escalation](./escalation.md): ladders, repeats and the fallback topic.
- [Notifications](./notifications.md): channels and providers, ntfy pages, and acknowledging from them.
- [Escalation](./escalation.md): ladders, repeats and the fallback.
- [Dead man's switches](./dead-mans-switch.md): noticing that alerts stopped arriving.
**Integrating and contributing**
docs/api.md
+14 -8
View File
@@ -116,7 +116,12 @@ on anybody's.
| `DELETE` | `/api/users/{id}` | **admin** | Delete user (cascades to keys). `409` for yourself or the last administrator |
| `PUT` | `/api/users/{id}/admin` | **admin** | Grant or revoke the administrator flag `{"is_admin"}`. `409` for yourself, the last administrator, or an administrator granted by single sign-on |
| `PUT` | `/api/users/{id}/disabled` | **admin** | Take an account out of use, or put it back `{"disabled"}`. `409` for yourself or the last administrator |
| `PUT` | `/api/users/{id}/notify` | self or admin | Set push notification target `{"ntfy_topic"}` — empty string clears it |
| `GET` | `/api/users/{id}/channels` | self or admin | The user's [notification channels](./notifications.md#channels) `[{id, type, address, enabled, created_at}]`. `address` is only sent to the owner: it can be a secret |
| `POST` | `/api/users/{id}/channels` | self or admin | Add a channel `{"type","address"}`. `400` for a type this server has no provider for or an address the provider rejects; `409` for a duplicate or at the limit of 10 |
| `PATCH` | `/api/users/{id}/channels/{channelID}` | self or admin | Turn it on or off, or change its address `{"enabled"?,"address"?}` |
| `DELETE` | `/api/users/{id}/channels/{channelID}` | self or admin | Remove it |
| `POST` | `/api/users/{id}/channels/{channelID}/test` | **self** | Send a test message through it. `502` with the provider's reason when it does not land; an administrator cannot send one to somebody else |
| `GET` | `/api/notify/providers` | any | Which kinds of channel this server can deliver `[{type, label, address_label, address_hint}]`. Empty when notifications are off |
| `PUT` | `/api/users/{id}/password` | self or admin | Set web UI password `{"password","current_password"}`. `current_password` is required only when changing your own existing password. Ends the user's other sessions |
| `POST` | `/api/users/{id}/api-keys` | self or admin | Issue API key `{"name"}` — key shown once |
| `DELETE` | `/api/users/{id}/api-keys/{keyID}` | self or admin | Revoke API key |
@@ -197,7 +202,7 @@ the team gets the same `404` as anybody else.
| `POST` | `/api/teams` | any | Create a team `{"name"}`; a human creator becomes its first owner. An instance-scoped [service account](#service-accounts) may also create one, and it gets no owner at all — expected for a team an operator is about to hand a team-scoped credential to, not an orphaned team a human made |
| `PUT` | `/api/teams/{teamID}` | **owner** | Rename it `{"name"}`. `409` if the name is taken |
| `DELETE` | `/api/teams/{teamID}` | **owner** | Delete a team and everything under it. `409` while it has open incidents |
| `GET` | `/api/teams/{teamID}/members` | member | Who is in the team, with `status` (`oncall` if the rota has them today, `unpageable` when a page to them would go nowhere — even if they are on call — else `reachable`), `on_call`, `next_shift` (first rota day after today), `pageable` and `problem` (`has no ntfy topic` / `account is disabled`; never the topic itself) and `last_active_at` (their newest session or API-key use). Every member sees the same list |
| `GET` | `/api/teams/{teamID}/members` | member | Who is in the team, with `status` (`oncall` if the rota has them today, `unpageable` when a page to them would go nowhere — even if they are on call — else `reachable`), `on_call`, `next_shift` (first rota day after today), `pageable` and `problem` (`has no notification channel` / `account is disabled`; never a channel's address) and `last_active_at` (their newest session or API-key use). Every member sees the same list |
| `POST` | `/api/teams/{teamID}/members` | **owner** | Add a member, or change their role `{"user_id","role"}`. `409` when it would demote the last owner, or the membership is managed by single sign-on |
| `DELETE` | `/api/teams/{teamID}/members/{userID}` | **owner** | Remove a member. `409` for the last owner, or a membership managed by single sign-on |
| `GET` | `/api/teams/{teamID}/oidc-groups` | member | Which groups control this team's membership: `{"member_group","owner_group"}`. An empty string means no group grants that role here |
@@ -209,8 +214,8 @@ the team gets the same `404` as anybody else.
| `GET` | `/api/teams/{teamID}/invites` | **owner** | The team's invite links, with their uses and expiry. Never the tokens |
| `POST` | `/api/teams/{teamID}/invites` | **owner** | Mint one `{"role","max_uses"}` — the full URL is returned once |
| `DELETE` | `/api/teams/{teamID}/invites/{inviteID}` | **owner** | Revoke a link before it expires |
| `GET` | `/api/teams/{teamID}/escalation` | member | The team's [escalation ladder](./escalation.md#escalation) `{repeat_count, fallback_topic, levels[], last_escalated_at?, last_escalated_incident_id?}`. Empty levels means the team has none. Each level also carries `status` (`ready`, `escalating` when an unanswered incident has climbed to it, `unreachable` when nobody on it could be woken), `waiting` (ids of the open incidents on it) and, per target, `username` (who it means today — the person on call, for a rota target), `reachable` and `problem`. The extra fields are output only; `PUT` takes the plain shape |
| `PUT` | `/api/teams/{teamID}/escalation` | **owner** | Replace it wholesale. `400` for a level with no targets or no timeout — a rung that pages nobody is a silence with a number on it |
| `GET` | `/api/teams/{teamID}/escalation` | member | The team's [escalation ladder](./escalation.md#escalation) `{repeat_count, fallback: {type, address}, levels[], last_escalated_at?, last_escalated_incident_id?}`. Empty levels means the team has none. Each level also carries `status` (`ready`, `escalating` when an unanswered incident has climbed to it, `unreachable` when nobody on it could be woken), `waiting` (ids of the open incidents on it) and, per target, `username` (who it means today — the person on call, for a rota target), `reachable` and `problem`. The extra fields are output only; `PUT` takes the plain shape |
| `PUT` | `/api/teams/{teamID}/escalation` | **owner** | Replace it wholesale (`fallback` is `{"type","address"}`, both empty for none). `400` for a level with no targets or no timeout, or a fallback no provider here can deliver — a rung that pages nobody is a silence with a number on it |
| `GET` | `/api/teams/{teamID}/deadman/switches` | member | The team's [dead man's switches](./dead-mans-switch.md), each `{id, name, matcher, timeout_seconds, severity, status, last_heartbeat_at, last_triggered_at, open_incident_id, sources[]}`. `status` is `healthy`, `dead` or `dormant`; `sources` has one entry per heartbeat fingerprint. Empty when the team watches nothing |
| `POST` | `/api/teams/{teamID}/deadman/switches` | **owner** | Add one: `{name?, matcher, timeout_seconds, severity?}`. `400` when the matcher names no `alertname` or holds several, or the timeout is not positive — a switch that silently watches nothing is the failure this feature exists to prevent |
| `PUT` | `/api/teams/{teamID}/deadman/switches/{switchID}` | **owner** | Replace one in place, same body and validation as create. Its id is unchanged — for an automated caller reconciling a spec change, unlike delete-and-recreate |
@@ -292,11 +297,12 @@ Types written today: `triggered`, `alert_added`, `alert_resolved`,
(absent on assignments made before they were recorded). New types may be added; render
unknown ones generically rather than dropping them.
On `notified` and `notify_failed`, `detail` carries the notification kind
(`triggered` | `reminder` | `resolved`), and on a failure the reason after it.
On `notified` and `notify_failed`, `detail` is `<kind> via <channel type>` —
kind is `triggered` | `reminder` | `escalated` | `resolved`, e.g. `triggered via ntfy` — and on a
failure the reason follows it. A person with two channels has one event per channel per page.
`user_id` is who was paged — absent means the page went to the shared fallback
topic and so belongs to nobody. The topic itself is never written to the
timeline: it is a shared secret with the ntfy server, and every API key can read
and so belongs to nobody. A channel's address is never written to the
timeline: it can be a shared secret (an ntfy topic is), and every API key can read
this.
## Alerts
docs/configuration.md
+2 -2
View File
@@ -21,9 +21,9 @@ over an administrator's edit.
| `TERDUT_DB_DSN` | — | **Required.** Postgres connection string, e.g. `postgres://terdut:secret@localhost:5432/terdut?sslmode=require` |
| `TERDUT_ARCHIVE_AFTER` | `168h` (7d) | **seed.** How long a resolved alert or incident stays in the default list before being auto-archived |
| `TERDUT_STALE_AFTER` | `6h` | **seed.** How long a firing alert may go without a refreshing webhook before it is treated as resolved — **must exceed your Alertmanager `repeat_interval`** |
| `TERDUT_NTFY_URL` | — | ntfy server to publish push notifications to. Empty disables notifications entirely |
| `TERDUT_NTFY_URL` | — | ntfy server to publish push notifications to; configures the `ntfy` [notification provider](./notifications.md). With no provider configured, notifications are off entirely |
| `TERDUT_NTFY_TOKEN` | — | Bearer token for an access-controlled ntfy |
| `TERDUT_NTFY_FALLBACK_TOPIC` | — | Topic used when nobody is on call |
| `TERDUT_NTFY_FALLBACK_TOPIC` | — | ntfy topic used when nobody on call can be reached |
| `TERDUT_PUBLIC_URL` | — | Base URL a phone uses to reach this server: the notification's link into the web UI, its Acknowledge button, and whether the session cookie is `Secure` |
| `TERDUT_NOTIFY_REPEAT` | `15m` | **seed.** How long an incident may sit unacknowledged before it is paged again. `0` notifies once and never repeats |
| `TERDUT_PASSWORD_LOGIN` | `true` | `false` refuses password login and password sign-up (`403`), leaving single sign-on the only way in. Refused at startup unless SSO is configured |
docs/escalation.md
+6 -5
View File
@@ -15,12 +15,12 @@ changes and nobody remembers to edit the policy.
level 1 5m oncall the rota gets first refusal
level 2 5m user:bob then a named second
then repeat_count more rounds
then the team's fallback topic, once
then the team's fallback, once
```
When a level's timeout passes with the incident still `triggered`, the next
level is paged. Off the end of the ladder the whole thing runs again
`repeat_count` times, and after that the team's `fallback_topic` is paged once
`repeat_count` times, and after that the team's `fallback` is paged once
as the end of the line. The incident stays open throughout: running out of
people to wake is not the same as somebody answering.
@@ -31,7 +31,7 @@ it is, and it resumes when the snooze runs out.
Every step is on the incident's timeline with the level and the names it woke,
so somebody reading it afterwards can tell why their phone rang at 04:00. A
level whose targets are all unreachable — no ntfy topic, a disabled account, an
level whose targets are all unreachable — no notification channel, a disabled account, an
empty rota — is recorded as `nobody reachable` and the ladder moves on rather
than stalling on a rung that cannot ring.
@@ -39,7 +39,8 @@ than stalling on a rung that cannot ring.
escalation; a team without keeps the reminder behaviour exactly as it was. Two
pages for one silence is the surest way to get a tool muted.
The ladder's `fallback_topic` is per team, unlike `TERDUT_NTFY_FALLBACK_TOPIC`,
which is the install-wide topic used when an incident opens with nobody on call.
The ladder's `fallback` is per team and names a provider and an address
(`{"type": "ntfy", "address": "..."}`), unlike `TERDUT_NTFY_FALLBACK_TOPIC`,
which is the install-wide ntfy topic used when an incident opens with nobody on call.
They answer different questions: one is "nobody was scheduled", the other is
"everybody scheduled has been tried".
docs/notifications.md
+60 -28
View File
@@ -1,27 +1,56 @@
# Push notifications
# Notifications
_Pages through ntfy, who gets them and how to acknowledge from the notification._ Back to the [README](../README.md) and the [documentation index](./README.md).
_How pages reach people: channels and providers, ntfy, who gets them and how to acknowledge from the notification._ Back to the [README](../README.md) and the [documentation index](./README.md).
With `TERDUT_NTFY_URL` set, an incident that opens is pushed to the on-call
person's phone through [ntfy](https://ntfy.sh). Everybody sets their own topic
under *Account* in the web UI, where a **Send a test push** button proves it
before an incident has to; `PUT /api/users/{id}/notify` is the same thing over
the API, and an administrator may set somebody else's. A user with no topic
falls back to `TERDUT_NTFY_FALLBACK_TOPIC`, as does an incident that opens with
nobody on call. If neither yields a topic, nothing is queued.
A page reaches a person through their **channels**. A channel is one way of
reaching them: a *provider* (the kind of thing that delivers — `ntfy` today)
and an *address* that provider understands (an ntfy topic). A person can have
several, and a page goes to **every channel that is switched on**, each queued
and retried independently, so one provider being down does not hold back the
others.
The **server** is the install's one ntfy, from `TERDUT_NTFY_URL`, and is not
something a user picks. Only the topic is per-person.
## Providers
A topic is a shared secret with the ntfy server: anyone who knows it can both
read the pages and publish to it, so an unguessable one is worth the trouble.
That is also why the topic never appears in an incident's timeline, which every
API key can read.
A provider is configured by the **server**, in its deployment, and is not
something a user picks. Only the kinds the server has been configured with can
be added as a channel: the Account page asks the server (`GET
/api/notify/providers`) and offers just those. With none configured,
notifications are off entirely: no goroutine, and nothing is ever queued.
Three things get pushed:
| Provider | Configured by | Address | Acknowledge button |
|---|---|---|---|
| `ntfy` | `TERDUT_NTFY_URL` (+ `TERDUT_NTFY_TOKEN`) | a topic | yes |
- **triggered** — an incident opened. Priority follows severity (`critical` maps
to ntfy's max priority, the one that overrides the phone's quiet settings).
A provider that cannot show a button gets the incident's link in the message
body instead. Adding a provider is one Go type in `internal/notify` and a line
in `cmd/terdut/main.go`; the notifier, queue, API and web UI do not change.
## Channels
Everybody manages their own under *Account* in the web UI: add a channel, send
a **test** through it (the point is that a phone buzzes, and only the person
holding it can tell), turn it off, remove it. `GET|POST
/api/users/{id}/channels` and `PATCH|DELETE .../{channelID}` are the same thing
over the API, and an administrator may manage somebody else's, but never sees
where they go: a channel's **address** is only sent to its owner, because it can
be a secret. An account holds at most 10 channels.
Turning a channel off is the off switch for it: pages skip it, including the
reminders of an incident that already paged it. A user with no usable channel
falls back to the install's fallback target (`TERDUT_NTFY_FALLBACK_TOPIC`), as
does an incident that opens with nobody on call. If neither yields a target,
nothing is queued.
An ntfy topic is a shared secret with the ntfy server: anyone who knows it can
both read the pages and publish to it, so an unguessable one is worth the
trouble. That is also why an address never appears in an incident's timeline,
which every API key can read — the timeline names the channel *type* only.
Three things get sent:
- **triggered** — an incident opened. Urgency follows severity (on ntfy,
`critical` maps to max priority, the one that overrides the phone's quiet
settings).
- **reminder** — the incident is still `triggered` after `TERDUT_NOTIFY_REPEAT`.
Repeats until somebody acts. Acknowledging, snoozing, resolving or archiving
all stop it — snooze is the mute button.
@@ -29,12 +58,12 @@ Three things get pushed:
whoever was paged in the first place, and only for the automatic cascade:
resolving by hand pushes nothing, since the person who did it already knows.
Notifications carry an **Acknowledge** button that acknowledges the incident
without opening anything. It POSTs to `/api/notify/ack/{token}`, an
Notifications on a provider that can show one carry an **Acknowledge** button
that acknowledges the incident without opening anything. It POSTs to `/api/notify/ack/{token}`, an
unauthenticated route authorised by the 256-bit token in its path — minted fresh
per notification, scoped to one incident and one action, and valid for 24 hours.
A real API key is never put in a notification, because the message is stored on
the ntfy server and cached on the device.
the provider's server and cached on the device.
The token is **not** consumed by use. Acknowledging is idempotent, so a token
stays valid for its full 24 hours and a second tap is a no-op that reports the
@@ -47,16 +76,19 @@ Two consequences worth planning for:
- `/api/notify/ack/{token}` **must stay publicly reachable**, or the button will
not work when the responder is off your network.
- Notifications sent to the fallback topic carry **no** Acknowledge button. The
topic is shared, and a button on it would let any subscriber acknowledge as
- Notifications sent to the fallback target carry **no** Acknowledge button. The
target is shared, and a button on it would let any recipient acknowledge as
somebody else.
Delivery is a queue, not an inline call: the webhook writes a row and a
background notifier sends it within 30 seconds, retrying with exponential
backoff up to 8 attempts. Nothing about ingestion blocks on ntfy being reachable.
Delivery is a queue, not an inline call: the webhook writes a row per channel
and a background notifier sends it within 30 seconds, retrying with exponential
backoff up to 8 attempts. Nothing about ingestion blocks on a provider being
reachable. Each row keeps a snapshot of its address, so a message already
queued still goes out if the channel is edited or removed; later reminders and
the all-clear follow the channels as they are then.
Every delivery is recorded on the incident's timeline: a `notified` event once
ntfy accepts the publish, and a `notify_failed` event when a notification
exhausts its retries. Written from the result rather than at enqueue, so the
the provider accepts it (`triggered via ntfy`), and a `notify_failed` event when
a notification exhausts its retries. Written from the result rather than at enqueue, so the
timeline says what actually happened — and a page that never landed is visible
instead of looking the same as one that did.
docs/web-ui.md
+1 -1
View File
@@ -5,7 +5,7 @@ _What the web UI offers, how sign-in and sessions work, and the Team and Admin t
The server serves a web UI at `/`: the incident queue, each incident's alerts
and timeline with every action (acknowledge, assign, snooze, note, resolve,
archive), who is on call, the alert feed, and an *Account* tab for your own
password and the ntfy topic your pages go to. It is built for a phone first. On a phone
password and the notification channels your pages go to. It is built for a phone first. On a phone
it navigates through a hamburger menu and has a sticky action bar, it follows the
system's dark mode, and it can be added to the home screen. From 900px wide it switches
to a sidebar with the queue and the incident side by side. The Stats page shows