Compare commits

..

18 Commits

Author SHA1 Message Date
Niklas Ye 4e8c52c28c Set the chart's placeholder version to 0.14.0
CI / test (push) Successful in 4s
CI / chart (push) Successful in 1s
CI / security (push) Successful in 12s
Release / test (push) Successful in 4s
Release / chart (push) Successful in 2s
Release / binaries (push) Successful in 24s
Release / image (push) Successful in 53s
Release / scan-image (push) Successful in 2s
Cosmetic, as in 4c85e76 and 041e159. `make helm-package` passes --version
and --app-version from the tag, so neither field decides anything about
what release.yaml publishes.

Done anyway because a tree heading for v0.14.0 that still says 0.13.0
tells a reader something false. appVersion keeps the v, per
APPVERSION_PREFIX in .release.conf.
2026-09-20 21:29:05 +02:00
niklas fb927aa67b Merge pull request 'Put a team's own settings in the web UI' (#18) from team-settings-ui into main
CI / test (push) Successful in 5s
CI / chart (push) Successful in 1s
CI / security (push) Successful in 11s
Reviewed-on: #18
2026-09-20 19:26:22 +00:00
Niklas Ye d728af53b1 Put a team's own settings in the web UI
CI / chart (pull_request) Successful in 1s
CI / security (pull_request) Successful in 13s
CI / test (pull_request) Successful in 2m19s
Closes #17. Everything a team owner configures was API-only: escalation,
integrations, dead man's switches, membership, and the rota -- which the
on-call view still described as the TUI's job, and the TUI has been broken
against this server since teams landed. Setting up the feature this whole
line of work exists for meant using curl.

A Team tab now holds all of it, one team at a time, with a picker for
somebody in more than one. An owner edits; a member sees the same page
without the controls, because the server refuses their writes anyway --
hiding a button is a courtesy to the reader, not the thing enforcing
anything.

The escalation editor holds a draft and sends the whole ladder, because
the API replaces it wholesale: the levels are an order, and patching one
rung leaves the numbering of the others undecided. Adding a level
defaults to five minutes and the rota, which is the shape almost every
ladder starts as.

An integration key is returned exactly once, so creating one opens a
panel that says so, shows the URL large with a copy button, and renders
the Alertmanager receiver snippet with the URL already in it -- the next
thing anybody does with that key is paste it into a config. The panel
stays until it is dismissed rather than disappearing on the next
re-render.

The incident view gains where an incident is on the ladder and when the
next page is due, which is the question somebody looking at an
unacknowledged incident actually has. The API carries it: the incident
payload now includes escalation_level and escalation_due_at, the latter
computed in the incident SELECT by joining the level's timeout, so a list
costs no extra queries.

Verified against a live server by making every call the page makes,
including the writes: the six reads the Team tab issues, a two-level
ladder saved and read back, an integration created and its key returned
once, three days of rota assigned, switches set, and an incident showing
level 1 with a due time five minutes out.

Claude-Session: https://claude.ai/code/session_01RHPj4ggeFdEjKKfm4SHbD7
2026-09-20 21:22:24 +02:00
Niklas Ye 53e5e03f4e Set the chart's placeholder version to 0.13.0
CI / test (push) Successful in 4s
CI / chart (push) Successful in 1s
CI / security (push) Successful in 11s
Release / test (push) Successful in 4s
Release / chart (push) Successful in 2s
Release / binaries (push) Successful in 18s
Release / image (push) Successful in 58s
Release / scan-image (push) Successful in 7s
Cosmetic, as in 4c85e76 and 041e159. `make helm-package` passes --version
and --app-version from the tag, so neither field decides anything about
what release.yaml publishes.

Done anyway because a tree heading for v0.13.0 that still says 0.12.0
tells a reader something false. appVersion keeps the v, per
APPVERSION_PREFIX in .release.conf.
2026-09-20 20:47:52 +02:00
niklas 4d62c1130b Merge pull request 'Page the next person when nobody answers' (#16) from escalation into main
CI / chart (push) Successful in 1s
CI / test (push) Successful in 6s
CI / security (push) Successful in 12s
Reviewed-on: #16
2026-09-20 16:43:03 +00:00
Niklas Ye 3183e7e5c5 Page the next person when nobody answers
CI / chart (pull_request) Successful in 1s
CI / security (pull_request) Successful in 16s
CI / test (pull_request) Successful in 2m21s
Closes #6, and closes the thing this whole line of work was opened for.
Until now an unacknowledged incident re-paged the same topic every
notify_repeat forever, which is a louder version of the same silence: if
the person on call is asleep, out of signal or has left the company,
nothing else happened.

A team can now configure an ordered ladder. Each level has a timeout and
a set of targets; a target is a named person or whoever the team's rota
says is on call today. That second kind is the one that keeps working
when the rota changes and nobody remembers to edit the policy. When a
level's timeout passes with the incident still triggered, the next level
is paged; off the end the chain repeats repeat_count times and then the
team's fallback topic is paged once. The incident stays open throughout,
because running out of people to wake is not somebody answering.

Escalation rides the notifier's existing 30-second tick and its outbox
rather than adding a second scheduler, and runs before delivery so a
level that comes due on a tick is paged on that tick. Each target gets
its own outbox row and therefore its own Acknowledge token: the button in
a notification must acknowledge as the person holding the phone, not as
whoever was paged first.

Acknowledging or resolving takes the incident off the ladder. Snoozing
pauses it -- a deliberate "not now" holds the ladder where it is and it
resumes when the snooze runs out, rather than carrying on without the
person who asked for quiet.

Reminders and escalation never both run. A team with a ladder gets
escalation; a team without keeps today's behaviour exactly. Both would
mean two pages for one silence, which is how a tool gets muted.

A level whose targets cannot be reached -- no topic, a disabled account,
an empty rota -- is entered anyway, recorded as "nobody reachable", and
the ladder moves on. Stalling on a rung that cannot ring would be the
failure this feature exists to prevent, wearing the feature's clothes. A
policy with such a level cannot be created, but an older row could hold
one.

The API replaces the ladder wholesale rather than patching a rung,
because the levels are an order: editing one has to answer what happens
to the numbering of the others, and a whole-ladder PUT makes that the
client's decision and the edit atomic.

Verified against a live server as well as in tests: alice paged, nobody
answers, bob paged, nobody answers, the fallback topic paged once and the
timeline reading "level 2: bob" then "escalation exhausted: paged
terdut-oncall-all" -- and a second incident acknowledged before its
timeout, which woke nobody else.

No UI yet. The team-settings screens for escalation, integrations and
dead man's switches are all still missing, and they are one piece of work
rather than three.

Claude-Session: https://claude.ai/code/session_01RHPj4ggeFdEjKKfm4SHbD7
2026-09-20 18:37:54 +02:00
niklas 94d23a593c Merge pull request 'Add an admin page, and move the behaviour settings into the database' (#15) from admin-settings into main
CI / test (push) Successful in 4s
CI / chart (push) Successful in 1s
CI / security (push) Successful in 11s
Reviewed-on: #15
2026-09-20 16:26:27 +00:00
Niklas Ye b0a02c010b Add an admin page, and move the behaviour settings into the database
CI / chart (pull_request) Successful in 1s
CI / security (pull_request) Successful in 13s
CI / test (pull_request) Successful in 2m1s
Closes #5. Three of the server's tunables were environment variables,
which meant changing how long an incident waits before being paged again
required editing a chart, merging it and waiting for a reconcile. They
are behaviour rather than infrastructure, and the difference is who needs
to change them and how often.

The split is by who owns the value. What stays in the environment is
where the server is plugged in: the listen address, the DSN, the ntfy URL
and token, the public URL. Those are needed before the database is open
and two of them are credentials -- the settings endpoint reports that
ntfy is configured and that a token is set, and never what either is.

What moves is how it behaves: the notify repeat interval, the stale
window and the archive window. The environment variable becomes the seed
rather than the setting, written once on first start and never
overwritten, so a redeploy cannot put a chart's default back over an
administrator's edit -- the rule the per-team dead man's switches already
follow. The loops read the current value per tick, so a change at 02:00
is obeyed at 02:00.

Key/value rather than a column per knob: #6 and #7 will both add
settings, and a table shaped one-column-per-setting needs a migration for
each. The cost is that values are text and the accessor has to say what
type it wanted, which settings.go does in one place. Unknown keys are
refused rather than stored -- a typo that wrote notify_repeat_second
would otherwise sit in the table looking like configuration and doing
nothing -- and each value has bounds loose enough to catch a slipped
decimal point without having an opinion about anybody's rota.

Disabling an account is new, and is not deleting one. Deleting a user
nulls acknowledged_by and assigned_to, which quietly rewrites who did
what during an incident months after the fact. A disabled user cannot
authenticate by either credential, loses their sessions immediately, and
stays the name on every acknowledgement they made. The check is part of
the lookup in serveAs rather than a test afterwards, so there is no path
where the row is loaded and the flag is then forgotten.

The page itself is a fourth tab, shown only to an administrator and only
as a courtesy: every endpoint under it is refused with 403 regardless, so
somebody who types /admin gets an explanation rather than a blank screen.
It lists teams with their size and open-incident count, users with their
flags, and the settings with their bounds -- plus the environment half,
read-only, so somebody hunting for the ntfy URL learns where it lives
instead of concluding the server has none.

Delete is disabled rather than offered-and-refused for a team with open
incidents, and neither admin action is offered on your own account, since
the server refuses both.

Claude-Session: https://claude.ai/code/session_01RHPj4ggeFdEjKKfm4SHbD7
2026-09-20 18:23:46 +02:00
niklas 303e7a3365 Merge pull request 'Remove the unauthenticated webhook and the SQLite migration script' (#14) from cleanup-after-teams into main
CI / chart (push) Successful in 1s
CI / test (push) Successful in 7s
CI / security (push) Successful in 16s
Reviewed-on: #14
2026-09-20 16:14:15 +00:00
Niklas Ye 7c87ae2af8 Remove the unauthenticated webhook and the SQLite migration script
CI / chart (pull_request) Successful in 1s
CI / security (pull_request) Successful in 13s
CI / test (pull_request) Successful in 1m54s
Both existed to carry an upgrade across, and both upgrades are done.

/api/alertmanager/webhook took no credential at all: anything able to
reach the port could open an incident for anybody. v0.12.0 kept it,
deprecated, so the teams release did not stop delivery while the
Alertmanager config was edited, and logged a line per payload asking to
be moved. The cluster's Alertmanager now posts on an integration key --
verified in the log, every two minutes, with no deprecation line since
the rollout -- so the door can be shut rather than left ajar until
somebody remembers. A sender still posting there gets the JSON 404 every
unknown /api path gets.

The tests move with it, which they should have done anyway: the harness
mints an integration key for the default team and posts on that, so they
exercise the path production uses rather than one only they still used.

scripts/sqlite-to-postgres.go goes the same way. It was written to be
temporary, it was the last thing needing modernc.org/sqlite, and this
install migrated on 2026-09-20. `go mod tidy` drops the driver and its
six transitive dependencies with it; the module graph is now chi, pgx,
pgerrcode and x/crypto. Anyone still on v0.10.x can take the script out
of the v0.12.0 tag, which the README now says.

Claude-Session: https://claude.ai/code/session_01RHPj4ggeFdEjKKfm4SHbD7
2026-09-20 18:11:30 +02:00
Niklas Ye 4c85e7646c Set the chart's placeholder version to 0.12.0
CI / test (push) Successful in 5s
CI / chart (push) Successful in 2s
CI / security (push) Successful in 15s
Release / test (push) Successful in 4s
Release / chart (push) Successful in 2s
Release / binaries (push) Successful in 20s
Release / image (push) Successful in 54s
Release / scan-image (push) Successful in 2s
Cosmetic, as in 041e159 and 989425e. `make helm-package` passes --version
and --app-version from the tag, so neither field decides anything about
what release.yaml publishes.

Done anyway because a tree heading for v0.12.0 that still says 0.11.1
tells a reader something false. appVersion keeps the v, per
APPVERSION_PREFIX in .release.conf.
2026-09-20 15:36:46 +02:00
niklas 05f82220a6 Merge pull request 'Per-team dead man's switches, and the UI's team badge, filter and cards' (#13) from teams into main
CI / test (push) Successful in 5s
CI / chart (push) Successful in 1s
CI / security (push) Successful in 11s
2026-09-20 13:29:09 +00:00
niklas 5227eb0d5f Merge pull request 'Give each team its own dead man's switches, and the UI a team to show' (#12) from deadman-per-team into teams
CI / test (pull_request) Successful in 5s
CI / chart (pull_request) Successful in 1s
CI / security (pull_request) Successful in 16s
2026-09-20 13:28:30 +00:00
Niklas Ye 74359c72ab Give each team its own dead man's switches, and the UI a team to show
CI / chart (pull_request) Successful in 1s
CI / security (pull_request) Successful in 14s
CI / test (pull_request) Successful in 1m57s
The rest of #4. Two halves that belong together because they are the
same sentence from opposite ends: a team decides which of its alerts are
heartbeats, and the UI has to be able to say which team it is talking
about.

Switches were three environment variables, which made them one setting
for the whole install. That was the last piece of the alerting path a
team could not control: it could take its own alerts on its own key and
still not say which of them were heartbeats, or how long a silence had
to last. They are a row per team now, edited by an owner through
PUT /api/teams/{teamID}/deadman, and the sweeper runs each team against
its own matchers, timeout and severity.

The environment variables become the starting point rather than the
setting. Every team without a configuration is seeded from them at
startup, so an upgrade keeps watching exactly what it was watching, and
SeedDeadmanConfigs never overwrites -- a redeploy must not put the
environment's value back over an owner's edit. A team created later
watches nothing until somebody says otherwise: inheriting an
install-wide heartbeat would page a new team about a source it has never
heard of, and a switch nobody chose is the kind that gets muted rather
than fixed.

A matcher string with no alertname in it is refused at the door instead
of stored. Storing it would produce a switch that watches nothing
silently, which is the exact failure the feature exists to prevent.

NewRouter and Sweep lose their DeadmanConfig parameter -- there is no
longer one answer to hand them. The type stays, because parsing a
matcher string is still parsing a matcher string.

The UI side: rows in the queue carry a team badge, the filter row gains
a team chip per team, and "on call now" shows one card per team. All
three appear only when the viewer is in more than one team -- otherwise
they are the same word repeated down a list, which is noise rather than
information, and the single-team install reads exactly as it did before
teams existed.

Verified against a live two-team server as well as in tests: the
combined queue labelled by team, the team_id filter, a heartbeat that is
a heartbeat in one team and an ordinary alert in another, and a new
team's switches starting empty while the upgraded team keeps the
environment's.

Claude-Session: https://claude.ai/code/session_01RHPj4ggeFdEjKKfm4SHbD7
2026-09-20 15:18:22 +02:00
niklas 43f69272f0 Merge pull request 'Add a system administrator role, and gate account management behind it' (#10) from admin-role into main
CI / test (push) Successful in 4s
CI / chart (push) Successful in 1s
CI / security (push) Successful in 11s
Reviewed-on: #10
2026-09-20 13:10:47 +00:00
niklas 2de5c8412d Merge pull request 'Scope everything to a team, and route alerts by integration key' (#11) from teams into admin-role
CI / test (pull_request) Successful in 4s
CI / chart (pull_request) Successful in 1s
CI / security (pull_request) Successful in 11s
Reviewed-on: #11
2026-09-20 11:39:33 +00:00
Niklas Ye a4fbd60441 Scope everything to a team, and route alerts by integration key
CI / chart (pull_request) Successful in 1s
CI / security (pull_request) Successful in 13s
CI / test (pull_request) Successful in 1m49s
The core of #4, and what #1 is for: terdut stops being one shared space.
A team owns its incidents, alerts, schedule and integrations; a user sees
exactly the teams they are in. Everything that existed moves into one
Default team and every existing user becomes an owner of it, so the
upgrade is a no-op for the people using it.

Ingestion is the load-bearing half. An alert arrives on a team's
integration key, and the key is both the credential and the routing: it
says that the sender may post, and which team the alerts belong to. That
also closes the unauthenticated webhook -- the old path stays for one
release, deprecated and routed to the oldest team, so an upgrade does not
stop delivering while somebody edits the Alertmanager config.

Scoping is enforced in as few places as possible, because the failure
mode is silent. serveAs loads the caller's memberships once; list queries
carry `team_id = ANY(...)`; and every incident route goes through
incidentIDParam, which now parses the id AND checks the team in the same
call, so a new handler cannot remember the first half and forget the
second. Anything in another team is 404, never 403: whether an incident
exists is that team's business.

Two bugs this found, both of which would have been silent:

  * upsertAlerts decided "is this a new occurrence" by looking up the
    fingerprint alone. Across teams that made team B's first alert look
    like a re-send of team A's, so it opened no incident at all. The
    lookups are keyed on (team_id, fingerprint) now, as the index is.

  * Every uniqueness rule was written for one tenant. Two teams watching
    two clusters legitimately see the same fingerprint, the same
    groupKey, and want somebody on call on the same day; all three
    constraints move to include team_id.

Roles inside a team are separate from the system administrator flag: an
owner configures the team, a member works its incidents, and an admin is
NOT implicitly in every team -- administration is about accounts, not
about reading other people's incidents. An admin can still repair a team
whose owner has left, which is why requireTeamOwner lets them through.

A shift can only be given to somebody in the team. Paging a person who
cannot open the incident is worse than paging nobody.

The UI is updated only as far as keeping it working: it loads the
viewer's teams with the session and uses the first one, since nobody has
a second yet. "On call now" shows every team the viewer is in, named only
when there is more than one, so the common case reads exactly as before.
The team switcher, badges and per-team settings pages are the next step.

Breaking for API clients: the schedule endpoints moved under the team,
and /api/schedule/current returns an array rather than an object or a
404. terdut-tui will need a version for that.

Per-team dead-man configuration is deliberately not here. A heartbeat's
incident already opens in the team whose key received it, which is the
part that matters for isolation; moving the matchers out of env into
per-team rows is a change to how deadman.go is configured rather than to
who sees what.

Claude-Session: https://claude.ai/code/session_01RHPj4ggeFdEjKKfm4SHbD7
2026-09-20 13:36:24 +02:00
Niklas Ye 1377d9005b Add a system administrator role, and gate account management behind it
CI / chart (pull_request) Successful in 2s
CI / security (pull_request) Successful in 16s
CI / test (pull_request) Successful in 1m37s
Until now every authenticated caller could create and delete users, set
anybody's password and mint anybody's API keys -- auth.go said so in a
comment. Defensible with one operator and a hand-made account; not once
people sign themselves up (#7), and not in a multi-tenant install (#4),
where the user list is no longer everybody who works here.

users.is_admin is the flag. AdminOnly gates creating and deleting users
and granting the flag itself. The endpoints that are self-service for
your own account and administration for somebody else's -- password,
ntfy topic, API keys -- go through requireSelfOrAdmin instead, because
which rule applies depends on the {id} in the path rather than on the
route.

Minting your own API key stays self-service. A key carries exactly the
rights of the user it belongs to, so issuing one is no more than signing
in again; requiring an admin for it would mean a responder cannot set up
the TUI without somebody else in the room.

/api/users stays readable by everybody. The queue's assignment control
and the on-call schedule both have to name people, and hiding the roster
from the people on it buys nothing.

THE MIGRATION MAKES EVERY EXISTING USER AN ADMINISTRATOR. They already
hold these powers, so nobody's access changes on upgrade: it names what
is already true and leaves demotion as a deliberate act. Promoting only
user 1 would silently strip the others, and could leave an install whose
only administrator is an account nobody has a password for.

Two guards keep an install administrable: the last administrator can be
neither deleted nor demoted, and nobody can delete or demote themselves
-- the likelier accident, where the only admin clears their own flag
while tidying up and locks the door behind them.

No UI changes: there are no account-management screens yet. models.User
carries is_admin (not omitempty, so a client can tell false from an old
server), which is what #5's admin page will render from.
2026-09-20 13:20:04 +02:00
52 changed files with 5394 additions and 514 deletions
+265 -40
View File
@@ -85,6 +85,16 @@ How a browser stays signed in:
With `TERDUT_PUBLIC_URL` set, tapping a push notification opens the incident in
the web UI (`/incidents/{id}`).
A **Team** tab holds everything a team owns: the on-call rota, the escalation
ladder, the alert sources with their keys, the dead man's switches and the
membership. An owner edits it; a member sees the same page read-only, because
the server refuses their writes anyway. Somebody in more than one team picks
between them at the top.
The **Admin** tab appears only for a system administrator, and holds what
belongs to the whole server rather than to one team: every team, every user, and
the settings that used to be environment variables.
### Docker
```bash
@@ -155,20 +165,33 @@ somewhere to exec. The sidecar, the PVC and the `backupSidecar` values are all g
## Configuration
Two kinds of setting, split by who changes them and how often.
**Where the server is plugged in** stays in the environment: the listen address,
the database DSN, the ntfy URL and token, the public URL. They are needed before
the database is open, and two of them are credentials.
**How the server behaves** lives in the database and is edited by an
administrator in the web UI or through `PUT /api/admin/settings`, taking effect
on the next sweep rather than at the next restart. The variables below marked
**seed** are the value each of those starts from: written once, on first start,
and never overwritten afterwards — a redeploy cannot put a chart's default back
over an administrator's edit.
| Variable | Default | Description |
|---|---|---|
| `TERDUT_ADDR` | `:8080` | TCP address to listen on |
| `TERDUT_DB_DSN` | — | **Required.** Postgres connection string, e.g. `postgres://terdut:secret@localhost:5432/terdut?sslmode=require` |
| `TERDUT_ARCHIVE_AFTER` | `168h` (7d) | How long a resolved alert or incident stays in the default list before being auto-archived |
| `TERDUT_STALE_AFTER` | `6h` | How long a firing alert may go without a refreshing webhook before it is treated as resolved — **must exceed your Alertmanager `repeat_interval`** |
| `TERDUT_DEADMAN_MATCHERS` | `alertname=Watchdog` | Which alerts are [dead man's switches](#dead-mans-switch). `;` separates matchers, `,` the label conditions within one, `=` is exact equality. Every matcher must name an `alertname` |
| `TERDUT_ARCHIVE_AFTER` | `168h` (7d) | **seed.** How long a resolved alert or incident stays in the default list before being auto-archived |
| `TERDUT_STALE_AFTER` | `6h` | **seed.** How long a firing alert may go without a refreshing webhook before it is treated as resolved — **must exceed your Alertmanager `repeat_interval`** |
| `TERDUT_DEADMAN_MATCHERS` | `alertname=Watchdog` | The **default** matchers a team starts with — switches are per team now, and this seeds teams that have no configuration of their own. `;` separates matchers, `,` the label conditions within one, `=` is exact equality. Every matcher must name an `alertname` |
| `TERDUT_DEADMAN_TIMEOUT` | `15m` | How long a heartbeat may go unheard before its switch is declared dead — **must be shorter than the `repeat_interval` of the route carrying it**. `0` disables dead man's switch handling |
| `TERDUT_DEADMAN_SEVERITY` | `critical` | Severity a dead man's switch incident opens at |
| `TERDUT_NTFY_URL` | — | ntfy server to publish push notifications to. Empty disables notifications entirely |
| `TERDUT_NTFY_TOKEN` | — | Bearer token for an access-controlled ntfy |
| `TERDUT_NTFY_FALLBACK_TOPIC` | — | Topic used when nobody is on call |
| `TERDUT_PUBLIC_URL` | — | Base URL a phone uses to reach this server: the notification's link into the web UI, its Acknowledge button, and whether the session cookie is `Secure` |
| `TERDUT_NOTIFY_REPEAT` | `15m` | How long an incident may sit unacknowledged before it is paged again. `0` notifies once and never repeats |
| `TERDUT_NOTIFY_REPEAT` | `15m` | **seed.** How long an incident may sit unacknowledged before it is paged again. `0` notifies once and never repeats |
Durations use Go syntax (`30m`, `12h`, `168h`). An unparseable value falls back to the default.
@@ -182,19 +205,39 @@ In the Helm chart the two sweeper durations are set via `sweeper.staleAfter` and
## Alertmanager configuration
Add terdut-server as a webhook receiver in your `alertmanager.yml`:
Alerts arrive on a team's **integration key**, which says both that the sender
may post and which team the alerts belong to. Mint one as an owner of the team:
```bash
curl -X POST https://terdut.example.com/api/teams/1/integrations \
-H "Authorization: Bearer $TERDUT_API_KEY" \
-H 'Content-Type: application/json' \
-d '{"name":"prod alertmanager"}'
```
The response carries the key and the full URL **once**; only a SHA-256 hash is
stored. Put it in your `alertmanager.yml`:
```yaml
receivers:
- name: terdut
webhook_configs:
- url: http://terdut-server:8080/api/alertmanager/webhook
- url: http://terdut-server:8080/api/integrations/<key>/alertmanager
send_resolved: true
route:
receiver: terdut
```
The whole URL is a credential, so treat it like one. Alertmanager 0.26 and
later can read it from a file with `url_file:` instead, which keeps it out of
your configuration repository:
```yaml
- url_file: /etc/alertmanager/secrets/terdut-webhook-url/url
send_resolved: true
```
The webhook endpoint requires no authentication.
If you use the [dead man's switch](#dead-mans-switch) — and the default configuration does — give
@@ -344,6 +387,50 @@ exhausts its retries. Written from the result rather than at enqueue, so the
timeline says what actually happened — and a page that never landed is visible
instead of looking the same as one that did.
### Escalation
Without a ladder, an unacknowledged incident re-pages the same topic every
`notify_repeat` forever. That is a louder version of the same silence: if the
person on call is asleep, out of signal, or has left, nothing else happens.
A team can configure an ordered ladder instead. Each level has a timeout and a
set of targets, and a target is either a named person or **whoever the team's
rota says is on call today** — the target that keeps working when the rota
changes and nobody remembers to edit the policy.
```
level 1 5m oncall the rota gets first refusal
level 2 5m user:bob then a named second
then repeat_count more rounds
then the team's fallback topic, once
```
When a level's timeout passes with the incident still `triggered`, the next
level is paged. Off the end of the ladder the whole thing runs again
`repeat_count` times, and after that the team's `fallback_topic` is paged once
as the end of the line. The incident stays open throughout: running out of
people to wake is not the same as somebody answering.
**Acknowledging or resolving stops it**, which is the point — continuing to wake
people after somebody has said "I have this" is how a tool teaches people to
mute it. **Snoozing pauses it**: a deliberate "not now" holds the ladder where
it is, and it resumes when the snooze runs out.
Every step is on the incident's timeline with the level and the names it woke,
so somebody reading it afterwards can tell why their phone rang at 04:00. A
level whose targets are all unreachable — no ntfy topic, a disabled account, an
empty rota — is recorded as `nobody reachable` and the ladder moves on rather
than stalling on a rung that cannot ring.
**Reminders and escalation never both run.** A team with a ladder gets
escalation; a team without keeps the reminder behaviour exactly as it was. Two
pages for one silence is the surest way to get a tool muted.
The ladder's `fallback_topic` is per team, unlike `TERDUT_NTFY_FALLBACK_TOPIC`,
which is the install-wide topic used when an incident opens with nobody on call.
They answer different questions: one is "nobody was scheduled", the other is
"everybody scheduled has been tried".
### Stale alert expiry
A resolved webhook is the only signal that an alert has stopped firing, so a
@@ -378,11 +465,23 @@ kube-prometheus-stack already ships the alert for this. `Watchdog` is
nothing unless something downstream notices it stop. That is what
`TERDUT_DEADMAN_MATCHERS` defaults to.
**Switches belong to a team**, which decides which of its own alerts are
heartbeats and how long a silence has to last. An owner sets them through
`PUT /api/teams/{teamID}/deadman`; a missed heartbeat opens an incident in the
team whose integration received it.
The environment variables are the starting point, not the setting: at startup
every team **without** a configuration of its own is given one from them, and an
owner's later edit is never overwritten by a redeploy. A team created after
that starts watching nothing until its owner says otherwise — inheriting an
install-wide heartbeat would page a new team about a source it has never heard
of.
A matcher is a set of exact label conditions, one of which must be the
`alertname`:
`alertname`, in the same format the environment variable uses:
```
TERDUT_DEADMAN_MATCHERS="alertname=Watchdog,cluster=prod; alertname=EdgeHeartbeat"
alertname=Watchdog,cluster=prod; alertname=EdgeHeartbeat
```
**The unit of monitoring is the fingerprint, not the alert name.** Two clusters
@@ -435,7 +534,7 @@ of the last heartbeat, and the heartbeat's labels are on the incident's
### Authentication
All endpoints except `/api/bootstrap`, `/api/alertmanager/webhook`,
All endpoints except `/api/bootstrap`, `/api/integrations/{key}/alertmanager`,
`/api/notify/ack/{token}`, `/api/login` and `/api/logout` require either an API key:
```
@@ -445,6 +544,30 @@ Authorization: Bearer <api-key>
or the web UI's session cookie. A request that carries an `Authorization` header
is judged on that header alone.
Two kinds of user exist. An **administrator** manages accounts: creating and
deleting users, setting anybody's password, minting keys for anybody, and
granting the flag itself. Everybody else works incidents — acknowledging,
assigning, snoozing, resolving, noting — and manages their own account and
nobody else's. An API key carries exactly the rights of the user it belongs to.
The first user, from `/api/bootstrap`, is an administrator. Users created
afterwards are not, until an administrator says so. An install always keeps at
least one: the last administrator can be neither deleted nor demoted, and
nobody can delete or demote themselves.
Endpoints that require the flag answer `403` with
`{"error":"administrator access required"}`.
**Teams** are the unit of tenancy, and are a separate axis from the administrator
flag. A team owns its incidents, alerts, schedule and integrations, and a user
sees exactly the teams they belong to — an administrator is not implicitly in
every team, because administration is about accounts, not about reading other
people's incidents. Within a team an **owner** configures it (schedule,
integrations, membership) and a **member** works its incidents.
Anything belonging to a team you are not in answers `404`, not `403`: whether an
incident exists is itself something only its team should learn.
| Method | Path | Description |
|---|---|---|
| `POST` | `/api/login` | `{"username","password"}` → sets the session cookie, returns `{user, has_password}`. `429` after too many failures |
@@ -455,20 +578,64 @@ is judged on that header alone.
| Method | Path | Description |
|---|---|---|
| `POST` | `/api/bootstrap` | Create first user + API key `{"username","email","password"?}` (only works on empty DB) |
| `GET` | `/api/users` | List users |
| `POST` | `/api/users` | Create user `{"username","email"}` |
| `DELETE` | `/api/users/{id}` | Delete user (cascades to keys) |
| `PUT` | `/api/users/{id}/notify` | Set push notification target `{"ntfy_topic"}` — empty string clears it |
| `PUT` | `/api/users/{id}/password` | Set web UI password `{"password","current_password"}`. `current_password` is required only when changing your own existing password. Ends the user's other sessions |
| `POST` | `/api/users/{id}/api-keys` | Issue API key `{"name"}` — key shown once |
| `DELETE` | `/api/users/{id}/api-keys/{keyID}` | Revoke API key |
**admin** marks an endpoint that requires the administrator flag; **self or
admin** marks one you may use on your own account and an administrator may use
on anybody's.
| Method | Path | Who | Description |
|---|---|---|---|
| `POST` | `/api/bootstrap` | — | Create first user + API key `{"username","email","password"?}` (only works on empty DB). The user is an administrator |
| `GET` | `/api/users` | any | List users. Open to everybody: the queue's assignment control and the schedule both have to name people |
| `POST` | `/api/users` | **admin** | Create user `{"username","email"}`. Not an administrator |
| `DELETE` | `/api/users/{id}` | **admin** | Delete user (cascades to keys). `409` for yourself or the last administrator |
| `PUT` | `/api/users/{id}/admin` | **admin** | Grant or revoke the administrator flag `{"is_admin"}`. `409` for yourself or the last administrator |
| `PUT` | `/api/users/{id}/disabled` | **admin** | Take an account out of use, or put it back `{"disabled"}`. `409` for yourself or the last administrator |
| `PUT` | `/api/users/{id}/notify` | self or admin | Set push notification target `{"ntfy_topic"}` — empty string clears it |
| `PUT` | `/api/users/{id}/password` | self or admin | Set web UI password `{"password","current_password"}`. `current_password` is required only when changing your own existing password. Ends the user's other sessions |
| `POST` | `/api/users/{id}/api-keys` | self or admin | Issue API key `{"name"}` — key shown once |
| `DELETE` | `/api/users/{id}/api-keys/{keyID}` | self or admin | Revoke API key |
### Administration
| Method | Path | Who | Description |
|---|---|---|---|
| `GET` | `/api/admin/teams` | **admin** | Every team on the server, with its member and open-incident counts. `/api/teams` answers "what am I in"; this answers "what is there" |
| `GET` | `/api/admin/settings` | **admin** | The editable settings with their bounds, plus the environment-configured ones, read-only. Never credentials |
| `PUT` | `/api/admin/settings` | **admin** | Change one or more `{"key": seconds}`. `400` for an unknown key or a value outside its bounds |
### Alert ingestion
Alerts arrive on a team's integration key. The key is both the credential and the
routing: it says that the sender may post, and which team the alerts belong to.
Create one with `POST /api/teams/{teamID}/integrations`, which returns the key
and the full URL once and stores only a SHA-256 hash.
| Method | Path | Description |
|---|---|---|
| `POST` | `/api/alertmanager/webhook` | Alertmanager v4 webhook receiver (no auth) |
| `POST` | `/api/integrations/{key}/alertmanager` | Alertmanager v4 webhook receiver for the key's team. `401` for an unknown key |
This is the only way in. The pre-teams `POST /api/alertmanager/webhook` took no
credential at all — anything able to reach the port could open an incident —
and was removed in v0.13.0 once senders had moved onto keys.
### Teams
| Method | Path | Who | Description |
|---|---|---|---|
| `GET` | `/api/teams` | any | The caller's own teams, each with their role |
| `POST` | `/api/teams` | any | Create a team `{"name"}`; the creator becomes its first owner |
| `PUT` | `/api/teams/{teamID}` | **owner** | Rename it `{"name"}`. `409` if the name is taken |
| `DELETE` | `/api/teams/{teamID}` | **owner** | Delete a team and everything under it. `409` while it has open incidents |
| `GET` | `/api/teams/{teamID}/members` | member | Who is in the team |
| `POST` | `/api/teams/{teamID}/members` | **owner** | Add a member, or change their role `{"user_id","role"}` |
| `DELETE` | `/api/teams/{teamID}/members/{userID}` | **owner** | Remove a member. `409` for the last owner |
| `GET` | `/api/teams/{teamID}/integrations` | member | List integrations. Never returns keys |
| `POST` | `/api/teams/{teamID}/integrations` | **owner** | Mint an integration `{"name","kind"}` — key and URL shown once |
| `DELETE` | `/api/teams/{teamID}/integrations/{integrationID}` | **owner** | Revoke an integration |
| `GET` | `/api/teams/{teamID}/escalation` | member | The team's [escalation ladder](#escalation) `{repeat_count, fallback_topic, levels[]}`. Empty levels means the team has none |
| `PUT` | `/api/teams/{teamID}/escalation` | **owner** | Replace it wholesale. `400` for a level with no targets or no timeout — a rung that pages nobody is a silence with a number on it |
| `GET` | `/api/teams/{teamID}/deadman` | member | The team's [dead man's switch](#dead-mans-switch) configuration `{matchers, timeout_seconds, severity}` |
| `PUT` | `/api/teams/{teamID}/deadman` | **owner** | Replace it. `400` when no matcher names an `alertname`, because a switch that silently watches nothing is the failure this feature exists to prevent |
### Notifications
@@ -655,13 +822,23 @@ unknown" rather than being rejected.
| Method | Path | Description |
|---|---|---|
| `POST` | `/api/schedule` | Assign user to dates `{"user_id", "dates":["YYYY-MM-DD",...], "replace"}` — all-or-nothing |
| `GET` | `/api/schedule` | List entries. Filters: `?from=YYYY-MM-DD`, `?to=YYYY-MM-DD` |
| `GET` | `/api/schedule/current` | Today's on-call user (UTC), 404 if none |
| `DELETE` | `/api/schedule/{id}` | Remove schedule entry |
Each team keeps its own rota, so two teams can have two different people on call
on the same day. The person taking a shift has to be in the team — paging
somebody who cannot open the incident is worse than paging nobody.
| Method | Path | Who | Description |
|---|---|---|---|
| `POST` | `/api/teams/{teamID}/schedule` | **owner** | Assign user to dates `{"user_id", "dates":["YYYY-MM-DD",...], "replace"}` — all-or-nothing |
| `GET` | `/api/teams/{teamID}/schedule` | member | List entries. Filters: `?from=YYYY-MM-DD`, `?to=YYYY-MM-DD` |
| `DELETE` | `/api/teams/{teamID}/schedule/{id}` | **owner** | Remove schedule entry |
| `GET` | `/api/schedule/current` | any | Who is on call today (UTC) in **every** team the caller is in — one entry per team, `[]` when nobody anywhere |
### Statistics
Every figure counts the caller's own teams only: a report that counted other
teams' incidents would leak their volume, and their alert names through the
top-alerts list, and would not be a number about the reader's work anyway.
All stat endpoints accept optional `?from=YYYY-MM-DD` and `?to=YYYY-MM-DD`, and exclude archived rows to match the default list views. Alert stats filter on `received_at`; incident stats filter on `triggered_at`.
| Method | Path | Description |
@@ -678,32 +855,80 @@ averages over incidents that have actually been acknowledged or resolved, and ar
---
## Upgrading from SQLite
## Upgrading to teams
Versions up to v0.10.2 stored everything in a SQLite file. From the Postgres release onwards
the server needs `TERDUT_DB_DSN` and keeps nothing on disk.
Everything that existed before teams moves into one team called **Default**, and
every existing user becomes an owner of it. The upgrade is a no-op for the
people using it: the same queue, the same schedule, the same incidents, with a
name on them.
The cutover is ordered — the server must not be running while the copy happens:
What changes, and will need attention:
- **Alert ingestion moved.** Mint a key with
`POST /api/teams/{teamID}/integrations` and point Alertmanager at the URL it
returns. In v0.12.0 the old `POST /api/alertmanager/webhook` still worked,
deprecated, routing everything to the oldest team; **v0.13.0 removes it**, so
upgrade straight from v0.11.x to v0.13.0 only after the senders are moved.
- **The schedule endpoints moved** under `/api/teams/{teamID}/schedule`, and
editing the rota is now an owner's job. `GET /api/schedule/current` stayed
where it was but now returns an **array** — one entry per team with somebody
on call — instead of a single object or a 404. This is a breaking API change
for anything that reads it, terdut-tui included.
- **Uniqueness is per team now.** Two teams can legitimately see the same alert
fingerprint, the same Alertmanager groupKey, and put somebody on call on the
same date.
**Dead man's switches moved too.** `TERDUT_DEADMAN_MATCHERS`, `_TIMEOUT` and
`_SEVERITY` are no longer the setting; they are the default each existing team
is seeded with at startup, after which an owner edits them per team through
`PUT /api/teams/{teamID}/deadman` and a redeploy never overwrites that.
Nothing else about an incident changes, and incidents never move between teams:
an alert belongs to whichever team's key it arrived on.
## Upgrading to roles
Before this release every authenticated caller could create and delete users,
set anybody's password and mint anybody's API keys. That is now the
administrator flag, and the migration **makes every existing user an
administrator** — they already held those powers, so nobody's access changes on
upgrade and demotion is a deliberate act afterwards. Promoting only the first
user would have silently stripped the rest, and could leave an install whose
only administrator is an account nobody has a password for.
Users created after the upgrade are not administrators. Hand the flag out with:
```bash
# 1. Stop the old server, keeping its database file.
# 2. Create an empty Postgres database, then let the new binary build the schema:
TERDUT_DB_DSN='postgres://terdut:secret@localhost:5432/terdut?sslmode=disable' ./terdut &
# ...watch for "listening on", then stop it again.
# 3. Copy the data across:
go run -tags migrate ./scripts/sqlite-to-postgres.go \
-sqlite /data/terdut.db \
-dsn 'postgres://terdut:secret@localhost:5432/terdut?sslmode=disable'
# 4. Start the new server for good.
curl -X PUT https://terdut.example.com/api/users/7/admin \
-H "Authorization: Bearer $TERDUT_API_KEY" \
-H 'Content-Type: application/json' \
-d '{"is_admin": true}'
```
Nothing in the API changed shape, so terdut-tui needs no new version — but a
non-administrator now gets `403` where a `200` used to come back.
## Upgrading from SQLite
Versions up to v0.10.2 stored everything in a SQLite file. From v0.11.1 the server needs
`TERDUT_DB_DSN` and keeps nothing on disk.
The copy was done by `scripts/sqlite-to-postgres.go`, which **was deleted in v0.13.0** along
with the SQLite driver it was the last user of. It is still in the history — check out the
`v0.12.0` tag to get it:
```bash
git show v0.12.0:scripts/sqlite-to-postgres.go > sqlite-to-postgres.go
```
The cutover is ordered, and the server must not be running while the copy happens: stop the
old version, let the new binary build the schema against an empty Postgres, run the script
with `-sqlite` and `-dsn`, then start the new version for good. On Kubernetes step three runs
as a Job with the same image against the PVC before it is removed.
The copy preserves every id, so incidents keep their numbers and the timeline, alert
membership, outbox and ack tokens all still point where they did. It refuses a target that
already has rows, so a second run cannot double-insert. On Kubernetes, step 3 runs as a Job
with the same image against the PVC before it is removed.
The script is deliberately temporary: it is the only thing left that needs the SQLite driver,
and both should be deleted once the installs that need them have migrated.
already has rows, so a second run cannot double-insert.
## Upgrading to incidents
+2 -2
View File
@@ -15,5 +15,5 @@ type: application
# appVersion and image.tag in values.yaml no longer agree, and that is not an oversight:
# image.tag stays "latest", which is what a local install actually pulls. appVersion is
# metadata and drives nothing.
version: 0.11.1
appVersion: "v0.11.1"
version: 0.14.0
appVersion: "v0.14.0"
+15 -2
View File
@@ -36,9 +36,22 @@ func main() {
RepeatEvery: cfg.NotifyRepeat,
}
// Dead man's switches live per team now. The environment variables are the
// defaults a team starts from: every team without a configuration of its
// own gets one from them here, and an owner's later edit is never
// overwritten by a redeploy.
deadman := api.ParseDeadmanConfig(cfg.DeadmanMatchers, cfg.DeadmanTimeout, cfg.DeadmanSeverity)
if err := api.SeedDeadmanConfigs(context.Background(), database, deadman); err != nil {
log.Fatalf("seed dead man's switch defaults: %v", err)
}
router := api.NewRouter(database, notify, deadman)
// The behaviour knobs move into the database on first start, after which an
// administrator owns them and a redeploy leaves them alone.
if err := api.SeedSettings(context.Background(), database, cfg); err != nil {
log.Fatalf("seed settings: %v", err)
}
router := api.NewRouter(database, notify, cfg)
srv := &http.Server{
Addr: cfg.Addr,
@@ -51,7 +64,7 @@ func main() {
ctx, stop := signal.NotifyContext(context.Background(), syscall.SIGINT, syscall.SIGTERM)
defer stop()
go api.StartArchiver(ctx, database, cfg.ArchiveAfter, cfg.StaleAfter, deadman, notify)
go api.StartArchiver(ctx, database, cfg.ArchiveAfter, cfg.StaleAfter, notify)
go api.StartNotifier(ctx, database, notify)
go func() {
-10
View File
@@ -7,22 +7,12 @@ require (
github.com/jackc/pgerrcode v0.0.0-20250907135507-afb5586c32a6
github.com/jackc/pgx/v5 v5.11.0
golang.org/x/crypto v0.55.0
modernc.org/sqlite v1.50.1
)
require (
github.com/dustin/go-humanize v1.0.1 // indirect
github.com/google/uuid v1.6.0 // indirect
github.com/jackc/pgpassfile v1.0.0 // indirect
github.com/jackc/pgservicefile v0.0.0-20240606120523-5a60cdf6a761 // indirect
github.com/jackc/puddle/v2 v2.2.2 // indirect
github.com/mattn/go-isatty v0.0.20 // indirect
github.com/ncruces/go-strftime v1.0.0 // indirect
github.com/remyoudompheng/bigfft v0.0.0-20230129092748-24d4a6f8daec // indirect
golang.org/x/sync v0.22.0 // indirect
golang.org/x/sys v0.47.0 // indirect
golang.org/x/text v0.41.0 // indirect
modernc.org/libc v1.72.3 // indirect
modernc.org/mathutil v1.7.1 // indirect
modernc.org/memory v1.11.0 // indirect
)
-49
View File
@@ -1,16 +1,8 @@
github.com/davecgh/go-spew v1.1.0/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c=
github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
github.com/dustin/go-humanize v1.0.1 h1:GzkhY7T5VNhEkwH0PVJgjz+fX1rhBrR7pRT3mDkpeCY=
github.com/dustin/go-humanize v1.0.1/go.mod h1:Mu1zIs6XwVuF/gI1OepvI0qD18qycQx+mFykh5fBlto=
github.com/go-chi/chi/v5 v5.2.5 h1:Eg4myHZBjyvJmAFjFvWgrqDTXFyOzjj7YIm3L3mu6Ug=
github.com/go-chi/chi/v5 v5.2.5/go.mod h1:X7Gx4mteadT3eDOMTsXzmI4/rwUpOwBHLpAfupzFJP0=
github.com/google/pprof v0.0.0-20250317173921-a4b03ec1a45e h1:ijClszYn+mADRFY17kjQEVQ1XRhq2/JR1M3sGqeJoxs=
github.com/google/pprof v0.0.0-20250317173921-a4b03ec1a45e/go.mod h1:boTsfXsheKC2y+lKOCMpSfarhxDeIzfZG1jqGcPl3cA=
github.com/google/uuid v1.6.0 h1:NIvaJDMOsjHA8n1jAhLSgzrAzy1Hgr+hNrb57e+94F0=
github.com/google/uuid v1.6.0/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo=
github.com/hashicorp/golang-lru/v2 v2.0.7 h1:a+bsQ5rvGLjzHuww6tVxozPZFVghXaHOwFs4luLUK2k=
github.com/hashicorp/golang-lru/v2 v2.0.7/go.mod h1:QeFd9opnmA6QUJc5vARoKUSoFhyfM2/ZepoAG6RGpeM=
github.com/jackc/pgerrcode v0.0.0-20250907135507-afb5586c32a6 h1:D/V0gu4zQ3cL2WKeVNVM4r2gLxGGf6McLwgXzRTo2RQ=
github.com/jackc/pgerrcode v0.0.0-20250907135507-afb5586c32a6/go.mod h1:a/s9Lp5W7n/DD0VrVoyJ00FbP2ytTPDVOivvn2bMlds=
github.com/jackc/pgpassfile v1.0.0 h1:/6Hmqy13Ss2zCq62VdNG8tM1wchn8zjSGOBJ6icpsIM=
@@ -21,14 +13,8 @@ github.com/jackc/pgx/v5 v5.11.0 h1:IzBBtyK9AHqf98cctWFifYSci2hgQR/cd56wB4p+ogg=
github.com/jackc/pgx/v5 v5.11.0/go.mod h1:mal1tBGAFfLHvZzaYh77YS/eC6IX9OWbRV1QIIM0Jn4=
github.com/jackc/puddle/v2 v2.2.2 h1:PR8nw+E/1w0GLuRFSmiioY6UooMp6KJv0/61nB7icHo=
github.com/jackc/puddle/v2 v2.2.2/go.mod h1:vriiEXHvEE654aYKXXjOvZM39qJ0q+azkZFrfEOc3H4=
github.com/mattn/go-isatty v0.0.20 h1:xfD0iDuEKnDkl03q4limB+vH+GxLEtL/jb4xVJSWWEY=
github.com/mattn/go-isatty v0.0.20/go.mod h1:W+V8PltTTMOvKvAeJH7IuucS94S2C6jfK/D7dTCTo3Y=
github.com/ncruces/go-strftime v1.0.0 h1:HMFp8mLCTPp341M/ZnA4qaf7ZlsbTc+miZjCLOFAw7w=
github.com/ncruces/go-strftime v1.0.0/go.mod h1:Fwc5htZGVVkseilnfgOVb9mKy6w1naJmn9CehxcKcls=
github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM=
github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4=
github.com/remyoudompheng/bigfft v0.0.0-20230129092748-24d4a6f8daec h1:W09IVJc94icq4NjY3clb7Lk8O1qJ8BdBEF8z0ibU0rE=
github.com/remyoudompheng/bigfft v0.0.0-20230129092748-24d4a6f8daec/go.mod h1:qqbHyh8v60DhA7CoWK5oRCqLrMHRGoxYCSS9EjAz6Eo=
github.com/stretchr/objx v0.1.0/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME=
github.com/stretchr/testify v1.3.0/go.mod h1:M5WIy9Dh21IEIfnGCwXGc5bZfKNJtfHm1UVUgZn+9EI=
github.com/stretchr/testify v1.7.0/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/h/Wwjteg=
@@ -36,46 +22,11 @@ github.com/stretchr/testify v1.11.1 h1:7s2iGBzp5EwR7/aIZr8ao5+dra3wiQyKjjFuvgVKu
github.com/stretchr/testify v1.11.1/go.mod h1:wZwfW3scLgRK+23gO65QZefKpKQRnfz6sD981Nm4B6U=
golang.org/x/crypto v0.55.0 h1:+KWHjbgOaAQ66dh/YlkZKHlz9ZUlq61AFirAR9ntP8M=
golang.org/x/crypto v0.55.0/go.mod h1:uq0V9dE/fzQuJtbnL+2EhWOE63vo164FY8xqEnV9xis=
golang.org/x/mod v0.38.0 h1:MECBjubtXD7yj4HrhIUcywNaGeNVUdfVnxmPajOk4yk=
golang.org/x/mod v0.38.0/go.mod h1:V6Xz0pq8TQ3dGqVQ1FVHuelZpAL0uNhSkk9ogYP3c40=
golang.org/x/sync v0.22.0 h1:SZjpbeLmrCk4xhRSZFNZW5gFUeCeFgjekvI/+gfScek=
golang.org/x/sync v0.22.0/go.mod h1:9xrNwdLfx4jkKbNva9FpL6vEN7evnE43NNNJQ2LF3+0=
golang.org/x/sys v0.6.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.47.0 h1:o7XGOvZQCADBQQ4Y7VNq2dRWQR7JmOUW8Kxx4ZsNgWs=
golang.org/x/sys v0.47.0/go.mod h1:4GL1E5IUh+htKOUEOaiffhrAeqysfVGipDYzABqnCmw=
golang.org/x/text v0.41.0 h1:vz/seA0lnX87Othu2f/0L24RcgrXD9/YFTSuGjj3rH8=
golang.org/x/text v0.41.0/go.mod h1:jvf1O8ajNzZqhSrQBPbutR/EB83Cc0CFrezNQIwbb5M=
golang.org/x/tools v0.48.0 h1:3+hClM1aLL5mjMKm5ovokw9epgRXPuu2tILgismM6RE=
golang.org/x/tools v0.48.0/go.mod h1:08xX0orndb/F7jJxGDicx061tyd5pcMto75YMAXr6lk=
gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
gopkg.in/yaml.v3 v3.0.0-20200313102051-9f266ea9e77c/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=
gopkg.in/yaml.v3 v3.0.1 h1:fxVm/GzAzEWqLHuvctI91KS9hhNmmWOoWu0XTYJS7CA=
gopkg.in/yaml.v3 v3.0.1/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=
modernc.org/cc/v4 v4.28.2 h1:3tQ0lf2ADtoby2EtSP+J7IE2SHwEJdP8ioR59wx7XpY=
modernc.org/cc/v4 v4.28.2/go.mod h1:OnovgIhbbMXMu1aISnJ0wvVD1KnW+cAUJkIrAWh+kVI=
modernc.org/ccgo/v4 v4.34.0 h1:yRLPFZieg532OT4rp4JFNIVcquwalMX26G95WQDqwCQ=
modernc.org/ccgo/v4 v4.34.0/go.mod h1:AS5WYMyBakQ+fhsHhtP8mWB82KTGPkNNJDGfGQCe0/A=
modernc.org/fileutil v1.4.0 h1:j6ZzNTftVS054gi281TyLjHPp6CPHr2KCxEXjEbD6SM=
modernc.org/fileutil v1.4.0/go.mod h1:EqdKFDxiByqxLk8ozOxObDSfcVOv/54xDs/DUHdvCUU=
modernc.org/gc/v2 v2.6.5 h1:nyqdV8q46KvTpZlsw66kWqwXRHdjIlJOhG6kxiV/9xI=
modernc.org/gc/v2 v2.6.5/go.mod h1:YgIahr1ypgfe7chRuJi2gD7DBQiKSLMPgBQe9oIiito=
modernc.org/gc/v3 v3.1.2 h1:ZtDCnhonXSZexk/AYsegNRV1lJGgaNZJuKjJSWKyEqo=
modernc.org/gc/v3 v3.1.2/go.mod h1:HFK/6AGESC7Ex+EZJhJ2Gni6cTaYpSMmU/cT9RmlfYY=
modernc.org/goabi0 v0.2.0 h1:HvEowk7LxcPd0eq6mVOAEMai46V+i7Jrj13t4AzuNks=
modernc.org/goabi0 v0.2.0/go.mod h1:CEFRnnJhKvWT1c1JTI3Avm+tgOWbkOu5oPA8eH8LnMI=
modernc.org/libc v1.72.3 h1:ZnDF4tXn4NBXFutMMQC4vtbTFSXhhKzR73fv0beZEAU=
modernc.org/libc v1.72.3/go.mod h1:dn0dZNnnn1clLyvRxLxYExxiKRZIRENOfqQ8XEeg4Qs=
modernc.org/mathutil v1.7.1 h1:GCZVGXdaN8gTqB1Mf/usp1Y/hSqgI2vAGGP4jZMCxOU=
modernc.org/mathutil v1.7.1/go.mod h1:4p5IwJITfppl0G4sUEDtCr4DthTaT47/N3aT6MhfgJg=
modernc.org/memory v1.11.0 h1:o4QC8aMQzmcwCK3t3Ux/ZHmwFPzE6hf2Y5LbkRs+hbI=
modernc.org/memory v1.11.0/go.mod h1:/JP4VbVC+K5sU2wZi9bHoq2MAkCnrt2r98UGeSK7Mjw=
modernc.org/opt v0.2.0 h1:tGyef5ApycA7FSEOMraay9SaTk5zmbx7Tu+cJs4QKZg=
modernc.org/opt v0.2.0/go.mod h1:03fq9lsNfvkYSfxrfUhZCWPk1lm4cq4N+Bh//bEtgns=
modernc.org/sortutil v1.2.1 h1:+xyoGf15mM3NMlPDnFqrteY07klSFxLElE2PVuWIJ7w=
modernc.org/sortutil v1.2.1/go.mod h1:7ZI3a3REbai7gzCLcotuw9AC4VZVpYMjDzETGsSMqJE=
modernc.org/sqlite v1.50.1 h1:l+cQvn0sd0zJJtfygGHuQJ5AjlrwXmWPw4KP3ZMwr9w=
modernc.org/sqlite v1.50.1/go.mod h1:tcNzv5p84E0skkmJn038y+hWJbLQXQqEnQfeh5r2JLM=
modernc.org/strutil v1.2.1 h1:UneZBkQA+DX2Rp35KcM69cSsNES9ly8mQWD71HKlOA0=
modernc.org/strutil v1.2.1/go.mod h1:EHkiggD70koQxjVdSBM3JKM7k6L0FbGE5eymy9i3B9A=
modernc.org/token v1.1.0 h1:Xl7Ap9dKaEs5kLoOQeQmPWevfnk/DM5qcLcYlA8ys6Y=
modernc.org/token v1.1.0/go.mod h1:UGzOrNV1mAFSEB63lOFHIpNRUVMvYTc6yu1SMY/XTDM=
+264
View File
@@ -0,0 +1,264 @@
package api_test
import (
"bytes"
"encoding/json"
"io"
"net/http"
"strconv"
"testing"
)
// The bootstrap user is an administrator; everybody it creates afterwards is
// not. These tests are about the line between them.
// id64 spells an id into a path segment.
func id64(n int64) string { return strconv.FormatInt(n, 10) }
// member creates an ordinary user and an API key for it, and returns a caller
// that authenticates as them. Minting the key goes through the admin's own
// credentials, which is how a real install hands one out.
func member(t *testing.T, s *ts, username string) (id int64, call func(method, path string, body any) *http.Response) {
t.Helper()
resp := s.req(t, http.MethodPost, "/api/users",
map[string]string{"username": username, "email": username + "@test.com"})
if resp.StatusCode != http.StatusCreated {
t.Fatalf("create %s: %d", username, resp.StatusCode)
}
var user struct {
ID int64 `json:"id"`
IsAdmin bool `json:"is_admin"`
}
decode(t, resp, &user)
if user.IsAdmin {
t.Fatalf("a created user must not be an administrator")
}
// Into the default team as a plain member: being in a team is what lets
// somebody work its incidents, and is separate from administering accounts.
resp = s.req(t, http.MethodPost, "/api/teams/"+defaultTeam+"/members",
map[string]any{"user_id": user.ID, "role": "member"})
resp.Body.Close()
if resp.StatusCode != http.StatusNoContent {
t.Fatalf("add %s to the team: %d", username, resp.StatusCode)
}
resp = s.req(t, http.MethodPost, "/api/users/"+id64(user.ID)+"/api-keys",
map[string]string{"name": "test"})
if resp.StatusCode != http.StatusCreated {
t.Fatalf("mint key for %s: %d", username, resp.StatusCode)
}
var key struct {
Key string `json:"key"`
}
decode(t, resp, &key)
return user.ID, func(method, path string, body any) *http.Response {
t.Helper()
var r io.Reader
if body != nil {
data, _ := json.Marshal(body)
r = bytes.NewReader(data)
}
req, _ := http.NewRequest(method, s.URL+path, r)
req.Header.Set("Authorization", "Bearer "+key.Key)
if body != nil {
req.Header.Set("Content-Type", "application/json")
}
resp, err := http.DefaultClient.Do(req)
if err != nil {
t.Fatalf("%s %s: %v", method, path, err)
}
return resp
}
}
// The whole point of the release: a user who is not an administrator cannot
// manage other people's accounts. Every one of these was open to any
// authenticated caller before.
func TestAdmin_MemberIsRefusedAdministration(t *testing.T) {
s := newTS(t)
memberID, call := member(t, s, "member")
cases := []struct {
name string
method string
path string
body any
}{
{"create a user", http.MethodPost, "/api/users",
map[string]string{"username": "sneaky", "email": "sneaky@test.com"}},
{"delete the admin", http.MethodDelete, "/api/users/1", nil},
{"grant themselves admin", http.MethodPut, "/api/users/" + id64(memberID) + "/admin",
map[string]bool{"is_admin": true}},
{"set the admin's password", http.MethodPut, "/api/users/1/password",
map[string]string{"password": "hunter2-hunter2"}},
{"mint a key for the admin", http.MethodPost, "/api/users/1/api-keys",
map[string]string{"name": "borrowed"}},
{"retarget the admin's notifications", http.MethodPut, "/api/users/1/notify",
map[string]string{"ntfy_topic": "attacker-topic"}},
}
for _, c := range cases {
resp := call(c.method, c.path, c.body)
resp.Body.Close()
if resp.StatusCode != http.StatusForbidden {
t.Errorf("%s: expected 403, got %d", c.name, resp.StatusCode)
}
}
}
// Being refused other people's accounts must not cost a user their own.
func TestAdmin_MemberKeepsTheirOwnAccount(t *testing.T) {
s := newTS(t)
memberID, call := member(t, s, "member")
self := "/api/users/" + id64(memberID)
resp := call(http.MethodPut, self+"/notify", map[string]string{"ntfy_topic": "terdut-member"})
resp.Body.Close()
if resp.StatusCode != http.StatusOK {
t.Errorf("own notify target: %d", resp.StatusCode)
}
resp = call(http.MethodPut, self+"/password", map[string]string{"password": "correct-horse-battery"})
resp.Body.Close()
if resp.StatusCode != http.StatusOK && resp.StatusCode != http.StatusNoContent {
t.Errorf("own password: %d", resp.StatusCode)
}
// An API key carries exactly the rights of its owner, so minting your own
// is no more than signing in again.
resp = call(http.MethodPost, self+"/api-keys", map[string]string{"name": "laptop"})
resp.Body.Close()
if resp.StatusCode != http.StatusCreated {
t.Errorf("own API key: %d", resp.StatusCode)
}
// And the queue still has to be able to name people.
resp = call(http.MethodGet, "/api/users", nil)
resp.Body.Close()
if resp.StatusCode != http.StatusOK {
t.Errorf("list users: %d", resp.StatusCode)
}
}
// Incident work is everybody's job; none of it is administration.
func TestAdmin_MemberCanWorkIncidents(t *testing.T) {
s := newTS(t)
_, call := member(t, s, "responder")
postWebhook(t, s, []map[string]any{
amAlert("fp-admin", "DiskFull", "firing", "2026-09-20T10:00:00Z", zeroTime, nil),
})
for _, c := range []struct {
name string
method string
path string
}{
{"list", http.MethodGet, "/api/incidents"},
{"acknowledge", http.MethodPost, "/api/incidents/1/acknowledge"},
{"resolve", http.MethodPost, "/api/incidents/1/resolve"},
} {
resp := call(c.method, c.path, nil)
resp.Body.Close()
if resp.StatusCode != http.StatusOK {
t.Errorf("%s: expected 200, got %d", c.name, resp.StatusCode)
}
}
}
// An install must never be left with nobody who can administer it.
func TestAdmin_LastAdministratorIsProtected(t *testing.T) {
s := newTS(t)
resp := s.req(t, http.MethodPut, "/api/users/1/admin", map[string]bool{"is_admin": false})
resp.Body.Close()
if resp.StatusCode != http.StatusConflict {
t.Errorf("self-demotion: expected 409, got %d", resp.StatusCode)
}
resp = s.req(t, http.MethodDelete, "/api/users/1", nil)
resp.Body.Close()
if resp.StatusCode != http.StatusConflict {
t.Errorf("deleting yourself: expected 409, got %d", resp.StatusCode)
}
// With a second administrator the first may stand down, but not while they
// are the only one — which is the same rule from the other side.
otherID, _ := member(t, s, "second")
resp = s.req(t, http.MethodPut, "/api/users/"+id64(otherID)+"/admin", map[string]bool{"is_admin": true})
resp.Body.Close()
if resp.StatusCode != http.StatusOK {
t.Fatalf("granting admin: %d", resp.StatusCode)
}
resp = s.req(t, http.MethodDelete, "/api/users/"+id64(otherID), nil)
resp.Body.Close()
if resp.StatusCode != http.StatusNoContent {
t.Errorf("deleting the second admin: expected 204, got %d", resp.StatusCode)
}
}
// A promoted user gets the powers with the flag, and loses them with it.
func TestAdmin_GrantAndRevokeChangeWhatIsAllowed(t *testing.T) {
s := newTS(t)
memberID, call := member(t, s, "promotee")
admin := "/api/users/" + id64(memberID) + "/admin"
resp := call(http.MethodPost, "/api/users", map[string]string{"username": "a", "email": "a@test.com"})
resp.Body.Close()
if resp.StatusCode != http.StatusForbidden {
t.Fatalf("before the grant: %d", resp.StatusCode)
}
resp = s.req(t, http.MethodPut, admin, map[string]bool{"is_admin": true})
resp.Body.Close()
if resp.StatusCode != http.StatusOK {
t.Fatalf("grant: %d", resp.StatusCode)
}
resp = call(http.MethodPost, "/api/users", map[string]string{"username": "b", "email": "b@test.com"})
resp.Body.Close()
if resp.StatusCode != http.StatusCreated {
t.Errorf("after the grant: expected 201, got %d", resp.StatusCode)
}
resp = s.req(t, http.MethodPut, admin, map[string]bool{"is_admin": false})
resp.Body.Close()
if resp.StatusCode != http.StatusOK {
t.Fatalf("revoke: %d", resp.StatusCode)
}
resp = call(http.MethodPost, "/api/users", map[string]string{"username": "c", "email": "c@test.com"})
resp.Body.Close()
if resp.StatusCode != http.StatusForbidden {
t.Errorf("after the revoke: expected 403, got %d", resp.StatusCode)
}
}
// The flag has to reach the client, or the web UI cannot decide what to show.
func TestAdmin_MeReportsTheFlag(t *testing.T) {
s := newTS(t)
var me struct {
User struct {
IsAdmin bool `json:"is_admin"`
} `json:"user"`
}
decode(t, s.req(t, http.MethodGet, "/api/me", nil), &me)
if !me.User.IsAdmin {
t.Error("the bootstrap user should be an administrator")
}
_, call := member(t, s, "plain")
var theirs struct {
User struct {
IsAdmin bool `json:"is_admin"`
} `json:"user"`
}
decode(t, call(http.MethodGet, "/api/me", nil), &theirs)
if theirs.User.IsAdmin {
t.Error("a created user should not be an administrator")
}
}
+73 -34
View File
@@ -4,9 +4,12 @@ import (
"context"
"database/sql"
"encoding/json"
"errors"
"log"
"net/http"
"time"
"github.com/go-chi/chi/v5"
)
// Values for alerts.resolution_source, recording why an alert left the firing
@@ -70,36 +73,63 @@ type ingested struct {
deadman bool
}
func handleAlertmanagerWebhook(db *sql.DB, notify NotifyConfig, deadman DeadmanConfig) http.HandlerFunc {
// handleIntegrationWebhook receives alerts on a team's own integration key.
// The key in the path is both the credential and the routing: it says who may
// post, and which team the alerts belong to.
func handleIntegrationWebhook(db *sql.DB, notify NotifyConfig) http.HandlerFunc {
return func(w http.ResponseWriter, r *http.Request) {
var payload amPayload
if err := decodeJSON(r, &payload); err != nil {
respond(w, http.StatusBadRequest, errResp("invalid payload"))
teamID, err := teamIDForKey(r.Context(), db, chi.URLParam(r, "key"))
if err != nil {
if errors.Is(err, errUnknownIntegration) {
// 401 and not 404: the path is real, the key is not, and a
// sender misconfigured this way should say so in its own logs
// rather than believe it is delivering.
respond(w, http.StatusUnauthorized, errResp("unknown integration key"))
return
}
respond(w, http.StatusInternalServerError, errResp("internal error"))
return
}
// Alertmanager retries anything that is not 2xx, and a retry of a payload
// we failed to store is more useful than an error it cannot act on — so
// failures are logged, not surfaced.
if err := ingest(r.Context(), db, notify, deadman, payload); err != nil {
log.Printf("webhook ingest (group %q): %v", payload.GroupKey, err)
}
w.WriteHeader(http.StatusOK)
receiveWebhook(w, r, db, notify, teamID)
}
}
func receiveWebhook(w http.ResponseWriter, r *http.Request, db *sql.DB, notify NotifyConfig, teamID int64) {
var payload amPayload
if err := decodeJSON(r, &payload); err != nil {
respond(w, http.StatusBadRequest, errResp("invalid payload"))
return
}
// Alertmanager retries anything that is not 2xx, and a retry of a payload
// we failed to store is more useful than an error it cannot act on — so
// failures are logged, not surfaced.
if err := ingest(r.Context(), db, notify, teamID, payload); err != nil {
log.Printf("webhook ingest (team %d, group %q): %v", teamID, payload.GroupKey, err)
}
w.WriteHeader(http.StatusOK)
}
// ingest stores a payload's alerts and reconciles the incident for its group.
// The whole payload is one transaction: an incident that opened but whose alerts
// failed to link would be a work item nobody could act on.
func ingest(ctx context.Context, db *sql.DB, notify NotifyConfig, deadman DeadmanConfig, payload amPayload) error {
func ingest(ctx context.Context, db *sql.DB, notify NotifyConfig, teamID int64, payload amPayload) error {
tx, err := db.BeginTx(ctx, nil)
if err != nil {
return err
}
defer tx.Rollback() //nolint:errcheck
accepted, err := upsertAlerts(ctx, tx, deadman, payload.Alerts)
// Which arriving alerts are heartbeats is the team's own answer, read
// inside the transaction so an owner editing it mid-payload cannot split
// one webhook across two interpretations.
deadman, err := deadmanConfigForTeam(ctx, tx, teamID)
if err != nil {
return err
}
accepted, err := upsertAlerts(ctx, tx, deadman, teamID, payload.Alerts)
if err != nil {
return err
}
@@ -108,7 +138,7 @@ func ingest(ctx context.Context, db *sql.DB, notify NotifyConfig, deadman Deadma
// resolution cascade are recomputed once per incident at the end.
touched := map[int64]bool{}
incidentID, err := incidentForGroup(ctx, tx, notify, payload, accepted)
incidentID, err := incidentForGroup(ctx, tx, notify, teamID, payload, accepted)
if err != nil {
return err
}
@@ -156,7 +186,7 @@ func ingest(ctx context.Context, db *sql.DB, notify NotifyConfig, deadman Deadma
// upsertAlerts stores each alert of a payload and reports what changed. Payloads
// the ordering guard rejected are left out entirely.
func upsertAlerts(ctx context.Context, tx *sql.Tx, deadman DeadmanConfig, alerts []amAlert) ([]ingested, error) {
func upsertAlerts(ctx context.Context, tx *sql.Tx, deadman DeadmanConfig, teamID int64, alerts []amAlert) ([]ingested, error) {
now := time.Now().Unix()
accepted := make([]ingested, 0, len(alerts))
@@ -171,7 +201,8 @@ func upsertAlerts(ctx context.Context, tx *sql.Tx, deadman DeadmanConfig, alerts
var prevStartsAt int64
existed := true
switch err := tx.QueryRowContext(ctx,
"SELECT status, starts_at FROM alerts WHERE fingerprint = $1", a.Fingerprint,
"SELECT status, starts_at FROM alerts WHERE team_id = $1 AND fingerprint = $2",
teamID, a.Fingerprint,
).Scan(&prevStatus, &prevStartsAt); {
case err == sql.ErrNoRows:
existed = false
@@ -210,10 +241,10 @@ func upsertAlerts(ctx context.Context, tx *sql.Tx, deadman DeadmanConfig, alerts
// undone by a stale retry.
if _, err := tx.ExecContext(ctx, `
INSERT INTO alerts
(fingerprint, name, status, labels, annotations, starts_at, ends_at,
(team_id, fingerprint, name, status, labels, annotations, starts_at, ends_at,
generator_url, received_at, resolution_source)
VALUES ($1, $2, $3, $4::jsonb, $5::jsonb, $6, $7, $8, $9, $10)
ON CONFLICT (fingerprint) DO UPDATE SET
VALUES ($1, $2, $3, $4, $5::jsonb, $6::jsonb, $7, $8, $9, $10, $11)
ON CONFLICT (team_id, fingerprint) DO UPDATE SET
status = excluded.status,
labels = excluded.labels,
annotations = excluded.annotations,
@@ -233,7 +264,7 @@ func upsertAlerts(ctx context.Context, tx *sql.Tx, deadman DeadmanConfig, alerts
OR (excluded.starts_at = alerts.starts_at
AND (alerts.resolution_source = '`+resolutionDeadman+`'
OR NOT (alerts.status = 'resolved' AND excluded.status = 'firing')))`,
a.Fingerprint, name, a.Status,
teamID, a.Fingerprint, name, a.Status,
string(labelsJSON), string(annotationsJSON),
a.StartsAt.Unix(), endsAtUnix,
a.GeneratorURL, now, resolutionSource,
@@ -245,7 +276,8 @@ func upsertAlerts(ctx context.Context, tx *sql.Tx, deadman DeadmanConfig, alerts
var curStatus string
var curStartsAt int64
if err := tx.QueryRowContext(ctx,
"SELECT id, status, starts_at FROM alerts WHERE fingerprint = $1", a.Fingerprint,
"SELECT id, status, starts_at FROM alerts WHERE team_id = $1 AND fingerprint = $2",
teamID, a.Fingerprint,
).Scan(&id, &curStatus, &curStartsAt); err != nil {
return nil, err
}
@@ -284,7 +316,7 @@ func upsertAlerts(ctx context.Context, tx *sql.Tx, deadman DeadmanConfig, alerts
// Heartbeats do not count as anything here. A group of nothing but dead man's
// switch alerts opens no incident at all, and a mixed group gets an incident for
// its real alerts only.
func incidentForGroup(ctx context.Context, tx *sql.Tx, notify NotifyConfig, payload amPayload, accepted []ingested) (int64, error) {
func incidentForGroup(ctx context.Context, tx *sql.Tx, notify NotifyConfig, teamID int64, payload amPayload, accepted []ingested) (int64, error) {
var firstName string
anyFiring, anyNew := false, false
for _, a := range accepted {
@@ -315,7 +347,8 @@ func incidentForGroup(ctx context.Context, tx *sql.Tx, notify NotifyConfig, payl
var id int64
switch err := tx.QueryRowContext(ctx,
"SELECT id FROM incidents WHERE group_key = $1 AND resolved_at IS NULL", groupKey,
"SELECT id FROM incidents WHERE team_id = $1 AND group_key = $2 AND resolved_at IS NULL",
teamID, groupKey,
).Scan(&id); {
case err == nil:
return id, nil
@@ -326,7 +359,7 @@ func incidentForGroup(ctx context.Context, tx *sql.Tx, notify NotifyConfig, payl
if !anyNew {
return 0, nil
}
return openIncident(ctx, tx, notify, groupKey,
return openIncident(ctx, tx, notify, teamID, groupKey,
incidentTitle(payload.GroupLabels, firstName), payload.GroupLabels, nil)
}
@@ -338,8 +371,8 @@ func incidentForGroup(ctx context.Context, tx *sql.Tx, notify NotifyConfig, payl
// its own. Hence the querier rather than a *sql.Tx. A nil severity leaves the
// column for refreshSeverity to fill from the member alerts; the sweeper passes
// one because its incidents have no members to derive it from.
func openIncident(ctx context.Context, q querier, notify NotifyConfig, groupKey, title string, groupLabels map[string]string, severity *string) (int64, error) {
onCall, err := currentOnCall(ctx, q)
func openIncident(ctx context.Context, q querier, notify NotifyConfig, teamID int64, groupKey, title string, groupLabels map[string]string, severity *string) (int64, error) {
onCall, err := currentOnCall(ctx, q, teamID)
if err != nil {
return 0, err
}
@@ -351,10 +384,10 @@ func openIncident(ctx context.Context, q querier, notify NotifyConfig, groupKey,
var id int64
err = q.QueryRowContext(ctx, `
INSERT INTO incidents (group_key, title, group_labels, status, severity, triggered_at, assigned_to)
VALUES ($1, $2, $3::jsonb, 'triggered', $4, $5, $6)
INSERT INTO incidents (team_id, group_key, title, group_labels, status, severity, triggered_at, assigned_to)
VALUES ($1, $2, $3, $4::jsonb, 'triggered', $5, $6, $7)
RETURNING id`,
groupKey, title, string(labelsJSON), severity,
teamID, groupKey, title, string(labelsJSON), severity,
time.Now().Unix(), onCall).Scan(&id)
if err != nil {
return 0, err
@@ -370,12 +403,18 @@ func openIncident(ctx context.Context, q querier, notify NotifyConfig, groupKey,
}
}
// Queue the page, but do not send it here: this runs inside a transaction on
// a single-connection pool, so an HTTP call would hold up every other
// request. The notifier picks the row up within a tick.
// Queue the page, but do not send it here: this runs inside the webhook's
// transaction, and an HTTP call would hold a connection open across a
// network round trip. The notifier picks the row up within a tick.
if err := enqueueOpened(ctx, q, notify, id, onCall); err != nil {
return 0, err
}
// And start the escalation clock, if the team keeps one. In the same
// transaction, so an incident is never briefly open with nobody counting.
if err := startEscalation(ctx, q, id, teamID); err != nil {
return 0, err
}
return id, nil
}
+15 -6
View File
@@ -19,7 +19,7 @@ import (
// incident_alerts rather than as a column here, because one alert row is reused
// across occurrences and belongs to a different incident each time.
const alertSelectFrom = `
SELECT a.id, a.fingerprint, a.name, a.status,
SELECT a.id, a.team_id, t.name, a.fingerprint, a.name, a.status,
a.labels, a.annotations,
a.starts_at, a.ends_at, a.generator_url, a.received_at,
(SELECT ia.incident_id
@@ -29,7 +29,8 @@ const alertSelectFrom = `
ORDER BY i.triggered_at DESC, i.id DESC
LIMIT 1),
a.resolution_source, a.archived_at
FROM alerts a`
FROM alerts a
JOIN teams t ON t.id = a.team_id`
func handleListAlerts(db *sql.DB) http.HandlerFunc {
return func(w http.ResponseWriter, r *http.Request) {
@@ -38,6 +39,13 @@ func handleListAlerts(db *sql.DB) http.HandlerFunc {
where := []string{}
args := &sqlArgs{}
where = append(where, "a.team_id = ANY("+args.add(callerTeamIDs(r.Context()))+")")
if team := q.Get("team_id"); team != "" {
if n, err := strconv.ParseInt(team, 10, 64); err == nil {
where = append(where, "a.team_id = "+args.add(n))
}
}
if status := q.Get("status"); status != "" {
where = append(where, "a.status = "+args.add(status))
}
@@ -107,7 +115,7 @@ func handleGetAlert(db *sql.DB) http.HandlerFunc {
respond(w, http.StatusBadRequest, errResp("invalid alert id"))
return
}
a, err := fetchAlert(r.Context(), db, id)
a, err := fetchAlert(r.Context(), db, id, callerTeamIDs(r.Context()))
if err == sql.ErrNoRows {
respond(w, http.StatusNotFound, errResp("alert not found"))
return
@@ -121,8 +129,9 @@ func handleGetAlert(db *sql.DB) http.HandlerFunc {
}
// fetchAlert loads a single alert by ID using the shared query.
func fetchAlert(ctx context.Context, db *sql.DB, id int64) (models.Alert, error) {
return scanAlert(db.QueryRowContext(ctx, alertSelectFrom+" WHERE a.id = $1", id))
func fetchAlert(ctx context.Context, db *sql.DB, id int64, teamIDs []int64) (models.Alert, error) {
return scanAlert(db.QueryRowContext(ctx,
alertSelectFrom+" WHERE a.id = $1 AND a.team_id = ANY($2)", id, teamIDs))
}
// scanner is satisfied by both *sql.Row and *sql.Rows.
@@ -137,7 +146,7 @@ func scanAlert(s scanner) (models.Alert, error) {
var endsAtUnix, archivedAtUnix *int64
if err := s.Scan(
&a.ID, &a.Fingerprint, &a.Name, &a.Status,
&a.ID, &a.TeamID, &a.TeamName, &a.Fingerprint, &a.Name, &a.Status,
&labelsJSON, &annotationsJSON,
&startsAtUnix, &endsAtUnix,
&a.GeneratorURL, &receivedAtUnix,
+88 -26
View File
@@ -9,6 +9,8 @@ import (
"io"
"net/http"
"net/http/httptest"
"sort"
"strings"
"testing"
"time"
@@ -19,10 +21,14 @@ import (
// tests can age rows directly — the sweeper's inputs are wall-clock timestamps.
type ts struct {
*httptest.Server
key string
db *sql.DB
notify api.NotifyConfig
deadman api.DeadmanConfig
key string
// ingestKey is an integration key for the default team: the only way in
// since the unauthenticated webhook was removed, so the tests exercise the
// same path production does.
ingestKey string
db *sql.DB
notify api.NotifyConfig
deadman api.DeadmanConfig
}
// newTS builds a server over a fresh database. Notifications are off
@@ -37,7 +43,7 @@ func newTS(t *testing.T, notify ...api.NotifyConfig) *ts {
return newDeadmanTS(t, api.DeadmanConfig{}, cfg)
}
// newDeadmanTS is newTS with dead man's switch handling configured.
// newDeadmanTS is newTS with the default team's dead man's switches configured.
func newDeadmanTS(t *testing.T, deadman api.DeadmanConfig, notify ...api.NotifyConfig) *ts {
t.Helper()
var cfg api.NotifyConfig
@@ -46,7 +52,7 @@ func newDeadmanTS(t *testing.T, deadman api.DeadmanConfig, notify ...api.NotifyC
}
database := newTestDB(t)
srv := httptest.NewServer(api.NewRouter(database, cfg, deadman))
srv := httptest.NewServer(api.NewRouter(database, cfg, testConfig()))
t.Cleanup(srv.Close)
body, _ := json.Marshal(map[string]string{"username": "admin", "email": "admin@test.com"})
@@ -62,7 +68,48 @@ func newDeadmanTS(t *testing.T, deadman api.DeadmanConfig, notify ...api.NotifyC
json.NewDecoder(resp.Body).Decode(&result)
key := result["api_key"].(map[string]any)["key"].(string)
return &ts{Server: srv, key: key, db: database, notify: cfg, deadman: deadman}
s := &ts{Server: srv, key: key, db: database, notify: cfg, deadman: deadman}
var integration struct {
Key string `json:"key"`
}
decode(t, s.req(t, http.MethodPost, "/api/teams/"+defaultTeam+"/integrations",
map[string]string{"name": "test"}), &integration)
if integration.Key == "" {
t.Fatal("no integration key was returned")
}
s.ingestKey = integration.Key
// Dead man's switches belong to a team now, so a test that wants them
// configures the default team the way an owner would.
if deadman.Timeout > 0 {
setTeamDeadman(t, s, deadman)
}
return s
}
// setTeamDeadman configures the default team's switches over the API, rendering
// the matchers back into the string form the endpoint takes.
func setTeamDeadman(t *testing.T, s *ts, cfg api.DeadmanConfig) {
t.Helper()
matchers := make([]string, 0, len(cfg.Matchers))
for _, m := range cfg.Matchers {
parts := []string{"alertname=" + m.Name}
for k, v := range m.Labels {
parts = append(parts, k+"="+v)
}
sort.Strings(parts[1:])
matchers = append(matchers, strings.Join(parts, ","))
}
resp := s.req(t, http.MethodPut, "/api/teams/"+defaultTeam+"/deadman", map[string]any{
"matchers": strings.Join(matchers, "; "),
"timeout_seconds": int64(cfg.Timeout.Seconds()),
"severity": cfg.Severity,
})
defer resp.Body.Close()
if resp.StatusCode != http.StatusOK {
t.Fatalf("configure the team's dead man's switches: %d", resp.StatusCode)
}
}
// exec runs a statement against the test database.
@@ -201,7 +248,8 @@ func postWebhook(t *testing.T, s *ts, alerts []map[string]any, groupKey ...strin
}
}
data, _ := json.Marshal(payload)
resp, err := http.Post(s.URL+"/api/alertmanager/webhook", "application/json", bytes.NewReader(data))
resp, err := http.Post(s.URL+"/api/integrations/"+s.ingestKey+"/alertmanager",
"application/json", bytes.NewReader(data))
if err != nil {
t.Fatalf("post webhook: %v", err)
}
@@ -276,14 +324,14 @@ func TestAlertUpsert_DifferentFingerprintsStored(t *testing.T) {
func TestSchedule_ConflictOnSameDate(t *testing.T) {
s := newTS(t)
first := s.req(t, http.MethodPost, "/api/schedule",
first := s.req(t, http.MethodPost, "/api/teams/"+defaultTeam+"/schedule",
map[string]any{"user_id": 1, "dates": []string{"2026-06-01"}})
if first.StatusCode != http.StatusCreated {
t.Fatalf("first assignment returned %d", first.StatusCode)
}
first.Body.Close()
second := s.req(t, http.MethodPost, "/api/schedule",
second := s.req(t, http.MethodPost, "/api/teams/"+defaultTeam+"/schedule",
map[string]any{"user_id": 1, "dates": []string{"2026-06-01"}})
if second.StatusCode != http.StatusConflict {
t.Errorf("expected 409 on duplicate date, got %d", second.StatusCode)
@@ -295,11 +343,11 @@ func TestSchedule_MultiDateRollbackOnConflict(t *testing.T) {
s := newTS(t)
// Claim 2026-06-10 first.
s.req(t, http.MethodPost, "/api/schedule",
s.req(t, http.MethodPost, "/api/teams/"+defaultTeam+"/schedule",
map[string]any{"user_id": 1, "dates": []string{"2026-06-10"}}).Body.Close()
// Try to assign two dates in one request where the second conflicts.
resp := s.req(t, http.MethodPost, "/api/schedule",
resp := s.req(t, http.MethodPost, "/api/teams/"+defaultTeam+"/schedule",
map[string]any{"user_id": 1, "dates": []string{"2026-06-09", "2026-06-10"}})
if resp.StatusCode != http.StatusConflict {
t.Fatalf("expected 409, got %d", resp.StatusCode)
@@ -307,7 +355,7 @@ func TestSchedule_MultiDateRollbackOnConflict(t *testing.T) {
resp.Body.Close()
// 2026-06-09 must NOT have been committed (transaction rolled back).
listResp := s.req(t, http.MethodGet, "/api/schedule?from=2026-06-09&to=2026-06-09", nil)
listResp := s.req(t, http.MethodGet, "/api/teams/"+defaultTeam+"/schedule?from=2026-06-09&to=2026-06-09", nil)
var entries []any
decode(t, listResp, &entries)
if len(entries) != 0 {
@@ -321,21 +369,35 @@ func TestSchedule_MultiDateRollbackOnConflict(t *testing.T) {
// addUser creates a second person to hand a shift to. The bootstrap user is
// admin, id 1.
// addUser creates a user and puts them in the default team, because a user who
// is in no team can be paged by nobody and take no shift — which is the rule
// these tests exercise around, not the one they are testing.
func addUser(t *testing.T, s *ts, username string) {
t.Helper()
resp := s.req(t, http.MethodPost, "/api/users",
map[string]any{"username": username, "email": username + "@test.com"})
defer resp.Body.Close()
if resp.StatusCode != http.StatusCreated {
resp.Body.Close()
t.Fatalf("create user returned %d", resp.StatusCode)
}
var user struct {
ID int64 `json:"id"`
}
decode(t, resp, &user)
member := s.req(t, http.MethodPost, "/api/teams/"+defaultTeam+"/members",
map[string]any{"user_id": user.ID, "role": "member"})
defer member.Body.Close()
if member.StatusCode != http.StatusNoContent {
t.Fatalf("add %s to the team returned %d", username, member.StatusCode)
}
}
// scheduleHolder reports who is on call for one date, or "" for nobody.
func scheduleHolder(t *testing.T, s *ts, date string) string {
t.Helper()
var entries []map[string]any
decode(t, s.req(t, http.MethodGet, "/api/schedule?from="+date+"&to="+date, nil), &entries)
decode(t, s.req(t, http.MethodGet, "/api/teams/"+defaultTeam+"/schedule?from="+date+"&to="+date, nil), &entries)
if len(entries) == 0 {
return ""
}
@@ -347,10 +409,10 @@ func TestSchedule_ReplaceTakesAnAssignedDate(t *testing.T) {
s := newTS(t)
addUser(t, s, "alex")
s.req(t, http.MethodPost, "/api/schedule",
s.req(t, http.MethodPost, "/api/teams/"+defaultTeam+"/schedule",
map[string]any{"user_id": 1, "dates": []string{"2026-06-01"}}).Body.Close()
resp := s.req(t, http.MethodPost, "/api/schedule",
resp := s.req(t, http.MethodPost, "/api/teams/"+defaultTeam+"/schedule",
map[string]any{"user_id": 2, "dates": []string{"2026-06-01"}, "replace": true})
if resp.StatusCode != http.StatusCreated {
t.Fatalf("expected replace to succeed, got %d", resp.StatusCode)
@@ -364,7 +426,7 @@ func TestSchedule_ReplaceTakesAnAssignedDate(t *testing.T) {
// One row, not two: two entries for a date would mean two people believing
// they are on call for it.
var entries []map[string]any
decode(t, s.req(t, http.MethodGet, "/api/schedule?from=2026-06-01&to=2026-06-01", nil), &entries)
decode(t, s.req(t, http.MethodGet, "/api/teams/"+defaultTeam+"/schedule?from=2026-06-01&to=2026-06-01", nil), &entries)
if len(entries) != 1 {
t.Errorf("expected exactly one entry for the date, got %d", len(entries))
}
@@ -376,11 +438,11 @@ func TestSchedule_ReplaceMixedWeek(t *testing.T) {
s := newTS(t)
addUser(t, s, "alex")
s.req(t, http.MethodPost, "/api/schedule",
s.req(t, http.MethodPost, "/api/teams/"+defaultTeam+"/schedule",
map[string]any{"user_id": 1, "dates": []string{"2026-06-02", "2026-06-04"}}).Body.Close()
week := []string{"2026-06-01", "2026-06-02", "2026-06-03", "2026-06-04", "2026-06-05"}
resp := s.req(t, http.MethodPost, "/api/schedule",
resp := s.req(t, http.MethodPost, "/api/teams/"+defaultTeam+"/schedule",
map[string]any{"user_id": 2, "dates": week, "replace": true})
if resp.StatusCode != http.StatusCreated {
t.Fatalf("expected the mixed week to succeed, got %d", resp.StatusCode)
@@ -399,10 +461,10 @@ func TestSchedule_ReplaceDefaultsOff(t *testing.T) {
s := newTS(t)
addUser(t, s, "alex")
s.req(t, http.MethodPost, "/api/schedule",
s.req(t, http.MethodPost, "/api/teams/"+defaultTeam+"/schedule",
map[string]any{"user_id": 1, "dates": []string{"2026-06-01"}}).Body.Close()
resp := s.req(t, http.MethodPost, "/api/schedule",
resp := s.req(t, http.MethodPost, "/api/teams/"+defaultTeam+"/schedule",
map[string]any{"user_id": 2, "dates": []string{"2026-06-01"}})
if resp.StatusCode != http.StatusConflict {
t.Fatalf("expected 409 without replace, got %d", resp.StatusCode)
@@ -420,7 +482,7 @@ func TestSchedule_ReplaceDefaultsOff(t *testing.T) {
func TestSchedule_ReplaceCollapsesRepeatedDates(t *testing.T) {
s := newTS(t)
resp := s.req(t, http.MethodPost, "/api/schedule",
resp := s.req(t, http.MethodPost, "/api/teams/"+defaultTeam+"/schedule",
map[string]any{"user_id": 1, "dates": []string{"2026-06-01", "2026-06-01"}, "replace": true})
if resp.StatusCode != http.StatusCreated {
t.Fatalf("expected a repeated date to be accepted under replace, got %d", resp.StatusCode)
@@ -428,7 +490,7 @@ func TestSchedule_ReplaceCollapsesRepeatedDates(t *testing.T) {
resp.Body.Close()
var entries []map[string]any
decode(t, s.req(t, http.MethodGet, "/api/schedule?from=2026-06-01&to=2026-06-01", nil), &entries)
decode(t, s.req(t, http.MethodGet, "/api/teams/"+defaultTeam+"/schedule?from=2026-06-01&to=2026-06-01", nil), &entries)
if len(entries) != 1 {
t.Errorf("expected one entry for the repeated date, got %d", len(entries))
}
@@ -498,7 +560,7 @@ func TestArchive_AlertListFilter(t *testing.T) {
}
// 2. Let the sweeper archive it: ends_at is already well past archiveAfter.
api.Sweep(context.Background(), s.db, time.Hour, 6*time.Hour, s.deadman, s.notify)
api.Sweep(context.Background(), s.db, time.Hour, 6*time.Hour, s.notify)
// 3. Default list excludes it.
decode(t, s.req(t, http.MethodGet, "/api/alerts", nil), &alerts)
@@ -539,7 +601,7 @@ func postAlert(t *testing.T, s *ts, fingerprint, status, startsAt, endsAt string
func sweep(t *testing.T, s *ts, staleAfter time.Duration) {
t.Helper()
api.Sweep(context.Background(), s.db, noArchive, staleAfter, s.deadman, s.notify)
api.Sweep(context.Background(), s.db, noArchive, staleAfter, s.notify)
}
// A firing alert Alertmanager stopped refreshing is resolved via the
+13 -5
View File
@@ -19,15 +19,19 @@ const (
// StartArchiver runs the alert sweeper until ctx is cancelled, starting with an
// immediate pass so a restart reconciles state right away.
func StartArchiver(ctx context.Context, db *sql.DB, archiveAfter, staleAfter time.Duration, deadman DeadmanConfig, notify NotifyConfig) {
// archiveAfter and staleAfter are the values the server started with. They are
// the fallback, not the setting: each pass reads the current value from the
// settings table, so an administrator's change takes effect on the next tick
// instead of at the next restart.
func StartArchiver(ctx context.Context, db *sql.DB, archiveAfter, staleAfter time.Duration, notify NotifyConfig) {
ticker := time.NewTicker(sweepInterval)
defer ticker.Stop()
Sweep(ctx, db, archiveAfter, staleAfter, deadman, notify)
Sweep(ctx, db, archiveAfter, staleAfter, notify)
for {
select {
case <-ticker.C:
Sweep(ctx, db, archiveAfter, staleAfter, deadman, notify)
Sweep(ctx, db, archiveAfter, staleAfter, notify)
case <-ctx.Done():
return
}
@@ -44,8 +48,12 @@ func StartArchiver(ctx context.Context, db *sql.DB, archiveAfter, staleAfter tim
// touch: a heartbeat answers to its own, much tighter, timeout, and the generic
// staleness rules would otherwise resolve it as 'expiry' long before that.
// Exported so tests can drive a pass without waiting on the ticker.
func Sweep(ctx context.Context, db *sql.DB, archiveAfter, staleAfter time.Duration, deadman DeadmanConfig, notify NotifyConfig) {
heartbeats := sweepDeadman(ctx, db, deadman, notify)
func Sweep(ctx context.Context, db *sql.DB, archiveAfter, staleAfter time.Duration, notify NotifyConfig) {
settings := NewSettings(db)
staleAfter = settings.Duration(ctx, SettingStaleAfter, staleAfter)
archiveAfter = settings.Duration(ctx, SettingArchiveAfter, archiveAfter)
heartbeats := sweepDeadman(ctx, db, notify)
expireStale(ctx, db, staleAfter, heartbeats)
resolveSettledIncidents(ctx, db)
archiveResolved(ctx, db, archiveAfter)
+6 -2
View File
@@ -266,8 +266,9 @@ func handleMe(db *sql.DB) http.HandlerFunc {
//
// Changing your own password takes the current one, when there is one, so an
// unattended signed-in browser cannot be used to take the account over. Setting
// somebody else's is how an admin gives a user their first password, and like
// the other user endpoints it is open to any authenticated caller.
// somebody else's is how an admin gives a user their first password, and is
// restricted to administrators: it hands over an account outright, without
// knowing the password it replaces.
//
// Every other session of the target is ended: a password change is what you
// do when you think someone else is signed in.
@@ -278,6 +279,9 @@ func handleSetPassword(db *sql.DB) http.HandlerFunc {
respond(w, http.StatusBadRequest, errResp("invalid user id"))
return
}
if !requireSelfOrAdmin(w, r, id) {
return
}
var req struct {
Password string `json:"password"`
CurrentPassword string `json:"current_password"`
+2 -2
View File
@@ -248,7 +248,7 @@ func TestSession_ExpiredIsRejected(t *testing.T) {
if code := status(t, b.do(t, http.MethodGet, "/api/me", nil)); code != http.StatusUnauthorized {
t.Errorf("expired session: %d", code)
}
api.Sweep(t.Context(), s.db, 0, 0, api.DeadmanConfig{}, api.NotifyConfig{})
api.Sweep(t.Context(), s.db, 0, 0, api.NotifyConfig{})
var n int
s.db.QueryRow("SELECT COUNT(*) FROM sessions").Scan(&n)
if n != 0 {
@@ -301,7 +301,7 @@ func TestSetPassword_EndsOtherSessionsButNotThisOne(t *testing.T) {
func TestBootstrap_WithPassword(t *testing.T) {
database := newTestDB(t)
srv := httptest.NewServer(api.NewRouter(database, api.NotifyConfig{}, api.DeadmanConfig{}))
srv := httptest.NewServer(api.NewRouter(database, api.NotifyConfig{}, testConfig()))
t.Cleanup(srv.Close)
body := `{"username":"admin","email":"a@test.com","password":"` + adminPassword + `"}`
+150 -28
View File
@@ -171,6 +171,7 @@ func ParseDeadmanConfig(matchers string, timeout time.Duration, severity string)
// deadmanAlert is one switch: the alert row carrying its last heartbeat.
type deadmanAlert struct {
id int64
teamID int64
fingerprint string
labels map[string]string
matcher DeadmanMatcher
@@ -188,35 +189,42 @@ func (a deadmanAlert) groupKey() string { return deadmanGroupPrefix + a.fingerpr
// It returns the ids of the alerts it owns, because the generic staleness
// expiry must leave them alone — staleAfter and ends_at would otherwise resolve
// a heartbeat long before its own, much tighter, timeout ever fired.
func sweepDeadman(ctx context.Context, db *sql.DB, cfg DeadmanConfig, notify NotifyConfig) map[int64]bool {
// Each team is swept against its own configuration: its own matchers, its own
// timeout, its own severity. A team watching nothing is skipped entirely, which
// is most of them.
func sweepDeadman(ctx context.Context, db *sql.DB, notify NotifyConfig) map[int64]bool {
owned := map[int64]bool{}
if !cfg.enabled() {
return owned
}
switches, err := deadmanAlerts(ctx, db, cfg)
configs, err := deadmanConfigs(ctx, db)
if err != nil {
log.Printf("deadman: load switches: %v", err)
log.Printf("deadman: load configs: %v", err)
return owned
}
now := time.Now()
cutoff := now.Add(-cfg.Timeout).Unix()
for _, sw := range switches {
owned[sw.id] = true
// An explicit resolved from Alertmanager is a stronger death signal than
// mere absence: the sender is telling us the heartbeat stopped, so there
// is nothing left to wait out.
if sw.resolved || sw.receivedAt < cutoff {
if err := deadmanDied(ctx, db, cfg, notify, sw, now); err != nil {
log.Printf("deadman: open incident for %s: %v", sw.matcher.Name, err)
}
for teamID, cfg := range configs {
switches, err := deadmanAlerts(ctx, db, teamID, cfg)
if err != nil {
log.Printf("deadman: load switches for team %d: %v", teamID, err)
continue
}
if err := deadmanRecovered(ctx, db, sw); err != nil {
log.Printf("deadman: resolve incident for %s: %v", sw.matcher.Name, err)
cutoff := now.Add(-cfg.Timeout).Unix()
for _, sw := range switches {
owned[sw.id] = true
// An explicit resolved from Alertmanager is a stronger death signal
// than mere absence: the sender is telling us the heartbeat
// stopped, so there is nothing left to wait out.
if sw.resolved || sw.receivedAt < cutoff {
if err := deadmanDied(ctx, db, cfg, notify, sw, now); err != nil {
log.Printf("deadman: open incident for %s: %v", sw.matcher.Name, err)
}
continue
}
if err := deadmanRecovered(ctx, db, sw); err != nil {
log.Printf("deadman: resolve incident for %s: %v", sw.matcher.Name, err)
}
}
}
return owned
@@ -227,7 +235,7 @@ func sweepDeadman(ctx context.Context, db *sql.DB, cfg DeadmanConfig, notify Not
// happens in Go, which keeps one implementation of the rules. The rows are read
// in full before the caller writes, so the writes do not run against an open
// cursor over the same table.
func deadmanAlerts(ctx context.Context, db *sql.DB, cfg DeadmanConfig) ([]deadmanAlert, error) {
func deadmanAlerts(ctx context.Context, db *sql.DB, teamID int64, cfg DeadmanConfig) ([]deadmanAlert, error) {
names := cfg.names()
args := &sqlArgs{}
nameList := make([]any, len(names))
@@ -236,9 +244,10 @@ func deadmanAlerts(ctx context.Context, db *sql.DB, cfg DeadmanConfig) ([]deadma
}
rows, err := db.QueryContext(ctx, `
SELECT id, fingerprint, labels, status, received_at
SELECT id, team_id, fingerprint, labels, status, received_at
FROM alerts
WHERE name IN (`+args.addList(nameList)+`)
WHERE team_id = `+args.add(teamID)+`
AND name IN (`+args.addList(nameList)+`)
AND archived_at IS NULL`, args.all()...)
if err != nil {
return nil, err
@@ -249,7 +258,7 @@ func deadmanAlerts(ctx context.Context, db *sql.DB, cfg DeadmanConfig) ([]deadma
for rows.Next() {
var a deadmanAlert
var labelsJSON, status string
if err := rows.Scan(&a.id, &a.fingerprint, &labelsJSON, &status, &a.receivedAt); err != nil {
if err := rows.Scan(&a.id, &a.teamID, &a.fingerprint, &labelsJSON, &status, &a.receivedAt); err != nil {
return nil, err
}
json.Unmarshal([]byte(labelsJSON), &a.labels) //nolint:errcheck
@@ -280,8 +289,8 @@ func deadmanDied(ctx context.Context, db *sql.DB, cfg DeadmanConfig, notify Noti
if err := db.QueryRowContext(ctx, `
SELECT COALESCE(MAX(triggered_at), 0),
COUNT(*) FILTER (WHERE resolved_at IS NULL)
FROM incidents WHERE group_key = $1`,
sw.groupKey()).Scan(&lastTriggered, &open); err != nil {
FROM incidents WHERE team_id = $1 AND group_key = $2`,
sw.teamID, sw.groupKey()).Scan(&lastTriggered, &open); err != nil {
return err
}
if open > 0 || sw.receivedAt <= lastTriggered {
@@ -314,7 +323,9 @@ func deadmanDied(ctx context.Context, db *sql.DB, cfg DeadmanConfig, notify Noti
sev = &severity
}
incidentID, err := openIncident(ctx, tx, notify, sw.groupKey(),
// The incident opens in the team whose integration received the heartbeat:
// the switch belongs to whoever is watching that source, not to the install.
incidentID, err := openIncident(ctx, tx, notify, sw.teamID, sw.groupKey(),
"No heartbeat from "+sw.matcher.String(), sw.labels, sev)
if err != nil {
return err
@@ -343,7 +354,8 @@ func deadmanRecovered(ctx context.Context, db *sql.DB, sw deadmanAlert) error {
var incidentID int64
switch err := db.QueryRowContext(ctx, `
SELECT id FROM incidents
WHERE group_key = $1 AND resolved_at IS NULL`, sw.groupKey()).Scan(&incidentID); {
WHERE team_id = $1 AND group_key = $2 AND resolved_at IS NULL`,
sw.teamID, sw.groupKey()).Scan(&incidentID); {
case err == sql.ErrNoRows:
return nil
case err != nil:
@@ -378,3 +390,113 @@ func deadmanRecovered(ctx context.Context, db *sql.DB, sw deadmanAlert) error {
log.Printf("deadman: %s is back, resolved incident %d", sw.matcher.String(), incidentID)
return nil
}
// ---------------------------------------------------------------------------
// Per-team configuration
// ---------------------------------------------------------------------------
// deadmanConfigForTeam reads one team's switches. A team with no row, or with
// nothing configured, gets a disabled config — which is the right answer rather
// than an error: most teams watch no heartbeat at all.
func deadmanConfigForTeam(ctx context.Context, q querier, teamID int64) (DeadmanConfig, error) {
var matchers, severity string
var timeout int64
err := q.QueryRowContext(ctx,
"SELECT matchers, timeout_seconds, severity FROM deadman_configs WHERE team_id = $1",
teamID).Scan(&matchers, &timeout, &severity)
if err == sql.ErrNoRows {
return DeadmanConfig{}, nil
}
if err != nil {
return DeadmanConfig{}, err
}
return parseDeadmanQuietly(matchers, time.Duration(timeout)*time.Second, severity), nil
}
// deadmanConfigs reads every team's switches in one query, for the sweeper.
func deadmanConfigs(ctx context.Context, db *sql.DB) (map[int64]DeadmanConfig, error) {
rows, err := db.QueryContext(ctx,
"SELECT team_id, matchers, timeout_seconds, severity FROM deadman_configs")
if err != nil {
return nil, err
}
defer rows.Close()
out := map[int64]DeadmanConfig{}
for rows.Next() {
var teamID, timeout int64
var matchers, severity string
if err := rows.Scan(&teamID, &matchers, &timeout, &severity); err != nil {
return nil, err
}
cfg := parseDeadmanQuietly(matchers, time.Duration(timeout)*time.Second, severity)
if cfg.enabled() {
out[teamID] = cfg
}
}
return out, rows.Err()
}
// SeedDeadmanConfigs gives every team without a row the server's environment
// configuration, so the install that upgrades into per-team switches keeps
// watching exactly what it was watching before.
//
// Idempotent, and never overwrites: once a team has a row it owns its own
// configuration, and a redeploy must not quietly put the environment's value
// back over an owner's edit.
//
// A team created after startup gets no row and therefore watches nothing until
// its owner says otherwise. That is deliberate: inheriting an install-wide
// heartbeat would page a new team about a source it has never heard of, and a
// switch nobody chose is the kind that gets muted rather than fixed.
func SeedDeadmanConfigs(ctx context.Context, db *sql.DB, cfg DeadmanConfig) error {
matchers := make([]string, 0, len(cfg.Matchers))
for _, m := range cfg.Matchers {
parts := []string{"alertname=" + m.Name}
for k, v := range m.Labels {
parts = append(parts, k+"="+v)
}
sort.Strings(parts[1:])
matchers = append(matchers, strings.Join(parts, ","))
}
_, err := db.ExecContext(ctx, `
INSERT INTO deadman_configs (team_id, matchers, timeout_seconds, severity)
SELECT id, $1, $2, $3 FROM teams
ON CONFLICT (team_id) DO NOTHING`,
strings.Join(matchers, "; "), int64(cfg.Timeout.Seconds()), cfg.Severity)
return err
}
// parseDeadmanQuietly is ParseDeadmanConfig without the startup logging: a
// team's configuration is read on every sweep and every webhook, and logging it
// each time would bury everything else.
func parseDeadmanQuietly(matchers string, timeout time.Duration, severity string) DeadmanConfig {
cfg := DeadmanConfig{Timeout: timeout, Severity: severity}
for _, entry := range strings.Split(matchers, ";") {
entry = strings.TrimSpace(entry)
if entry == "" {
continue
}
m := DeadmanMatcher{Labels: map[string]string{}}
malformed := false
for _, cond := range strings.Split(entry, ",") {
k, v, ok := strings.Cut(cond, "=")
k, v = strings.TrimSpace(k), strings.TrimSpace(v)
if !ok || k == "" || v == "" {
malformed = true
break
}
if k == "alertname" {
m.Name = v
continue
}
m.Labels[k] = v
}
if malformed || m.Name == "" {
continue
}
cfg.Matchers = append(cfg.Matchers, m)
}
return cfg
}
+121
View File
@@ -2,6 +2,7 @@ package api_test
import (
"net/http"
"strings"
"testing"
"time"
@@ -469,3 +470,123 @@ func TestDeadman_DisabledConfigIsInert(t *testing.T) {
t.Errorf("expected the generic sweeper to own the alert, got %v", source)
}
}
// ---------------------------------------------------------------------------
// Per-team configuration
// ---------------------------------------------------------------------------
// Each team decides for itself what a heartbeat is. The same alert is a
// heartbeat in one team and an ordinary problem in another.
func TestDeadman_ConfigurationIsPerTeam(t *testing.T) {
s, _ := deadmanTS(t, deadmanCfg())
watched := newTeam(t, s, "watched")
unwatched := newTeam(t, s, "unwatched")
// Only the first team calls Watchdog a heartbeat.
resp := s.req(t, http.MethodPut, "/api/teams/"+id64(watched.id)+"/deadman", map[string]any{
"matchers": "alertname=Watchdog",
"timeout_seconds": 3600,
"severity": "critical",
})
resp.Body.Close()
if resp.StatusCode != http.StatusOK {
t.Fatalf("configure the watched team: %d", resp.StatusCode)
}
postToIntegration(t, s, watched.key, "fp-watched", "Watchdog")
postToIntegration(t, s, unwatched.key, "fp-unwatched", "Watchdog")
// A heartbeat opens nothing where it is one; an ordinary alert opens an
// incident where it is not.
if got := len(list(t, watched.call(http.MethodGet, "/api/incidents", nil))); got != 0 {
t.Errorf("the watched team's heartbeat opened %d incident(s), want 0", got)
}
if got := len(list(t, unwatched.call(http.MethodGet, "/api/incidents", nil))); got != 1 {
t.Errorf("the unwatched team's Watchdog opened %d incident(s), want 1", got)
}
// Silence pages only the team that is watching.
s.exec(t, "UPDATE alerts SET received_at = $1 WHERE fingerprint = $2",
time.Now().Add(-2*time.Hour).Unix(), "fp-watched")
s.exec(t, "UPDATE alerts SET received_at = $1 WHERE fingerprint = $2",
time.Now().Add(-2*time.Hour).Unix(), "fp-unwatched")
sweep(t, s, noArchive)
watchedIncidents := list(t, watched.call(http.MethodGet, "/api/incidents", nil))
if len(watchedIncidents) != 1 {
t.Fatalf("silence opened %d incident(s) for the watching team, want 1", len(watchedIncidents))
}
if title := watchedIncidents[0]["title"].(string); title != "No heartbeat from Watchdog" {
t.Errorf("unexpected incident title %q", title)
}
if teamID := int64(watchedIncidents[0]["team_id"].(float64)); teamID != watched.id {
t.Errorf("the incident opened in team %d, want %d", teamID, watched.id)
}
// The unwatched team's alert went stale the ordinary way, so it has the one
// incident it always had — not a second, dead man's switch one.
if got := len(list(t, unwatched.call(http.MethodGet, "/api/incidents", nil))); got != 1 {
t.Errorf("the unwatched team ended with %d incident(s), want 1", got)
}
}
// Configuration is an owner's to change and a member's to read, like the rest of
// a team's settings.
func TestDeadman_ConfigurationIsOwnerOnly(t *testing.T) {
s, _ := deadmanTS(t, deadmanCfg())
team := newTeam(t, s, "red")
// A plain member of that team.
var user struct {
ID int64 `json:"id"`
}
decode(t, s.req(t, http.MethodPost, "/api/users",
map[string]string{"username": "plain", "email": "plain@test.com"}), &user)
s.req(t, http.MethodPost, "/api/teams/"+id64(team.id)+"/members",
map[string]any{"user_id": user.ID, "role": "member"}).Body.Close()
var key struct {
Key string `json:"key"`
}
decode(t, s.req(t, http.MethodPost, "/api/users/"+id64(user.ID)+"/api-keys",
map[string]string{"name": "test"}), &key)
req, _ := http.NewRequest(http.MethodPut,
s.URL+"/api/teams/"+id64(team.id)+"/deadman",
strings.NewReader(`{"matchers":"alertname=Watchdog","timeout_seconds":60}`))
req.Header.Set("Authorization", "Bearer "+key.Key)
req.Header.Set("Content-Type", "application/json")
resp, err := http.DefaultClient.Do(req)
if err != nil {
t.Fatalf("put: %v", err)
}
resp.Body.Close()
if resp.StatusCode != http.StatusForbidden {
t.Errorf("a member editing the switches: expected 403, got %d", resp.StatusCode)
}
read, _ := http.NewRequest(http.MethodGet, s.URL+"/api/teams/"+id64(team.id)+"/deadman", nil)
read.Header.Set("Authorization", "Bearer "+key.Key)
got, err := http.DefaultClient.Do(read)
if err != nil {
t.Fatalf("get: %v", err)
}
got.Body.Close()
if got.StatusCode != http.StatusOK {
t.Errorf("a member reading the switches: expected 200, got %d", got.StatusCode)
}
}
// A matcher with no alertname watches nothing, silently, which is the failure
// this feature exists to prevent — so it is refused at the door.
func TestDeadman_UnusableMatchersAreRejected(t *testing.T) {
s, _ := deadmanTS(t, deadmanCfg())
resp := s.req(t, http.MethodPut, "/api/teams/"+defaultTeam+"/deadman", map[string]any{
"matchers": "cluster=prod",
"timeout_seconds": 900,
})
resp.Body.Close()
if resp.StatusCode != http.StatusBadRequest {
t.Errorf("expected 400 for a matcher with no alertname, got %d", resp.StatusCode)
}
}
+506
View File
@@ -0,0 +1,506 @@
package api
import (
"context"
"database/sql"
"log"
"net/http"
"strconv"
"strings"
"time"
)
// evEscalated records a rung of the ladder on the incident's timeline: which
// level, and who it woke.
const evEscalated = "escalated"
// escalationPolicy is a team's ladder, loaded whole. It is small — a handful of
// levels with a few targets each — and every use needs all of it, so there is
// no point reading it a level at a time.
type escalationPolicy struct {
teamID int64
repeatCount int64
fallbackTopic string
levels []escalationLevel
}
type escalationLevel struct {
id int64
position int64
timeout time.Duration
targets []escalationTarget
}
type escalationTarget struct {
kind string // "user" or "oncall"
userID *int64
}
// configured reports whether this team has anything to escalate through. A
// policy row with no levels is the same as no policy: the team gets the
// pre-escalation behaviour, which is reminders on the assignee's topic.
func (p *escalationPolicy) configured() bool { return p != nil && len(p.levels) > 0 }
// level returns the level at a 1-based position.
func (p *escalationPolicy) level(pos int64) (escalationLevel, bool) {
for _, l := range p.levels {
if l.position == pos {
return l, true
}
}
return escalationLevel{}, false
}
// loadEscalationPolicy reads one team's ladder. A team with no policy row
// returns nil, which every caller treats as "not configured" rather than as an
// error: most teams will never set one up.
func loadEscalationPolicy(ctx context.Context, q querier, teamID int64) (*escalationPolicy, error) {
p := &escalationPolicy{teamID: teamID}
err := q.QueryRowContext(ctx,
"SELECT repeat_count, fallback_topic FROM escalation_policies WHERE team_id = $1",
teamID).Scan(&p.repeatCount, &p.fallbackTopic)
if err == sql.ErrNoRows {
return nil, nil
}
if err != nil {
return nil, err
}
rows, err := q.QueryContext(ctx, `
SELECT l.id, l.position, l.timeout_seconds, t.kind, t.user_id
FROM escalation_levels l
LEFT JOIN escalation_targets t ON t.level_id = l.id
WHERE l.team_id = $1
ORDER BY l.position, t.id`, teamID)
if err != nil {
return nil, err
}
defer rows.Close()
byPosition := map[int64]int{} // position -> index in p.levels
for rows.Next() {
var id, position, timeout int64
var kind *string
var userID *int64
if err := rows.Scan(&id, &position, &timeout, &kind, &userID); err != nil {
return nil, err
}
idx, seen := byPosition[position]
if !seen {
p.levels = append(p.levels, escalationLevel{
id: id,
position: position,
timeout: time.Duration(timeout) * time.Second,
})
idx = len(p.levels) - 1
byPosition[position] = idx
}
// LEFT JOIN: a level with no targets yet still produces a row, with a
// NULL kind. It is a rung that pages nobody, which the API refuses to
// store but an older row could still hold.
if kind != nil {
p.levels[idx].targets = append(p.levels[idx].targets,
escalationTarget{kind: *kind, userID: userID})
}
}
return p, rows.Err()
}
// escalate advances every incident whose current level has run out of time.
//
// Runs on the notifier's tick, beside the reminder pass, because it is the same
// question asked differently: reminders ask "has this been ignored long
// enough to say it again", escalation asks "long enough to say it to somebody
// else". Sharing the tick means one query cadence and one outbox.
func escalate(ctx context.Context, db *sql.DB, cfg NotifyConfig) {
rows, err := db.QueryContext(ctx, `
SELECT i.id, i.team_id, i.escalation_level, i.escalation_level_at, i.escalation_round
FROM incidents i
JOIN escalation_policies p ON p.team_id = i.team_id
WHERE i.resolved_at IS NULL
AND i.archived_at IS NULL
AND i.status = 'triggered'
AND (i.snoozed_until IS NULL OR i.snoozed_until <= $1)
AND i.escalation_level > 0`, time.Now().Unix())
if err != nil {
log.Printf("escalation: find due: %v", err)
return
}
type pending struct {
incidentID, teamID, level, round int64
levelAt int64
}
var due []pending
for rows.Next() {
var p pending
var levelAt *int64
if err := rows.Scan(&p.incidentID, &p.teamID, &p.level, &levelAt, &p.round); err != nil {
rows.Close()
log.Printf("escalation: scan: %v", err)
return
}
if levelAt == nil {
continue
}
p.levelAt = *levelAt
due = append(due, p)
}
rows.Close()
if err := rows.Err(); err != nil {
log.Printf("escalation: iterate: %v", err)
return
}
now := time.Now()
for _, d := range due {
policy, err := loadEscalationPolicy(ctx, db, d.teamID)
if err != nil {
log.Printf("escalation: load policy for team %d: %v", d.teamID, err)
continue
}
if !policy.configured() {
continue
}
current, ok := policy.level(d.level)
if !ok {
continue
}
if now.Sub(time.Unix(d.levelAt, 0)) < current.timeout {
continue
}
if err := advanceEscalation(ctx, db, cfg, policy, d.incidentID, d.level, d.round, now); err != nil {
log.Printf("escalation: advance incident %d: %v", d.incidentID, err)
}
}
}
// advanceEscalation moves one incident to its next rung, or off the end of the
// ladder.
//
// The whole move is one transaction: the level, the page and the timeline entry
// are one event, and an incident recorded as being at level 3 that nobody at
// level 3 was told about is the worst of the possible half-states.
func advanceEscalation(ctx context.Context, db *sql.DB, cfg NotifyConfig, policy *escalationPolicy, incidentID, level, round int64, now time.Time) error {
tx, err := db.BeginTx(ctx, nil)
if err != nil {
return err
}
defer tx.Rollback() //nolint:errcheck
next := level + 1
nextRound := round
if _, ok := policy.level(next); !ok {
// Off the end. Either start the chain again, or make the last call.
if round < policy.repeatCount {
next, nextRound = 1, round+1
} else {
if err := escalationExhausted(ctx, tx, policy, incidentID, now); err != nil {
return err
}
return tx.Commit()
}
}
target, ok := policy.level(next)
if !ok {
return nil
}
paged, err := pageLevel(ctx, tx, cfg, policy, incidentID, target)
if err != nil {
return err
}
if _, err := tx.ExecContext(ctx, `
UPDATE incidents
SET escalation_level = $1, escalation_level_at = $2, escalation_round = $3
WHERE id = $4`, next, now.Unix(), nextRound, incidentID); err != nil {
return err
}
detail := "level " + strconv.FormatInt(next, 10)
if nextRound > round {
detail += " (round " + strconv.FormatInt(nextRound+1, 10) + ")"
}
if len(paged) > 0 {
detail += ": " + strings.Join(paged, ", ")
} else {
// Worth recording loudly: the rung exists, its turn came, and it woke
// nobody. That is a policy that looks configured and is not.
detail += ": nobody reachable"
}
if err := logEvent(ctx, tx, incidentID, evEscalated, nil, nil, &detail); err != nil {
return err
}
return tx.Commit()
}
// escalationExhausted is the end of the line: the fallback topic, once, and a
// timeline entry saying the ladder is finished. The incident stays triggered —
// escalation running out is not the same as somebody answering.
func escalationExhausted(ctx context.Context, tx *sql.Tx, policy *escalationPolicy, incidentID int64, now time.Time) error {
detail := "escalation exhausted"
if policy.fallbackTopic != "" {
if err := enqueueNotification(ctx, tx, incidentID, nil, policy.fallbackTopic, notifyEscalated); err != nil {
return err
}
detail += ": paged " + policy.fallbackTopic
} else {
detail += ": no fallback topic configured"
}
// Level 0 again, so the sweep stops considering it. The round counter is
// left where it is, as the record of how far it got.
if _, err := tx.ExecContext(ctx,
"UPDATE incidents SET escalation_level = 0, escalation_level_at = NULL WHERE id = $1",
incidentID); err != nil {
return err
}
return logEvent(ctx, tx, incidentID, evEscalated, nil, nil, &detail)
}
// pageLevel notifies every target of one level and reports who was woken.
//
// Each target gets its own outbox row, so each gets its own Acknowledge token:
// the button in a notification must acknowledge as the person holding the
// phone, not as whoever was paged first.
func pageLevel(ctx context.Context, tx *sql.Tx, cfg NotifyConfig, policy *escalationPolicy, incidentID int64, level escalationLevel) ([]string, error) {
var paged []string
seen := map[int64]bool{}
for _, t := range level.targets {
userID := t.userID
if t.kind == "oncall" {
onCall, err := currentOnCall(ctx, tx, policy.teamID)
if err != nil {
return nil, err
}
if onCall == nil {
continue
}
userID = onCall
}
if userID == nil || seen[*userID] {
continue
}
seen[*userID] = true
var topic *string
var username string
if err := tx.QueryRowContext(ctx,
"SELECT ntfy_topic, username FROM users WHERE id = $1 AND disabled_at IS NULL",
*userID).Scan(&topic, &username); err != nil {
// A disabled or deleted account is not an error in the middle of an
// escalation: it is a target that cannot be woken, and the next
// level is the answer to that.
continue
}
if topic == nil || *topic == "" {
continue
}
if err := enqueueNotification(ctx, tx, incidentID, userID, *topic, notifyEscalated); err != nil {
return nil, err
}
paged = append(paged, username)
}
return paged, nil
}
// startEscalation puts a newly opened incident on the first rung, when its team
// has a ladder. Called from openIncident, inside the same transaction, so an
// incident is never briefly open with no escalation clock running.
func startEscalation(ctx context.Context, q querier, incidentID, teamID int64) error {
policy, err := loadEscalationPolicy(ctx, q, teamID)
if err != nil || !policy.configured() {
return err
}
_, err = q.ExecContext(ctx,
"UPDATE incidents SET escalation_level = 1, escalation_level_at = $1 WHERE id = $2",
time.Now().Unix(), incidentID)
return err
}
// stopEscalation takes an incident off the ladder. Acknowledging or resolving
// is somebody saying "I have this", and continuing to wake people after that is
// the behaviour that teaches people to ignore the tool.
func stopEscalation(ctx context.Context, q querier, incidentID int64) error {
_, err := q.ExecContext(ctx,
"UPDATE incidents SET escalation_level = 0, escalation_level_at = NULL WHERE id = $1",
incidentID)
return err
}
// handleGetEscalation returns a team's ladder.
func handleGetEscalation(db *sql.DB) http.HandlerFunc {
return func(w http.ResponseWriter, r *http.Request) {
teamID, ok := teamParam(w, r)
if !ok {
return
}
if !requireTeamMember(w, r, teamID) {
return
}
policy, err := loadEscalationPolicy(r.Context(), db, teamID)
if err != nil {
respond(w, http.StatusInternalServerError, errResp("internal error"))
return
}
respond(w, http.StatusOK, escalationResponse(policy, teamID))
}
}
type escalationLevelJSON struct {
Position int64 `json:"position"`
TimeoutSeconds int64 `json:"timeout_seconds"`
Targets []escalationTargetJSON `json:"targets"`
}
type escalationTargetJSON struct {
Kind string `json:"kind"`
UserID *int64 `json:"user_id,omitempty"`
}
type escalationJSON struct {
TeamID int64 `json:"team_id"`
RepeatCount int64 `json:"repeat_count"`
FallbackTopic string `json:"fallback_topic"`
Levels []escalationLevelJSON `json:"levels"`
}
func escalationResponse(p *escalationPolicy, teamID int64) escalationJSON {
out := escalationJSON{TeamID: teamID, Levels: []escalationLevelJSON{}}
if p == nil {
return out
}
out.RepeatCount = p.repeatCount
out.FallbackTopic = p.fallbackTopic
for _, l := range p.levels {
level := escalationLevelJSON{
Position: l.position,
TimeoutSeconds: int64(l.timeout.Seconds()),
Targets: []escalationTargetJSON{},
}
for _, t := range l.targets {
level.Targets = append(level.Targets, escalationTargetJSON{Kind: t.kind, UserID: t.userID})
}
out.Levels = append(out.Levels, level)
}
return out
}
// handleSetEscalation replaces a team's ladder wholesale.
//
// Replace rather than patch: the levels are an order, and an API that edits one
// rung has to answer what happens to the numbering of the others. Sending the
// whole ladder makes the order the client's to decide and the server's to
// store, and makes an edit atomic — there is no moment where level 2 exists
// twice.
func handleSetEscalation(db *sql.DB) http.HandlerFunc {
return func(w http.ResponseWriter, r *http.Request) {
teamID, ok := teamParam(w, r)
if !ok {
return
}
if !requireTeamOwner(w, r, teamID) {
return
}
var req escalationJSON
if err := decodeJSON(r, &req); err != nil {
respond(w, http.StatusBadRequest, errResp("invalid request body"))
return
}
if req.RepeatCount < 0 || req.RepeatCount > 10 {
respond(w, http.StatusBadRequest, errResp("repeat_count must be between 0 and 10"))
return
}
for i, l := range req.Levels {
if l.TimeoutSeconds <= 0 {
respond(w, http.StatusBadRequest, errResp("every level needs a timeout"))
return
}
if len(l.Targets) == 0 {
// A rung that pages nobody is not a delay, it is a silence with
// a number on it.
respond(w, http.StatusBadRequest,
errResp("level "+strconv.FormatInt(int64(i+1), 10)+" has no targets"))
return
}
for _, t := range l.Targets {
switch t.Kind {
case "oncall":
if t.UserID != nil {
respond(w, http.StatusBadRequest, errResp("an oncall target takes no user_id"))
return
}
case "user":
if t.UserID == nil {
respond(w, http.StatusBadRequest, errResp("a user target needs a user_id"))
return
}
default:
respond(w, http.StatusBadRequest, errResp("target kind must be user or oncall"))
return
}
}
}
tx, err := db.BeginTx(r.Context(), nil)
if err != nil {
respond(w, http.StatusInternalServerError, errResp("internal error"))
return
}
defer tx.Rollback() //nolint:errcheck
if _, err := tx.ExecContext(r.Context(), `
INSERT INTO escalation_policies (team_id, repeat_count, fallback_topic, updated_at)
VALUES ($1, $2, $3, `+nowEpoch+`)
ON CONFLICT (team_id) DO UPDATE SET
repeat_count = excluded.repeat_count,
fallback_topic = excluded.fallback_topic,
updated_at = excluded.updated_at`,
teamID, req.RepeatCount, req.FallbackTopic); err != nil {
respond(w, http.StatusInternalServerError, errResp("internal error"))
return
}
// The levels are replaced, not merged; the cascade takes the targets.
if _, err := tx.ExecContext(r.Context(),
"DELETE FROM escalation_levels WHERE team_id = $1", teamID); err != nil {
respond(w, http.StatusInternalServerError, errResp("internal error"))
return
}
for i, l := range req.Levels {
var levelID int64
if err := tx.QueryRowContext(r.Context(), `
INSERT INTO escalation_levels (team_id, position, timeout_seconds)
VALUES ($1, $2, $3) RETURNING id`,
teamID, int64(i+1), l.TimeoutSeconds).Scan(&levelID); err != nil {
respond(w, http.StatusInternalServerError, errResp("internal error"))
return
}
for _, t := range l.Targets {
if _, err := tx.ExecContext(r.Context(), `
INSERT INTO escalation_targets (level_id, kind, user_id)
VALUES ($1, $2, $3)`, levelID, t.Kind, t.UserID); err != nil {
// The only foreign key here is the user.
respond(w, http.StatusBadRequest, errResp("unknown user in targets"))
return
}
}
}
if err := tx.Commit(); err != nil {
respond(w, http.StatusInternalServerError, errResp("internal error"))
return
}
policy, err := loadEscalationPolicy(r.Context(), db, teamID)
if err != nil {
respond(w, http.StatusInternalServerError, errResp("internal error"))
return
}
respond(w, http.StatusOK, escalationResponse(policy, teamID))
}
}
+385
View File
@@ -0,0 +1,385 @@
package api_test
import (
"net/http"
"strings"
"testing"
"time"
"git.ryuvia.com/niklas/terdut-server/internal/api"
)
// teamUser creates a user in the default team with an ntfy topic, so they can
// actually be paged.
func teamUser(t *testing.T, s *ts, username, topic string) int64 {
t.Helper()
var user struct {
ID int64 `json:"id"`
}
decode(t, s.req(t, http.MethodPost, "/api/users",
map[string]string{"username": username, "email": username + "@test.com"}), &user)
resp := s.req(t, http.MethodPost, "/api/teams/"+defaultTeam+"/members",
map[string]any{"user_id": user.ID, "role": "member"})
resp.Body.Close()
setTopic(t, s, int(user.ID), topic)
return user.ID
}
// Escalation is all timeouts, and there is no fake clock in this package. The
// tests back-date escalation_level_at instead, which is the same trick the dead
// man's switch tests use on received_at: the sweeper reads a stored timestamp,
// so moving the timestamp is moving the clock.
// ladder configures the default team with two levels: the rota first, then a
// named person, then the fallback topic.
func ladder(t *testing.T, s *ts, secondUserID int64, repeat int64, fallback string) {
t.Helper()
resp := s.req(t, http.MethodPut, "/api/teams/"+defaultTeam+"/escalation", map[string]any{
"repeat_count": repeat,
"fallback_topic": fallback,
"levels": []map[string]any{
{"timeout_seconds": 300, "targets": []map[string]any{{"kind": "oncall"}}},
{"timeout_seconds": 300, "targets": []map[string]any{{"kind": "user", "user_id": secondUserID}}},
},
})
defer resp.Body.Close()
if resp.StatusCode != http.StatusOK {
t.Fatalf("configure the ladder: %d", resp.StatusCode)
}
}
// overdue back-dates an incident's current level so its timeout has passed.
func overdue(t *testing.T, s *ts, incidentID int64) {
t.Helper()
s.exec(t, "UPDATE incidents SET escalation_level_at = $1 WHERE id = $2",
time.Now().Add(-time.Hour).Unix(), incidentID)
}
func escalationLevel(t *testing.T, s *ts, incidentID int64) (level, round int64) {
t.Helper()
if err := s.db.QueryRow(
"SELECT escalation_level, escalation_round FROM incidents WHERE id = $1",
incidentID).Scan(&level, &round); err != nil {
t.Fatalf("read escalation state: %v", err)
}
return level, round
}
// The whole point: nobody answers, so somebody else is woken.
func TestEscalation_PagesTheNextLevel(t *testing.T) {
s, f := notifyTS(t, api.NotifyConfig{PublicURL: "https://terdut.example.com", RepeatEvery: 15 * time.Minute})
second := teamUser(t, s, "second", "terdut-second")
ladder(t, s, second, 0, "terdut-fallback")
postWebhook(t, s, []map[string]any{
amAlert("fp-esc", "DiskFull", "firing", "2026-05-20T10:00:00Z", zeroTime, nil),
})
s.sweepNotify(t)
// Level 1 is the rota, so the first page went to the admin.
if level, _ := escalationLevel(t, s, 1); level != 1 {
t.Fatalf("a new incident should start at level 1, got %d", level)
}
if got := f.topicsSince(t); len(got) == 0 || got[0] != "terdut-admin" {
t.Fatalf("the first page should go to the on-call user, went to %v", got)
}
// Time passes with no acknowledgement.
f.forget()
overdue(t, s, 1)
s.sweepNotify(t)
if level, _ := escalationLevel(t, s, 1); level != 2 {
t.Errorf("expected level 2, got %d", level)
}
if got := f.topicsSince(t); len(got) != 1 || got[0] != "terdut-second" {
t.Errorf("level 2 should page the named user, paged %v", got)
}
// And the timeline says so, which is what somebody reads afterwards to
// understand why their phone rang at 04:00.
timeline := list(t, s.req(t, http.MethodGet, "/api/incidents/1/timeline", nil))
found := ""
for _, e := range timeline {
if e["type"] == "escalated" {
found, _ = e["detail"].(string)
}
}
if found == "" {
t.Error("the timeline should record the escalation")
} else if !strings.HasPrefix(found, "level 2") || !strings.Contains(found, "second") {
t.Errorf("the escalation entry should say which level and who: %q", found)
}
}
// Acknowledging is somebody saying "I have this". Nobody else should be woken.
func TestEscalation_AcknowledgementStopsIt(t *testing.T) {
s, f := notifyTS(t, api.NotifyConfig{PublicURL: "https://terdut.example.com", RepeatEvery: 15 * time.Minute})
second := teamUser(t, s, "second", "terdut-second")
ladder(t, s, second, 0, "terdut-fallback")
postWebhook(t, s, []map[string]any{
amAlert("fp-ack", "DiskFull", "firing", "2026-05-20T10:00:00Z", zeroTime, nil),
})
s.sweepNotify(t)
s.req(t, http.MethodPost, "/api/incidents/1/acknowledge", nil).Body.Close()
if level, _ := escalationLevel(t, s, 1); level != 0 {
t.Errorf("acknowledging should take the incident off the ladder, level is %d", level)
}
f.forget()
overdue(t, s, 1) // no-op: level is 0, so there is nothing due
s.sweepNotify(t)
if got := f.topicsSince(t); len(got) != 0 {
t.Errorf("an acknowledged incident should page nobody, paged %v", got)
}
}
// Resolving stops it too, and by the same mechanism.
func TestEscalation_ResolutionStopsIt(t *testing.T) {
s, f := notifyTS(t, api.NotifyConfig{PublicURL: "https://terdut.example.com", RepeatEvery: 15 * time.Minute})
second := teamUser(t, s, "second", "terdut-second")
ladder(t, s, second, 0, "terdut-fallback")
postWebhook(t, s, []map[string]any{
amAlert("fp-res", "DiskFull", "firing", "2026-05-20T10:00:00Z", zeroTime, nil),
})
s.sweepNotify(t)
s.req(t, http.MethodPost, "/api/incidents/1/resolve", nil).Body.Close()
f.forget()
overdue(t, s, 1)
s.sweepNotify(t)
if level, _ := escalationLevel(t, s, 1); level != 0 {
t.Errorf("a resolved incident should be off the ladder, level is %d", level)
}
}
// Snoozing is a deliberate "not now", so the ladder waits rather than carrying
// on without the person who asked for quiet.
func TestEscalation_SnoozePausesIt(t *testing.T) {
s, f := notifyTS(t, api.NotifyConfig{PublicURL: "https://terdut.example.com", RepeatEvery: 15 * time.Minute})
second := teamUser(t, s, "second", "terdut-second")
ladder(t, s, second, 0, "terdut-fallback")
postWebhook(t, s, []map[string]any{
amAlert("fp-snooze", "DiskFull", "firing", "2026-05-20T10:00:00Z", zeroTime, nil),
})
s.sweepNotify(t)
resp := s.req(t, http.MethodPost, "/api/incidents/1/snooze", map[string]any{"duration": "1h"})
resp.Body.Close()
f.forget()
overdue(t, s, 1)
s.sweepNotify(t)
if level, _ := escalationLevel(t, s, 1); level != 1 {
t.Errorf("a snoozed incident should stay where it is, level is %d", level)
}
if got := f.topicsSince(t); len(got) != 0 {
t.Errorf("a snoozed incident should page nobody, paged %v", got)
}
// When the snooze ends, the ladder picks up where it left off.
s.exec(t, "UPDATE incidents SET snoozed_until = $1 WHERE id = 1", time.Now().Add(-time.Minute).Unix())
s.sweepNotify(t)
if level, _ := escalationLevel(t, s, 1); level != 2 {
t.Errorf("after the snooze the ladder should resume, level is %d", level)
}
}
// Running out of ladder pages the team's fallback topic once, and says so.
func TestEscalation_ExhaustionPagesTheFallback(t *testing.T) {
s, f := notifyTS(t, api.NotifyConfig{PublicURL: "https://terdut.example.com", RepeatEvery: 15 * time.Minute})
second := teamUser(t, s, "second", "terdut-second")
ladder(t, s, second, 0, "terdut-fallback")
postWebhook(t, s, []map[string]any{
amAlert("fp-end", "DiskFull", "firing", "2026-05-20T10:00:00Z", zeroTime, nil),
})
s.sweepNotify(t)
overdue(t, s, 1)
s.sweepNotify(t) // level 2
f.forget()
overdue(t, s, 1)
s.sweepNotify(t) // off the end
if got := f.topicsSince(t); len(got) != 1 || got[0] != "terdut-fallback" {
t.Errorf("exhaustion should page the fallback topic once, paged %v", got)
}
level, _ := escalationLevel(t, s, 1)
if level != 0 {
t.Errorf("an exhausted ladder should stop asking, level is %d", level)
}
// The incident is still open: running out of people is not an answer.
var status string
if err := s.db.QueryRow("SELECT status FROM incidents WHERE id = 1").Scan(&status); err != nil {
t.Fatal(err)
}
if status != "triggered" {
t.Errorf("exhaustion must not resolve the incident, status is %q", status)
}
}
// repeat_count walks the whole ladder again before giving up.
func TestEscalation_RepeatsTheChain(t *testing.T) {
s, f := notifyTS(t, api.NotifyConfig{PublicURL: "https://terdut.example.com", RepeatEvery: 15 * time.Minute})
second := teamUser(t, s, "second", "terdut-second")
ladder(t, s, second, 1, "terdut-fallback") // one extra round
postWebhook(t, s, []map[string]any{
amAlert("fp-repeat", "DiskFull", "firing", "2026-05-20T10:00:00Z", zeroTime, nil),
})
s.sweepNotify(t)
overdue(t, s, 1)
s.sweepNotify(t) // level 2
f.forget()
overdue(t, s, 1)
s.sweepNotify(t) // back to level 1, round 2
level, round := escalationLevel(t, s, 1)
if level != 1 || round != 1 {
t.Errorf("expected level 1 round 1, got level %d round %d", level, round)
}
if got := f.topicsSince(t); len(got) != 1 || got[0] != "terdut-admin" {
t.Errorf("the second round should start at the top again, paged %v", got)
}
}
// A team without a ladder keeps exactly the behaviour it had, and never gets
// both a reminder and an escalation for the same silence.
func TestEscalation_WithoutAPolicyRemindersStillRun(t *testing.T) {
s, f := notifyTS(t, api.NotifyConfig{PublicURL: "https://terdut.example.com", RepeatEvery: 15 * time.Minute})
postWebhook(t, s, []map[string]any{
amAlert("fp-noesc", "DiskFull", "firing", "2026-05-20T10:00:00Z", zeroTime, nil),
})
s.sweepNotify(t)
// Age the first notification past the repeat interval.
f.forget()
s.exec(t, "UPDATE notifications SET created_at = $1, sent_at = $1",
time.Now().Add(-time.Hour).Unix())
s.sweepNotify(t)
if got := f.topicsSince(t); len(got) != 1 || got[0] != "terdut-admin" {
t.Errorf("without a ladder the reminder should still fire, paged %v", got)
}
if level, _ := escalationLevel(t, s, 1); level != 0 {
t.Errorf("an incident in a team with no ladder should not be on one, level is %d", level)
}
}
// With a ladder, reminders stop: two pages for one silence is how people learn
// to mute the tool.
func TestEscalation_WithAPolicyRemindersDoNotAlsoFire(t *testing.T) {
s, f := notifyTS(t, api.NotifyConfig{PublicURL: "https://terdut.example.com", RepeatEvery: 15 * time.Minute})
second := teamUser(t, s, "second", "terdut-second")
ladder(t, s, second, 0, "terdut-fallback")
postWebhook(t, s, []map[string]any{
amAlert("fp-both", "DiskFull", "firing", "2026-05-20T10:00:00Z", zeroTime, nil),
})
s.sweepNotify(t)
f.forget()
// Old enough for a reminder, but not yet due for escalation.
s.exec(t, "UPDATE notifications SET created_at = $1, sent_at = $1",
time.Now().Add(-time.Hour).Unix())
s.sweepNotify(t)
if got := f.topicsSince(t); len(got) != 0 {
t.Errorf("a team with a ladder should not also get reminders, paged %v", got)
}
}
// The API refuses a ladder that cannot page anybody.
func TestEscalation_RejectsAnUnusablePolicy(t *testing.T) {
s := newTS(t)
for _, c := range []struct {
name string
body map[string]any
}{
{"a level with no targets", map[string]any{
"levels": []map[string]any{{"timeout_seconds": 300, "targets": []map[string]any{}}},
}},
{"a level with no timeout", map[string]any{
"levels": []map[string]any{{"timeout_seconds": 0, "targets": []map[string]any{{"kind": "oncall"}}}},
}},
{"a user target with no user", map[string]any{
"levels": []map[string]any{{"timeout_seconds": 300, "targets": []map[string]any{{"kind": "user"}}}},
}},
{"an unknown target kind", map[string]any{
"levels": []map[string]any{{"timeout_seconds": 300, "targets": []map[string]any{{"kind": "everybody"}}}},
}},
{"an absurd repeat count", map[string]any{
"repeat_count": 99,
"levels": []map[string]any{{"timeout_seconds": 300, "targets": []map[string]any{{"kind": "oncall"}}}},
}},
} {
resp := s.req(t, http.MethodPut, "/api/teams/"+defaultTeam+"/escalation", c.body)
resp.Body.Close()
if resp.StatusCode != http.StatusBadRequest {
t.Errorf("%s: expected 400, got %d", c.name, resp.StatusCode)
}
}
}
// Editing the ladder is an owner's job; reading it is any member's.
func TestEscalation_OwnerOnlyToEdit(t *testing.T) {
s := newTS(t)
_, call := member(t, s, "plain")
resp := call(http.MethodPut, "/api/teams/"+defaultTeam+"/escalation", map[string]any{
"levels": []map[string]any{{"timeout_seconds": 300, "targets": []map[string]any{{"kind": "oncall"}}}},
})
resp.Body.Close()
if resp.StatusCode != http.StatusForbidden {
t.Errorf("a member editing the ladder: expected 403, got %d", resp.StatusCode)
}
resp = call(http.MethodGet, "/api/teams/"+defaultTeam+"/escalation", nil)
resp.Body.Close()
if resp.StatusCode != http.StatusOK {
t.Errorf("a member reading the ladder: expected 200, got %d", resp.StatusCode)
}
}
// A target who cannot be woken is not a reason to stop: the next level is the
// answer to an unreachable one.
func TestEscalation_SkipsUnreachableTargets(t *testing.T) {
s, f := notifyTS(t, api.NotifyConfig{PublicURL: "https://terdut.example.com", RepeatEvery: 15 * time.Minute})
// Second user has no ntfy topic at all.
var user struct {
ID int64 `json:"id"`
}
decode(t, s.req(t, http.MethodPost, "/api/users",
map[string]string{"username": "silent", "email": "silent@test.com"}), &user)
s.req(t, http.MethodPost, "/api/teams/"+defaultTeam+"/members",
map[string]any{"user_id": user.ID, "role": "member"}).Body.Close()
ladder(t, s, user.ID, 0, "terdut-fallback")
postWebhook(t, s, []map[string]any{
amAlert("fp-silent", "DiskFull", "firing", "2026-05-20T10:00:00Z", zeroTime, nil),
})
s.sweepNotify(t)
f.forget()
overdue(t, s, 1)
s.sweepNotify(t)
// Level 2 was entered even though it woke nobody, so the ladder keeps
// moving toward the fallback rather than stalling on a silent rung.
if level, _ := escalationLevel(t, s, 1); level != 2 {
t.Errorf("expected the ladder to advance past an unreachable target, level is %d", level)
}
if got := f.topicsSince(t); len(got) != 0 {
t.Errorf("a target with no topic should page nothing, paged %v", got)
}
}
+29 -7
View File
@@ -51,12 +51,20 @@ type querier interface {
}
const incidentSelectFrom = `
SELECT i.id, i.group_key, i.title, i.group_labels, i.status, i.severity,
SELECT i.id, i.team_id, t.name, i.group_key, i.title, i.group_labels, i.status, i.severity,
i.escalation_level,
-- When this level runs out. Computed here rather than in Go because
-- the timeout lives beside the level in the policy, and one join is
-- cheaper than a second query per incident in a list.
(SELECT i.escalation_level_at + el.timeout_seconds
FROM escalation_levels el
WHERE el.team_id = i.team_id AND el.position = i.escalation_level),
i.triggered_at,
i.acknowledged_by, i.acknowledged_at, ack.username,
i.assigned_to, asg.username, i.snoozed_until,
i.resolved_at, i.resolution_source, i.archived_at
FROM incidents i
JOIN teams t ON t.id = i.team_id
LEFT JOIN users ack ON ack.id = i.acknowledged_by
LEFT JOIN users asg ON asg.id = i.assigned_to`
@@ -64,10 +72,11 @@ func scanIncident(s scanner) (models.Incident, error) {
var i models.Incident
var groupLabelsJSON string
var triggeredAt int64
var ackAt, snoozedUntil, resolvedAt, archivedAt *int64
var ackAt, snoozedUntil, resolvedAt, archivedAt, escalationDue *int64
if err := s.Scan(
&i.ID, &i.GroupKey, &i.Title, &groupLabelsJSON, &i.Status, &i.Severity,
&i.ID, &i.TeamID, &i.TeamName, &i.GroupKey, &i.Title, &groupLabelsJSON, &i.Status, &i.Severity,
&i.EscalationLevel, &escalationDue,
&triggeredAt,
&i.AcknowledgedByID, &ackAt, &i.AcknowledgedByUser,
&i.AssignedToID, &i.AssignedToUser, &snoozedUntil,
@@ -82,6 +91,7 @@ func scanIncident(s scanner) (models.Incident, error) {
i.SnoozedUntil = unixPtr(snoozedUntil)
i.ResolvedAt = unixPtr(resolvedAt)
i.ArchivedAt = unixPtr(archivedAt)
i.EscalationDueAt = unixPtr(escalationDue)
return i, nil
}
@@ -113,12 +123,17 @@ func todayUTC() string {
return time.Now().UTC().Format("2006-01-02")
}
// currentOnCall returns today's on-call user, or nil when nobody is scheduled.
// A missing schedule entry is not an error — incidents just open unassigned.
func currentOnCall(ctx context.Context, q querier) (*int64, error) {
// currentOnCall returns a team's on-call user for today, or nil when nobody is
// scheduled. A missing schedule entry is not an error — incidents just open
// unassigned.
//
// Per team: each team keeps its own rota, so two teams can have two different
// people on call on the same day, which was the point of scoping the schedule.
func currentOnCall(ctx context.Context, q querier, teamID int64) (*int64, error) {
var userID int64
err := q.QueryRowContext(ctx,
"SELECT user_id FROM schedule_entries WHERE date = $1", todayUTC()).Scan(&userID)
"SELECT user_id FROM schedule_entries WHERE team_id = $1 AND date = $2",
teamID, todayUTC()).Scan(&userID)
if err == sql.ErrNoRows {
return nil, nil
}
@@ -229,6 +244,9 @@ func resolveIfSettled(ctx context.Context, q querier, incidentID int64) (bool, e
if n == 0 {
return false, nil
}
if err := stopEscalation(ctx, q, incidentID); err != nil {
return false, err
}
if err := logEvent(ctx, q, incidentID, evResolved, nil, nil, nil); err != nil {
return false, err
}
@@ -254,6 +272,10 @@ func acknowledgeIncident(ctx context.Context, q querier, incidentID, userID int6
if n, _ := res.RowsAffected(); n == 0 {
return false, nil
}
// Somebody has it: stop waking anybody else.
if err := stopEscalation(ctx, q, incidentID); err != nil {
return false, err
}
return true, logEvent(ctx, q, incidentID, evAcknowledged, &userID, nil, nil)
}
+42 -15
View File
@@ -19,6 +19,15 @@ func handleListIncidents(db *sql.DB) http.HandlerFunc {
where := []string{}
args := &sqlArgs{}
// The combined queue: every team the caller belongs to, in one list. A
// caller in no team sees an empty queue rather than everybody's.
where = append(where, "i.team_id = ANY("+args.add(callerTeamIDs(r.Context()))+")")
if team := q.Get("team_id"); team != "" {
if n, err := strconv.ParseInt(team, 10, 64); err == nil {
where = append(where, "i.team_id = "+args.add(n))
}
}
// Without an explicit status the queue shows open work, which is what an
// on-call person opens the tool to see.
if status := q.Get("status"); status != "" {
@@ -96,7 +105,7 @@ func handleListIncidents(db *sql.DB) http.HandlerFunc {
func handleGetIncident(db *sql.DB) http.HandlerFunc {
return func(w http.ResponseWriter, r *http.Request) {
id, ok := incidentIDParam(w, r)
id, ok := incidentIDParam(w, r, db)
if !ok {
return
}
@@ -119,7 +128,7 @@ func handleGetIncident(db *sql.DB) http.HandlerFunc {
func handleIncidentAlerts(db *sql.DB) http.HandlerFunc {
return func(w http.ResponseWriter, r *http.Request) {
id, ok := incidentIDParam(w, r)
id, ok := incidentIDParam(w, r, db)
if !ok {
return
}
@@ -137,7 +146,7 @@ func handleIncidentAlerts(db *sql.DB) http.HandlerFunc {
func handleIncidentTimeline(db *sql.DB) http.HandlerFunc {
return func(w http.ResponseWriter, r *http.Request) {
id, ok := incidentIDParam(w, r)
id, ok := incidentIDParam(w, r, db)
if !ok {
return
}
@@ -176,7 +185,7 @@ func handleIncidentTimeline(db *sql.DB) http.HandlerFunc {
func handleIncidentAcknowledge(db *sql.DB) http.HandlerFunc {
return func(w http.ResponseWriter, r *http.Request) {
id, ok := incidentIDParam(w, r)
id, ok := incidentIDParam(w, r, db)
if !ok {
return
}
@@ -199,7 +208,7 @@ func handleIncidentAcknowledge(db *sql.DB) http.HandlerFunc {
func handleIncidentUnacknowledge(db *sql.DB) http.HandlerFunc {
return func(w http.ResponseWriter, r *http.Request) {
id, ok := incidentIDParam(w, r)
id, ok := incidentIDParam(w, r, db)
if !ok {
return
}
@@ -223,7 +232,7 @@ func handleIncidentUnacknowledge(db *sql.DB) http.HandlerFunc {
// re-send of an alert that never stopped firing. Use snooze for "not now".
func handleIncidentResolve(db *sql.DB) http.HandlerFunc {
return func(w http.ResponseWriter, r *http.Request) {
id, ok := incidentIDParam(w, r)
id, ok := incidentIDParam(w, r, db)
if !ok {
return
}
@@ -234,6 +243,11 @@ func handleIncidentResolve(db *sql.DB) http.HandlerFunc {
time.Now().Unix(), incidentResolutionManual, id) {
return
}
// A person closing an incident is the clearest possible "I have this".
if err := stopEscalation(r.Context(), db, id); err != nil {
respond(w, http.StatusInternalServerError, errResp("internal error"))
return
}
if err := logEvent(r.Context(), db, id, evResolved, &user.ID, nil, nil); err != nil {
respond(w, http.StatusInternalServerError, errResp("internal error"))
return
@@ -244,7 +258,7 @@ func handleIncidentResolve(db *sql.DB) http.HandlerFunc {
func handleIncidentAssign(db *sql.DB) http.HandlerFunc {
return func(w http.ResponseWriter, r *http.Request) {
id, ok := incidentIDParam(w, r)
id, ok := incidentIDParam(w, r, db)
if !ok {
return
}
@@ -285,7 +299,7 @@ func handleIncidentAssign(db *sql.DB) http.HandlerFunc {
// {"duration": "2h"}.
func handleIncidentSnooze(db *sql.DB) http.HandlerFunc {
return func(w http.ResponseWriter, r *http.Request) {
id, ok := incidentIDParam(w, r)
id, ok := incidentIDParam(w, r, db)
if !ok {
return
}
@@ -340,7 +354,7 @@ func handleIncidentSnooze(db *sql.DB) http.HandlerFunc {
func handleIncidentUnsnooze(db *sql.DB) http.HandlerFunc {
return func(w http.ResponseWriter, r *http.Request) {
id, ok := incidentIDParam(w, r)
id, ok := incidentIDParam(w, r, db)
if !ok {
return
}
@@ -359,7 +373,7 @@ func handleIncidentUnsnooze(db *sql.DB) http.HandlerFunc {
func handleIncidentArchive(db *sql.DB) http.HandlerFunc {
return func(w http.ResponseWriter, r *http.Request) {
id, ok := incidentIDParam(w, r)
id, ok := incidentIDParam(w, r, db)
if !ok {
return
}
@@ -379,7 +393,7 @@ func handleIncidentArchive(db *sql.DB) http.HandlerFunc {
func handleIncidentUnarchive(db *sql.DB) http.HandlerFunc {
return func(w http.ResponseWriter, r *http.Request) {
id, ok := incidentIDParam(w, r)
id, ok := incidentIDParam(w, r, db)
if !ok {
return
}
@@ -401,7 +415,7 @@ func handleIncidentUnarchive(db *sql.DB) http.HandlerFunc {
// single query renders the whole story of an incident in order.
func handleCreateNote(db *sql.DB) http.HandlerFunc {
return func(w http.ResponseWriter, r *http.Request) {
id, ok := incidentIDParam(w, r)
id, ok := incidentIDParam(w, r, db)
if !ok {
return
}
@@ -448,7 +462,7 @@ func handleCreateNote(db *sql.DB) http.HandlerFunc {
// rest of the timeline is what actually happened, and is not editable.
func handleDeleteNote(db *sql.DB) http.HandlerFunc {
return func(w http.ResponseWriter, r *http.Request) {
id, ok := incidentIDParam(w, r)
id, ok := incidentIDParam(w, r, db)
if !ok {
return
}
@@ -479,19 +493,32 @@ func handleDeleteNote(db *sql.DB) http.HandlerFunc {
// Shared handler plumbing
// ---------------------------------------------------------------------------
func incidentIDParam(w http.ResponseWriter, r *http.Request) (int64, bool) {
// incidentIDParam reads {id} from the path AND confirms the incident belongs to
// a team the caller is in. Both in one place, deliberately: every incident route
// goes through here, so scoping cannot be forgotten by writing a new handler
// that only remembers the first half.
//
// An incident in somebody else's team is reported as not found rather than
// forbidden, because "there is an incident 41 you may not see" is itself
// something only that team should know.
func incidentIDParam(w http.ResponseWriter, r *http.Request, db *sql.DB) (int64, bool) {
id, err := strconv.ParseInt(chi.URLParam(r, "id"), 10, 64)
if err != nil {
respond(w, http.StatusBadRequest, errResp("invalid incident id"))
return 0, false
}
if !incidentExists(w, r, db, id) {
return 0, false
}
return id, true
}
// incidentExists reports whether the incident is one the caller may see at all.
func incidentExists(w http.ResponseWriter, r *http.Request, db *sql.DB, id int64) bool {
var exists int
if err := db.QueryRowContext(r.Context(),
"SELECT 1 FROM incidents WHERE id = $1", id).Scan(&exists); err != nil {
"SELECT 1 FROM incidents WHERE id = $1 AND team_id = ANY($2)",
id, callerTeamIDs(r.Context())).Scan(&exists); err != nil {
respond(w, http.StatusNotFound, errResp("incident not found"))
return false
}
+2 -2
View File
@@ -425,7 +425,7 @@ func TestIncident_AutoAssignedToCurrentOnCall(t *testing.T) {
s := newTS(t)
today := time.Now().UTC().Format("2006-01-02")
resp := s.req(t, http.MethodPost, "/api/schedule",
resp := s.req(t, http.MethodPost, "/api/teams/"+defaultTeam+"/schedule",
map[string]any{"user_id": 1, "dates": []string{today}})
if resp.StatusCode != http.StatusCreated {
t.Fatalf("schedule assignment returned %d", resp.StatusCode)
@@ -609,7 +609,7 @@ func TestSweeper_ArchivesResolvedIncidents(t *testing.T) {
s.exec(t, "UPDATE incidents SET resolved_at = $1 WHERE id = 1",
time.Now().Add(-30*24*time.Hour).Unix())
api.Sweep(context.Background(), s.db, 7*24*time.Hour, 6*time.Hour, s.deadman, s.notify)
api.Sweep(context.Background(), s.db, 7*24*time.Hour, 6*time.Hour, s.notify)
if inc := getIncident(t, s, 1); inc["archived_at"] == nil {
t.Error("expected the sweeper to archive a long-resolved incident")
+135 -3
View File
@@ -17,6 +17,7 @@ type contextKey string
const (
ctxUser contextKey = "user"
ctxSession contextKey = "session"
ctxTeams contextKey = "teams"
)
// AuthMiddleware accepts either of the two credentials the server issues: an
@@ -66,6 +67,39 @@ func AuthMiddleware(db *sql.DB) func(http.Handler) http.Handler {
}
}
// AdminOnly rejects a caller who is not a system administrator. It runs inside
// AuthMiddleware's group, so by the time it sees a request the caller is known.
//
// 403 and not 404: the route exists and the caller is authenticated, they are
// simply not allowed. Hiding the endpoint would buy nothing — every one of them
// is in the README.
func AdminOnly(next http.Handler) http.Handler {
return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
caller, ok := userFromContext(r.Context())
if !ok || !caller.IsAdmin {
respond(w, http.StatusForbidden, errResp("administrator access required"))
return
}
next.ServeHTTP(w, r)
})
}
// requireSelfOrAdmin guards the endpoints that are self-service for your own
// account and administration for anybody else's: your password, your ntfy
// topic, your API keys. Reports whether the request may proceed, and answers it
// if not.
//
// An API key is not an escalation: it carries exactly the rights of the user it
// belongs to, so minting your own is no more than signing in again.
func requireSelfOrAdmin(w http.ResponseWriter, r *http.Request, targetID int64) bool {
caller, ok := userFromContext(r.Context())
if !ok || (caller.ID != targetID && !caller.IsAdmin) {
respond(w, http.StatusForbidden, errResp("administrator access required"))
return false
}
return true
}
// apiKeyUser resolves an API key to its user and stamps its last use.
func apiKeyUser(ctx context.Context, db *sql.DB, token string) (int64, bool) {
var keyID, userID int64
@@ -110,15 +144,29 @@ func sessionUser(ctx context.Context, db *sql.DB, token string) (sessionID, user
func serveAs(w http.ResponseWriter, r *http.Request, next http.Handler, db *sql.DB, userID, sessionID int64) {
var u models.User
var createdUnix int64
// disabled_at IS NULL is part of the lookup rather than a check afterwards:
// a disabled account is one that cannot authenticate, by either credential,
// and the way to be sure of that is for there to be no path where the row
// is loaded and the flag is then forgotten.
if err := db.QueryRowContext(r.Context(),
"SELECT id, username, email, created_at FROM users WHERE id = $1", userID,
).Scan(&u.ID, &u.Username, &u.Email, &createdUnix); err != nil {
"SELECT id, username, email, created_at, is_admin FROM users WHERE id = $1 AND disabled_at IS NULL", userID,
).Scan(&u.ID, &u.Username, &u.Email, &createdUnix, &u.IsAdmin); err != nil {
respond(w, http.StatusUnauthorized, errResp("unauthorized"))
return
}
u.CreatedAt = time.Unix(createdUnix, 0).UTC()
ctx := context.WithValue(r.Context(), ctxUser, u)
// Every scoped query needs the caller's teams, so they are loaded once here
// rather than per handler. One extra round trip per request, against a
// table with one row per membership.
teams, err := callerMemberships(r.Context(), db, userID)
if err != nil {
respond(w, http.StatusInternalServerError, errResp("internal error"))
return
}
ctx := context.WithValue(r.Context(), ctxTeams, teams)
ctx = context.WithValue(ctx, ctxUser, u)
if sessionID != 0 {
ctx = context.WithValue(ctx, ctxSession, sessionID)
}
@@ -135,6 +183,90 @@ func userFromContext(ctx context.Context) (models.User, bool) {
return u, ok
}
// membership is the caller's role in one team.
type membership struct {
teamID int64
role string
}
func callerMemberships(ctx context.Context, db *sql.DB, userID int64) ([]membership, error) {
rows, err := db.QueryContext(ctx,
"SELECT team_id, role FROM team_members WHERE user_id = $1 ORDER BY team_id", userID)
if err != nil {
return nil, err
}
defer rows.Close()
var out []membership
for rows.Next() {
var m membership
if err := rows.Scan(&m.teamID, &m.role); err != nil {
return nil, err
}
out = append(out, m)
}
return out, rows.Err()
}
// callerTeamIDs lists the teams the caller belongs to, for the `team_id = ANY`
// filter every list query carries. An admin is NOT implicitly in every team:
// administration is about accounts, not about reading other people's incidents,
// and an admin who needs to see a team's queue can add themselves to it.
func callerTeamIDs(ctx context.Context) []int64 {
ms, _ := ctx.Value(ctxTeams).([]membership)
ids := make([]int64, 0, len(ms))
for _, m := range ms {
ids = append(ids, m.teamID)
}
return ids
}
// callerRole reports the caller's role in one team, and whether they are in it
// at all.
func callerRole(ctx context.Context, teamID int64) (string, bool) {
ms, _ := ctx.Value(ctxTeams).([]membership)
for _, m := range ms {
if m.teamID == teamID {
return m.role, true
}
}
return "", false
}
// requireTeamMember answers the request and reports false unless the caller
// belongs to teamID.
//
// 404, not 403: whether a team exists is itself something only its members
// should learn, and the same reasoning applies to every incident and alert
// under it.
func requireTeamMember(w http.ResponseWriter, r *http.Request, teamID int64) bool {
if _, ok := callerRole(r.Context(), teamID); !ok {
respond(w, http.StatusNotFound, errResp("not found"))
return false
}
return true
}
// requireTeamOwner is requireTeamMember for the things only an owner may change:
// the schedule, the integrations and who is in the team. A system administrator
// passes without being a member, because somebody has to be able to repair a
// team whose owner has left.
func requireTeamOwner(w http.ResponseWriter, r *http.Request, teamID int64) bool {
role, ok := callerRole(r.Context(), teamID)
if ok && role == models.RoleOwner {
return true
}
if caller, _ := userFromContext(r.Context()); caller.IsAdmin {
return true
}
if !ok {
respond(w, http.StatusNotFound, errResp("not found"))
return false
}
respond(w, http.StatusForbidden, errResp("team owner access required"))
return false
}
// sessionFromContext returns the id of the session a request was authenticated
// with, or false for an API-key request.
func sessionFromContext(ctx context.Context) (int64, bool) {
+19 -3
View File
@@ -43,6 +43,10 @@ const (
notifyTriggered = "triggered"
notifyReminder = "reminder"
notifyResolved = "resolved"
// notifyEscalated is a page that went out because nobody answered the last
// one. Told apart from a reminder because it goes to somebody else.
notifyEscalated = "escalated"
)
// Timeline event types the notifier writes, so an incident's history says who
@@ -120,6 +124,9 @@ func StartNotifier(ctx context.Context, db *sql.DB, cfg NotifyConfig) {
// Exported so tests can drive a pass without waiting on the ticker.
func NotifySweep(ctx context.Context, db *sql.DB, cfg NotifyConfig) {
enqueueReminders(ctx, db, cfg)
// Escalation before delivery, so a level that comes due on this tick is
// paged on this tick rather than waiting for the next one.
escalate(ctx, db, cfg)
deliverPending(ctx, db, cfg)
}
@@ -134,7 +141,11 @@ func NotifySweep(ctx context.Context, db *sql.DB, cfg NotifyConfig) {
// queued, so an ntfy outage produces a retry backlog rather than a reminder
// backlog that all lands at once when it comes back.
func enqueueReminders(ctx context.Context, db *sql.DB, cfg NotifyConfig) {
if cfg.RepeatEvery <= 0 {
// cfg.RepeatEvery is what the server started with; the settings table is
// what it runs on. Read per tick, so an administrator lengthening the
// interval at 02:00 is obeyed at 02:00 and not at the next restart.
repeat := NewSettings(db).Duration(ctx, SettingNotifyRepeat, cfg.RepeatEvery)
if repeat <= 0 {
return
}
now := time.Now()
@@ -155,8 +166,13 @@ func enqueueReminders(ctx context.Context, db *sql.DB, cfg NotifyConfig) {
AND i.resolved_at IS NULL
AND i.archived_at IS NULL
AND i.status = 'triggered'
AND (i.snoozed_until IS NULL OR i.snoozed_until <= $2)`,
now.Add(-cfg.RepeatEvery).Unix(), now.Unix())
AND (i.snoozed_until IS NULL OR i.snoozed_until <= $2)
-- A team with an escalation ladder gets escalation instead. Both
-- would mean two pages for one silence, which is how people learn to
-- mute a tool.
AND NOT EXISTS (
SELECT 1 FROM escalation_levels el WHERE el.team_id = i.team_id)`,
now.Add(-repeat).Unix(), now.Unix())
if err != nil {
log.Printf("notifier: find reminders: %v", err)
return
+23 -2
View File
@@ -69,6 +69,27 @@ func (f *fakeNtfy) messages() []pushed {
return append([]pushed(nil), f.got...)
}
// topicsSince lists the topics published to since the last forget, which is how
// the escalation tests ask "who did this tick wake".
func (f *fakeNtfy) topicsSince(t *testing.T) []string {
t.Helper()
f.mu.Lock()
defer f.mu.Unlock()
out := make([]string, 0, len(f.got))
for _, m := range f.got {
out = append(out, m.Topic)
}
return out
}
// forget drops what has been published so far, so the next assertion is about
// this tick rather than the whole test.
func (f *fakeNtfy) forget() {
f.mu.Lock()
defer f.mu.Unlock()
f.got = nil
}
func (f *fakeNtfy) failWith(status int) {
f.mu.Lock()
defer f.mu.Unlock()
@@ -95,7 +116,7 @@ func notifyTS(t *testing.T, cfg api.NotifyConfig) (*ts, *fakeNtfy) {
func putOnCall(t *testing.T, s *ts, userID int) {
t.Helper()
today := time.Now().UTC().Format("2006-01-02")
resp := s.req(t, http.MethodPost, "/api/schedule",
resp := s.req(t, http.MethodPost, "/api/teams/"+defaultTeam+"/schedule",
map[string]any{"user_id": userID, "dates": []string{today}})
defer resp.Body.Close()
if resp.StatusCode != http.StatusCreated {
@@ -438,7 +459,7 @@ func TestNotify_SweepPurgesExpiredAckTokens(t *testing.T) {
s.sweepNotify(t)
s.exec(t, "UPDATE incident_ack_tokens SET expires_at = $1", time.Now().Add(-time.Minute).Unix())
api.Sweep(context.Background(), s.db, 168*time.Hour, 6*time.Hour, s.deadman, s.notify)
api.Sweep(context.Background(), s.db, 168*time.Hour, 6*time.Hour, s.notify)
var n int
if err := s.db.QueryRow("SELECT COUNT(*) FROM incident_ack_tokens").Scan(&n); err != nil {
+70 -12
View File
@@ -4,16 +4,17 @@ import (
"database/sql"
"net/http"
"git.ryuvia.com/niklas/terdut-server/internal/config"
"git.ryuvia.com/niklas/terdut-server/internal/web"
"github.com/go-chi/chi/v5"
"github.com/go-chi/chi/v5/middleware"
)
// NewRouter builds the HTTP surface. notify and deadman are passed through to
// the webhook, the only handler that has to decide where a new incident's page
// goes and which arriving alerts are heartbeats rather than problems. A zero
// notify disables notifications; a zero deadman disables dead man's switches.
func NewRouter(db *sql.DB, notify NotifyConfig, deadman DeadmanConfig) http.Handler {
// NewRouter builds the HTTP surface. notify is passed through to the webhook,
// the only handler that has to decide where a new incident's page goes; a zero
// notify disables notifications. Dead man's switches are per team and read from
// the database, so nothing about them is wired in here.
func NewRouter(db *sql.DB, notify NotifyConfig, cfg config.Config) http.Handler {
r := chi.NewRouter()
r.Use(middleware.Logger)
r.Use(middleware.Recoverer)
@@ -27,9 +28,17 @@ func NewRouter(db *sql.DB, notify NotifyConfig, deadman DeadmanConfig) http.Hand
// the scoped token in its path rather than an API key, and has to stay
// reachable from outside the cluster for the button to work.
r.Post("/api/bootstrap", handleBootstrap(db))
r.Post("/api/alertmanager/webhook", handleAlertmanagerWebhook(db, notify, deadman))
r.Post("/api/notify/ack/{token}", handleNotifyAck(db))
// Alert ingestion. The key in the path says both that the sender may post
// and which team the alerts belong to, which is why it needs no session.
//
// This is the only way in. The pre-teams /api/alertmanager/webhook, which
// took no credential at all, was removed in v0.13.0 once the cluster's
// Alertmanager had moved onto a key; a sender still posting there gets the
// JSON 404 every unknown /api path gets.
r.Post("/api/integrations/{key}/alertmanager", handleIntegrationWebhook(db, notify))
// Signing in to the web UI. Login trades a password for a session cookie,
// which AuthMiddleware accepts in place of an API key.
r.Post("/api/login", handleLogin(db, newLoginLimiter(), notify.PublicURL))
@@ -40,14 +49,37 @@ func NewRouter(db *sql.DB, notify NotifyConfig, deadman DeadmanConfig) http.Hand
r.Use(AuthMiddleware(db))
r.Get("/api/me", handleMe(db))
// Readable by anyone signed in: the queue's assignment control and the
// on-call schedule both need to name people.
r.Get("/api/users", handleListUsers(db))
r.Post("/api/users", handleCreateUser(db))
r.Delete("/api/users/{id}", handleDeleteUser(db))
// Your own account, or anybody's if you are an admin. The handlers call
// requireSelfOrAdmin rather than sitting behind AdminOnly, because
// which rule applies depends on the {id} in the path.
r.Put("/api/users/{id}/notify", handleSetNotifyTarget(db))
r.Put("/api/users/{id}/password", handleSetPassword(db))
r.Post("/api/users/{id}/api-keys", handleCreateAPIKey(db))
r.Delete("/api/users/{id}/api-keys/{keyID}", handleDeleteAPIKey(db))
// Administration: who exists, and who is an administrator. Until #3
// these were open to any authenticated caller, which meant every user
// could delete every other one.
r.Group(func(r chi.Router) {
r.Use(AdminOnly)
r.Post("/api/users", handleCreateUser(db))
r.Delete("/api/users/{id}", handleDeleteUser(db))
r.Put("/api/users/{id}/admin", handleSetAdmin(db))
r.Put("/api/users/{id}/disabled", handleSetUserDisabled(db))
// What exists on this server, and how it behaves. /api/teams
// answers "what am I in"; this one answers "what is there".
r.Get("/api/admin/teams", handleAdminListTeams(db))
r.Get("/api/admin/settings", handleGetSettings(db, cfg))
r.Put("/api/admin/settings", handleSetSettings(db))
})
// Alerts are read-only: they are Alertmanager's record, not a work
// queue. Everything a person does happens on the incident instead.
r.Get("/api/alerts", handleListAlerts(db))
@@ -68,10 +100,36 @@ func NewRouter(db *sql.DB, notify NotifyConfig, deadman DeadmanConfig) http.Hand
r.Post("/api/incidents/{id}/notes", handleCreateNote(db))
r.Delete("/api/incidents/{id}/notes/{eventID}", handleDeleteNote(db))
r.Post("/api/schedule", handleCreateSchedule(db))
r.Get("/api/schedule/current", handleCurrentSchedule(db)) // must be before /{id}
r.Get("/api/schedule", handleListSchedule(db))
r.Delete("/api/schedule/{id}", handleDeleteSchedule(db))
// Teams. A user sees the teams they belong to; an owner configures one.
r.Get("/api/teams", handleListTeams(db))
r.Post("/api/teams", handleCreateTeam(db))
r.Put("/api/teams/{teamID}", handleRenameTeam(db))
r.Delete("/api/teams/{teamID}", handleDeleteTeam(db))
r.Get("/api/teams/{teamID}/members", handleListTeamMembers(db))
r.Post("/api/teams/{teamID}/members", handleAddTeamMember(db))
r.Delete("/api/teams/{teamID}/members/{userID}", handleRemoveTeamMember(db))
// A team's escalation ladder: who is paged when nobody answers.
r.Get("/api/teams/{teamID}/escalation", handleGetEscalation(db))
r.Put("/api/teams/{teamID}/escalation", handleSetEscalation(db))
// A team's own dead man's switches: which of its alerts are heartbeats,
// and how long a silence has to last before somebody is paged.
r.Get("/api/teams/{teamID}/deadman", handleGetTeamDeadman(db))
r.Put("/api/teams/{teamID}/deadman", handleSetTeamDeadman(db))
// Integrations: where a team's alerts come in, and the key that says so.
r.Get("/api/teams/{teamID}/integrations", handleListIntegrations(db))
r.Post("/api/teams/{teamID}/integrations", handleCreateIntegration(db, notify.PublicURL))
r.Delete("/api/teams/{teamID}/integrations/{integrationID}", handleDeleteIntegration(db))
// The rota is per team. /api/schedule/current is the exception: it
// answers across every team the caller is in, which is what somebody on
// two rotas wants to see.
r.Get("/api/schedule/current", handleCurrentSchedule(db))
r.Post("/api/teams/{teamID}/schedule", handleCreateSchedule(db))
r.Get("/api/teams/{teamID}/schedule", handleListSchedule(db))
r.Delete("/api/teams/{teamID}/schedule/{id}", handleDeleteSchedule(db))
r.Get("/api/stats/incidents", handleStatsIncidents(db))
r.Get("/api/stats/alerts", handleStatsAlerts(db))
+70 -27
View File
@@ -12,8 +12,18 @@ import (
"github.com/go-chi/chi/v5"
)
// The schedule is per team: each team keeps its own rota, so two teams can have
// two different people on call on the same day. Editing it is an owner's job,
// like the rest of a team's configuration; reading it is any member's.
func handleCreateSchedule(db *sql.DB) http.HandlerFunc {
return func(w http.ResponseWriter, r *http.Request) {
teamID, ok := teamParam(w, r)
if !ok {
return
}
if !requireTeamOwner(w, r, teamID) {
return
}
var req struct {
UserID int64 `json:"user_id"`
Dates []string `json:"dates"`
@@ -43,10 +53,13 @@ func handleCreateSchedule(db *sql.DB) http.HandlerFunc {
}
}
// Verify the user exists.
// The person taking the shift has to be in the team: paging somebody
// who cannot open the incident is worse than paging nobody.
var exists int
if err := db.QueryRowContext(r.Context(), "SELECT 1 FROM users WHERE id = $1", req.UserID).Scan(&exists); err != nil {
respond(w, http.StatusNotFound, errResp("user not found"))
if err := db.QueryRowContext(r.Context(),
"SELECT 1 FROM team_members WHERE team_id = $1 AND user_id = $2",
teamID, req.UserID).Scan(&exists); err != nil {
respond(w, http.StatusNotFound, errResp("user is not a member of this team"))
return
}
@@ -64,13 +77,15 @@ func handleCreateSchedule(db *sql.DB) http.HandlerFunc {
for _, d := range req.Dates {
if req.Replace {
if _, err := tx.ExecContext(r.Context(),
"DELETE FROM schedule_entries WHERE date = $1", d); err != nil {
"DELETE FROM schedule_entries WHERE team_id = $1 AND date = $2",
teamID, d); err != nil {
respond(w, http.StatusInternalServerError, errResp("internal error"))
return
}
}
if _, err := tx.ExecContext(r.Context(),
"INSERT INTO schedule_entries (user_id, date) VALUES ($1, $2)", req.UserID, d); err != nil {
"INSERT INTO schedule_entries (team_id, user_id, date) VALUES ($1, $2, $3)",
teamID, req.UserID, d); err != nil {
if isUniqueViolation(err) {
respond(w, http.StatusConflict,
errResp("date already assigned: "+d+" (pass replace to take it)"))
@@ -90,7 +105,7 @@ func handleCreateSchedule(db *sql.DB) http.HandlerFunc {
for _, d := range req.Dates {
dateSet[d] = true
}
all, err := scheduleRange(r.Context(), db, req.Dates[0], req.Dates[len(req.Dates)-1])
all, err := scheduleRange(r.Context(), db, teamID, req.Dates[0], req.Dates[len(req.Dates)-1])
if err != nil {
respond(w, http.StatusInternalServerError, errResp("internal error"))
return
@@ -107,6 +122,13 @@ func handleCreateSchedule(db *sql.DB) http.HandlerFunc {
func handleListSchedule(db *sql.DB) http.HandlerFunc {
return func(w http.ResponseWriter, r *http.Request) {
teamID, ok := teamParam(w, r)
if !ok {
return
}
if !requireTeamMember(w, r, teamID) {
return
}
q := r.URL.Query()
from, to := q.Get("from"), q.Get("to")
@@ -123,7 +145,7 @@ func handleListSchedule(db *sql.DB) http.HandlerFunc {
}
}
entries, err := scheduleRange(r.Context(), db, from, to)
entries, err := scheduleRange(r.Context(), db, teamID, from, to)
if err != nil {
respond(w, http.StatusInternalServerError, errResp("internal error"))
return
@@ -134,12 +156,20 @@ func handleListSchedule(db *sql.DB) http.HandlerFunc {
func handleDeleteSchedule(db *sql.DB) http.HandlerFunc {
return func(w http.ResponseWriter, r *http.Request) {
teamID, ok := teamParam(w, r)
if !ok {
return
}
if !requireTeamOwner(w, r, teamID) {
return
}
id, err := strconv.ParseInt(chi.URLParam(r, "id"), 10, 64)
if err != nil {
respond(w, http.StatusBadRequest, errResp("invalid schedule id"))
return
}
res, err := db.ExecContext(r.Context(), "DELETE FROM schedule_entries WHERE id = $1", id)
res, err := db.ExecContext(r.Context(),
"DELETE FROM schedule_entries WHERE id = $1 AND team_id = $2", id, teamID)
if err != nil {
respond(w, http.StatusInternalServerError, errResp("internal error"))
return
@@ -152,35 +182,50 @@ func handleDeleteSchedule(db *sql.DB) http.HandlerFunc {
}
}
// handleCurrentSchedule answers "who is on call right now" for every team the
// caller belongs to — one entry per team, so somebody on two rotas sees both.
// A team with nobody scheduled today simply does not appear.
func handleCurrentSchedule(db *sql.DB) http.HandlerFunc {
return func(w http.ResponseWriter, r *http.Request) {
today := time.Now().UTC().Format("2006-01-02")
var e models.ScheduleEntry
var ts int64
err := db.QueryRowContext(r.Context(), `
SELECT s.id, s.user_id, u.username, s.date, s.created_at
rows, err := db.QueryContext(r.Context(), `
SELECT s.id, s.team_id, t.name, s.user_id, u.username, s.date, s.created_at
FROM schedule_entries s
JOIN users u ON u.id = s.user_id
WHERE s.date = $1`, today).Scan(&e.ID, &e.UserID, &e.Username, &e.Date, &ts)
if err == sql.ErrNoRows {
respond(w, http.StatusNotFound, errResp("no one is on call today"))
return
}
JOIN teams t ON t.id = s.team_id
WHERE s.date = $1 AND s.team_id = ANY($2)
ORDER BY t.name`, today, callerTeamIDs(r.Context()))
if err != nil {
respond(w, http.StatusInternalServerError, errResp("internal error"))
return
}
e.CreatedAt = time.Unix(ts, 0).UTC()
respond(w, http.StatusOK, e)
defer rows.Close()
entries := []models.ScheduleEntry{}
for rows.Next() {
var e models.ScheduleEntry
var ts int64
if err := rows.Scan(&e.ID, &e.TeamID, &e.TeamName, &e.UserID, &e.Username, &e.Date, &ts); err != nil {
respond(w, http.StatusInternalServerError, errResp("internal error"))
return
}
e.CreatedAt = time.Unix(ts, 0).UTC()
entries = append(entries, e)
}
if err := rows.Err(); err != nil {
respond(w, http.StatusInternalServerError, errResp("internal error"))
return
}
respond(w, http.StatusOK, entries)
}
}
// scheduleRange returns schedule entries ordered by date.
// from and to are YYYY-MM-DD strings; an empty string means unbounded on that side.
func scheduleRange(ctx context.Context, db *sql.DB, from, to string) ([]models.ScheduleEntry, error) {
where := []string{}
func scheduleRange(ctx context.Context, db *sql.DB, teamID int64, from, to string) ([]models.ScheduleEntry, error) {
args := &sqlArgs{}
where := []string{"s.team_id = " + args.add(teamID)}
if from != "" {
where = append(where, "s.date >= "+args.add(from))
}
@@ -188,15 +233,13 @@ func scheduleRange(ctx context.Context, db *sql.DB, from, to string) ([]models.S
where = append(where, "s.date <= "+args.add(to))
}
clause := "1=1"
if len(where) > 0 {
clause = strings.Join(where, " AND ")
}
clause := strings.Join(where, " AND ")
rows, err := db.QueryContext(ctx, `
SELECT s.id, s.user_id, u.username, s.date, s.created_at
SELECT s.id, s.team_id, t.name, s.user_id, u.username, s.date, s.created_at
FROM schedule_entries s
JOIN users u ON u.id = s.user_id
JOIN teams t ON t.id = s.team_id
WHERE `+clause+`
ORDER BY s.date ASC`, args.all()...)
if err != nil {
@@ -208,7 +251,7 @@ func scheduleRange(ctx context.Context, db *sql.DB, from, to string) ([]models.S
for rows.Next() {
var e models.ScheduleEntry
var ts int64
if err := rows.Scan(&e.ID, &e.UserID, &e.Username, &e.Date, &ts); err != nil {
if err := rows.Scan(&e.ID, &e.TeamID, &e.TeamName, &e.UserID, &e.Username, &e.Date, &ts); err != nil {
return nil, err
}
e.CreatedAt = time.Unix(ts, 0).UTC()
+346
View File
@@ -0,0 +1,346 @@
package api
import (
"context"
"database/sql"
"errors"
"net/http"
"strconv"
"time"
"git.ryuvia.com/niklas/terdut-server/internal/config"
"github.com/go-chi/chi/v5"
)
// The settings an administrator can change at runtime. Each is behaviour rather
// than infrastructure: what the server does, not where it is plugged in.
//
// The values are seconds, stored as text. A duration string would be friendlier
// to read in psql and worse everywhere else — it can be stored unparseable, and
// then the question is what a background loop should do at 02:00 with a
// tuning knob it cannot understand.
const (
SettingNotifyRepeat = "notify_repeat_seconds"
SettingStaleAfter = "stale_after_seconds"
SettingArchiveAfter = "archive_after_seconds"
)
// settingBounds keeps an edit from producing a server that cannot work. The
// ceilings are loose — they exist to catch a slipped decimal point, not to have
// an opinion about anybody's rota.
var settingBounds = map[string]struct {
min, max time.Duration
label string
}{
SettingNotifyRepeat: {0, 24 * time.Hour, "how long an incident may sit unacknowledged before it is paged again; 0 disables reminders"},
SettingStaleAfter: {5 * time.Minute, 30 * 24 * time.Hour, "how long a firing alert may go without a refreshing webhook before the sweeper resolves it"},
SettingArchiveAfter: {time.Minute, 365 * 24 * time.Hour, "how long a resolved alert or incident stays in the default list"},
}
// Settings reads the runtime configuration. It holds no cache: the readers are
// two background loops that tick every 30 seconds and 15 minutes, and handlers
// that run once per request, so a query each time costs nothing measurable and
// means an administrator's change takes effect on the next tick rather than at
// the next restart.
type Settings struct{ db *sql.DB }
// NewSettings returns a reader over db.
func NewSettings(db *sql.DB) *Settings { return &Settings{db: db} }
// Duration reads one setting, falling back to def when the row is missing or
// unreadable. A tuning knob is never worth failing a sweep over: the fallback
// is the value the server started with.
func (s *Settings) Duration(ctx context.Context, key string, def time.Duration) time.Duration {
var raw string
err := s.db.QueryRowContext(ctx, "SELECT value FROM settings WHERE key = $1", key).Scan(&raw)
if err != nil {
return def
}
secs, err := strconv.ParseInt(raw, 10, 64)
if err != nil {
return def
}
return time.Duration(secs) * time.Second
}
// SeedSettings writes each key from the server's environment configuration,
// once. Never overwrites: after the first start the database owns these, and a
// redeploy must not put a chart's default back over an administrator's edit —
// the same rule as the per-team dead man's switches.
func SeedSettings(ctx context.Context, db *sql.DB, cfg config.Config) error {
seeds := map[string]time.Duration{
SettingNotifyRepeat: cfg.NotifyRepeat,
SettingStaleAfter: cfg.StaleAfter,
SettingArchiveAfter: cfg.ArchiveAfter,
}
for key, d := range seeds {
if _, err := db.ExecContext(ctx, `
INSERT INTO settings (key, value) VALUES ($1, $2)
ON CONFLICT (key) DO NOTHING`,
key, strconv.FormatInt(int64(d.Seconds()), 10)); err != nil {
return err
}
}
return nil
}
// settingsResponse is what the admin page renders. The environment half is
// included and marked read-only, so somebody looking for the ntfy URL finds out
// where it lives rather than concluding the server does not have one.
type settingsResponse struct {
Editable map[string]settingValue `json:"editable"`
FromEnv map[string]string `json:"from_env"`
}
type settingValue struct {
Seconds int64 `json:"seconds"`
Description string `json:"description"`
MinSeconds int64 `json:"min_seconds"`
MaxSeconds int64 `json:"max_seconds"`
}
func handleGetSettings(db *sql.DB, cfg config.Config) http.HandlerFunc {
settings := NewSettings(db)
return func(w http.ResponseWriter, r *http.Request) {
out := settingsResponse{
Editable: map[string]settingValue{},
FromEnv: map[string]string{
// Never the ntfy token or the DSN: both are credentials, and an
// admin page that renders them turns a browser tab into a place
// they leak from.
"ntfy_url": cfg.NtfyURL,
"ntfy_configured": strconv.FormatBool(cfg.NtfyURL != ""),
"ntfy_token_set": strconv.FormatBool(cfg.NtfyToken != ""),
"public_url": cfg.PublicURL,
"listen_address": cfg.Addr,
},
}
for key, b := range settingBounds {
def := map[string]time.Duration{
SettingNotifyRepeat: cfg.NotifyRepeat,
SettingStaleAfter: cfg.StaleAfter,
SettingArchiveAfter: cfg.ArchiveAfter,
}[key]
out.Editable[key] = settingValue{
Seconds: int64(settings.Duration(r.Context(), key, def).Seconds()),
Description: b.label,
MinSeconds: int64(b.min.Seconds()),
MaxSeconds: int64(b.max.Seconds()),
}
}
respond(w, http.StatusOK, out)
}
}
// handleSetSettings changes one or more settings. Unknown keys are refused
// rather than stored: a typo that writes notify_repeat_second would otherwise
// sit in the table looking like configuration and doing nothing.
func handleSetSettings(db *sql.DB) http.HandlerFunc {
return func(w http.ResponseWriter, r *http.Request) {
var req map[string]int64
if err := decodeJSON(r, &req); err != nil {
respond(w, http.StatusBadRequest, errResp("invalid request body"))
return
}
if len(req) == 0 {
respond(w, http.StatusBadRequest, errResp("no settings given"))
return
}
for key, secs := range req {
b, known := settingBounds[key]
if !known {
respond(w, http.StatusBadRequest, errResp("unknown setting: "+key))
return
}
d := time.Duration(secs) * time.Second
if d < b.min || d > b.max {
respond(w, http.StatusBadRequest, errResp(
key+" must be between "+b.min.String()+" and "+b.max.String()))
return
}
}
tx, err := db.BeginTx(r.Context(), nil)
if err != nil {
respond(w, http.StatusInternalServerError, errResp("internal error"))
return
}
defer tx.Rollback() //nolint:errcheck
for key, secs := range req {
if _, err := tx.ExecContext(r.Context(), `
INSERT INTO settings (key, value, updated_at)
VALUES ($1, $2, `+nowEpoch+`)
ON CONFLICT (key) DO UPDATE SET
value = excluded.value, updated_at = excluded.updated_at`,
key, strconv.FormatInt(secs, 10)); err != nil {
respond(w, http.StatusInternalServerError, errResp("internal error"))
return
}
}
if err := tx.Commit(); err != nil {
respond(w, http.StatusInternalServerError, errResp("internal error"))
return
}
w.WriteHeader(http.StatusNoContent)
}
}
// handleAdminListTeams lists every team on the server, with its size. The
// ordinary /api/teams answers "what am I in"; this one answers "what exists",
// which only an administrator may ask.
func handleAdminListTeams(db *sql.DB) http.HandlerFunc {
return func(w http.ResponseWriter, r *http.Request) {
rows, err := db.QueryContext(r.Context(), `
SELECT t.id, t.name, t.created_at,
(SELECT COUNT(*) FROM team_members m WHERE m.team_id = t.id),
(SELECT COUNT(*) FROM incidents i
WHERE i.team_id = t.id AND i.resolved_at IS NULL)
FROM teams t
ORDER BY t.name`)
if err != nil {
respond(w, http.StatusInternalServerError, errResp("internal error"))
return
}
defer rows.Close()
type adminTeam struct {
ID int64 `json:"id"`
Name string `json:"name"`
CreatedAt time.Time `json:"created_at"`
Members int64 `json:"members"`
OpenIncidents int64 `json:"open_incidents"`
}
teams := []adminTeam{}
for rows.Next() {
var t adminTeam
var created int64
if err := rows.Scan(&t.ID, &t.Name, &created, &t.Members, &t.OpenIncidents); err != nil {
respond(w, http.StatusInternalServerError, errResp("internal error"))
return
}
t.CreatedAt = time.Unix(created, 0).UTC()
teams = append(teams, t)
}
if err := rows.Err(); err != nil {
respond(w, http.StatusInternalServerError, errResp("internal error"))
return
}
respond(w, http.StatusOK, teams)
}
}
// handleRenameTeam renames a team. An owner's job, and an administrator's when
// a team has nobody left to do it.
func handleRenameTeam(db *sql.DB) http.HandlerFunc {
return func(w http.ResponseWriter, r *http.Request) {
teamID, ok := teamParam(w, r)
if !ok {
return
}
if !requireTeamOwner(w, r, teamID) {
return
}
var req struct {
Name string `json:"name"`
}
if err := decodeJSON(r, &req); err != nil || req.Name == "" {
respond(w, http.StatusBadRequest, errResp("name is required"))
return
}
res, err := db.ExecContext(r.Context(),
"UPDATE teams SET name = $1 WHERE id = $2", req.Name, teamID)
if err != nil {
if isUniqueViolation(err) {
respond(w, http.StatusConflict, errResp("a team with that name already exists"))
return
}
respond(w, http.StatusInternalServerError, errResp("internal error"))
return
}
if n, _ := res.RowsAffected(); n == 0 {
respond(w, http.StatusNotFound, errResp("not found"))
return
}
w.WriteHeader(http.StatusNoContent)
}
}
// handleSetUserDisabled takes an account out of use, or puts it back.
//
// Not a delete: the person's acknowledgements, assignments and timeline entries
// stay attached to them. Deleting a user nulls those columns, which rewrites
// what happened during an incident months after the fact.
func handleSetUserDisabled(db *sql.DB) http.HandlerFunc {
return func(w http.ResponseWriter, r *http.Request) {
id, err := strconv.ParseInt(chi.URLParam(r, "id"), 10, 64)
if err != nil {
respond(w, http.StatusBadRequest, errResp("invalid user id"))
return
}
var req struct {
Disabled *bool `json:"disabled"`
}
if err := decodeJSON(r, &req); err != nil || req.Disabled == nil {
respond(w, http.StatusBadRequest, errResp("disabled is required"))
return
}
if *req.Disabled {
caller, _ := userFromContext(r.Context())
if caller.ID == id {
respond(w, http.StatusConflict, errResp("cannot disable your own account"))
return
}
last, err := isLastAdmin(r.Context(), db, id)
if err != nil {
respond(w, http.StatusInternalServerError, errResp("internal error"))
return
}
if last {
respond(w, http.StatusConflict, errResp("cannot disable the last administrator"))
return
}
}
var res sql.Result
if *req.Disabled {
res, err = db.ExecContext(r.Context(),
"UPDATE users SET disabled_at = "+nowEpoch+" WHERE id = $1 AND disabled_at IS NULL", id)
} else {
res, err = db.ExecContext(r.Context(),
"UPDATE users SET disabled_at = NULL WHERE id = $1", id)
}
if err != nil {
respond(w, http.StatusInternalServerError, errResp("internal error"))
return
}
if n, _ := res.RowsAffected(); n == 0 {
// Either no such user, or already in the state asked for. The
// second is not a failure, so check which before answering.
var exists int
if err := db.QueryRowContext(r.Context(),
"SELECT 1 FROM users WHERE id = $1", id).Scan(&exists); errors.Is(err, sql.ErrNoRows) {
respond(w, http.StatusNotFound, errResp("user not found"))
return
}
}
// Signing back in is the only way to use a re-enabled account, and a
// disabled one must not keep a live session.
if *req.Disabled {
db.ExecContext(r.Context(), "DELETE FROM sessions WHERE user_id = $1", id) //nolint:errcheck
}
user, err := fetchUser(r.Context(), db, id)
if err != nil {
respond(w, http.StatusInternalServerError, errResp("internal error"))
return
}
respond(w, http.StatusOK, user)
}
}
+275
View File
@@ -0,0 +1,275 @@
package api_test
import (
"net/http"
"testing"
"time"
"git.ryuvia.com/niklas/terdut-server/internal/api"
)
// The settings an administrator can change, and the ones they cannot.
func TestSettings_EditableAndReadOnly(t *testing.T) {
s := newTS(t)
var got struct {
Editable map[string]struct {
Seconds int64 `json:"seconds"`
Description string `json:"description"`
MinSeconds int64 `json:"min_seconds"`
MaxSeconds int64 `json:"max_seconds"`
} `json:"editable"`
FromEnv map[string]string `json:"from_env"`
}
decode(t, s.req(t, http.MethodGet, "/api/admin/settings", nil), &got)
// Seeded from the environment the server started with, not from zero.
if v := got.Editable["notify_repeat_seconds"].Seconds; v != 900 {
t.Errorf("notify_repeat_seconds seeded as %d, want 900", v)
}
if v := got.Editable["stale_after_seconds"].Seconds; v != 21600 {
t.Errorf("stale_after_seconds seeded as %d, want 21600", v)
}
if got.Editable["archive_after_seconds"].Description == "" {
t.Error("a setting without a description is a number nobody can act on")
}
// The environment half is visible so somebody can see where it lives, but
// never the credentials themselves.
if _, ok := got.FromEnv["public_url"]; !ok {
t.Error("public_url should be reported as environment-configured")
}
for _, leak := range []string{"ntfy_token", "dsn", "database_dsn", "password"} {
if v, ok := got.FromEnv[leak]; ok {
t.Errorf("%s must not be in the settings response (got %q)", leak, v)
}
}
}
// Changing a setting takes effect on the next tick, without a restart. This is
// the whole point of moving them out of the environment.
func TestSettings_ChangeTakesEffectOnTheNextSweep(t *testing.T) {
s := newTS(t)
// An alert whose last webhook was two hours ago. Under the seeded
// stale_after of six hours the sweeper leaves it alone.
postWebhook(t, s, []map[string]any{
amAlert("fp-settings", "Stale", "firing", "2026-05-20T10:00:00Z", zeroTime, nil),
})
s.exec(t, "UPDATE alerts SET received_at = $1 WHERE fingerprint = $2",
time.Now().Add(-2*time.Hour).Unix(), "fp-settings")
sweep(t, s, noArchive)
if status, _, _ := s.alertRow(t, "fp-settings"); status != "firing" {
t.Fatalf("before the change the alert should still be firing, got %q", status)
}
// Shorten it to an hour. Nothing restarts.
resp := s.req(t, http.MethodPut, "/api/admin/settings",
map[string]int64{"stale_after_seconds": 3600})
resp.Body.Close()
if resp.StatusCode != http.StatusNoContent {
t.Fatalf("change setting: %d", resp.StatusCode)
}
sweep(t, s, noArchive)
status, source, _ := s.alertRow(t, "fp-settings")
if status != "resolved" {
t.Errorf("after the change the alert should have expired, got %q", status)
}
if source == nil || *source != "expiry" {
t.Errorf("expected resolution_source expiry, got %v", source)
}
}
// A typo must not look like configuration, and a slipped decimal point must not
// produce a server that sweeps every second.
func TestSettings_RejectsUnknownKeysAndSillyValues(t *testing.T) {
s := newTS(t)
for _, c := range []struct {
name string
body map[string]int64
}{
{"unknown key", map[string]int64{"notify_repeat_second": 60}},
{"below the floor", map[string]int64{"stale_after_seconds": 30}},
{"above the ceiling", map[string]int64{"archive_after_seconds": 400 * 24 * 3600}},
{"nothing at all", map[string]int64{}},
} {
resp := s.req(t, http.MethodPut, "/api/admin/settings", c.body)
resp.Body.Close()
if resp.StatusCode != http.StatusBadRequest {
t.Errorf("%s: expected 400, got %d", c.name, resp.StatusCode)
}
}
}
// Settings are the server's behaviour, so only an administrator may change
// them — or see where the rest of the configuration comes from.
func TestSettings_AreAdminOnly(t *testing.T) {
s := newTS(t)
_, call := member(t, s, "member")
for _, c := range []struct {
method string
body any
}{
{http.MethodGet, nil},
{http.MethodPut, map[string]int64{"notify_repeat_seconds": 60}},
} {
resp := call(c.method, "/api/admin/settings", c.body)
resp.Body.Close()
if resp.StatusCode != http.StatusForbidden {
t.Errorf("%s /api/admin/settings: expected 403, got %d", c.method, resp.StatusCode)
}
}
resp := call(http.MethodGet, "/api/admin/teams", nil)
resp.Body.Close()
if resp.StatusCode != http.StatusForbidden {
t.Errorf("GET /api/admin/teams: expected 403, got %d", resp.StatusCode)
}
}
// An administrator sees every team, including ones they are not in — which is
// exactly what /api/teams must not show them.
func TestSettings_AdminSeesEveryTeam(t *testing.T) {
s := newTS(t)
newTeam(t, s, "red")
newTeam(t, s, "blue")
all := list(t, s.req(t, http.MethodGet, "/api/admin/teams", nil))
if len(all) != 3 { // Default, red, blue
t.Fatalf("admin should see all 3 teams, saw %d", len(all))
}
for _, team := range all {
if _, ok := team["members"]; !ok {
t.Error("the admin listing should say how big each team is")
}
}
// The admin created them, so they own them — but they are not a member of
// a team somebody else makes, and /api/teams still answers "what am I in".
mine := list(t, s.req(t, http.MethodGet, "/api/teams", nil))
if len(mine) != 3 {
t.Errorf("the creator is an owner of what they created, saw %d", len(mine))
}
}
// Disabling is not deleting: the account stops working and the history stays.
func TestSettings_DisablingAnAccountKeepsItsHistory(t *testing.T) {
s := newTS(t)
memberID, call := member(t, s, "leaver")
// They acknowledge an incident, so there is history to preserve.
postWebhook(t, s, []map[string]any{
amAlert("fp-leaver", "DiskFull", "firing", "2026-05-20T10:00:00Z", zeroTime, nil),
})
resp := call(http.MethodPost, "/api/incidents/1/acknowledge", nil)
resp.Body.Close()
if resp.StatusCode != http.StatusOK {
t.Fatalf("acknowledge: %d", resp.StatusCode)
}
resp = s.req(t, http.MethodPut, "/api/users/"+id64(memberID)+"/disabled",
map[string]bool{"disabled": true})
resp.Body.Close()
if resp.StatusCode != http.StatusOK {
t.Fatalf("disable: %d", resp.StatusCode)
}
// Their API key stops working.
resp = call(http.MethodGet, "/api/incidents", nil)
resp.Body.Close()
if resp.StatusCode != http.StatusUnauthorized {
t.Errorf("a disabled user's key: expected 401, got %d", resp.StatusCode)
}
// The acknowledgement still names them.
var incident map[string]any
decode(t, s.req(t, http.MethodGet, "/api/incidents/1", nil), &incident)
if incident["acknowledged_by"] != "leaver" {
t.Errorf("the acknowledgement should still name leaver, got %v", incident["acknowledged_by"])
}
if incident["status"] != "acknowledged" {
t.Errorf("the incident should still be acknowledged, got %v", incident["status"])
}
// And re-enabling gives the account back.
resp = s.req(t, http.MethodPut, "/api/users/"+id64(memberID)+"/disabled",
map[string]bool{"disabled": false})
resp.Body.Close()
resp = call(http.MethodGet, "/api/incidents", nil)
resp.Body.Close()
if resp.StatusCode != http.StatusOK {
t.Errorf("after re-enabling: expected 200, got %d", resp.StatusCode)
}
}
// The same two guards as deleting and demoting: an install must keep somebody
// who can administer it.
func TestSettings_CannotDisableYourselfOrTheLastAdmin(t *testing.T) {
s := newTS(t)
resp := s.req(t, http.MethodPut, "/api/users/1/disabled", map[string]bool{"disabled": true})
resp.Body.Close()
if resp.StatusCode != http.StatusConflict {
t.Errorf("disabling yourself: expected 409, got %d", resp.StatusCode)
}
}
// Renaming a team is an owner's job, and the name stays unique.
func TestSettings_TeamRename(t *testing.T) {
s := newTS(t)
team := newTeam(t, s, "red")
resp := s.req(t, http.MethodPut, "/api/teams/"+id64(team.id), map[string]string{"name": "Platform"})
resp.Body.Close()
if resp.StatusCode != http.StatusNoContent {
t.Fatalf("rename: %d", resp.StatusCode)
}
teams := list(t, s.req(t, http.MethodGet, "/api/admin/teams", nil))
found := false
for _, x := range teams {
if x["name"] == "Platform" {
found = true
}
}
if !found {
t.Error("the renamed team should be listed under its new name")
}
// Taking a name that exists is a conflict, not a silent second team with
// the same label.
resp = s.req(t, http.MethodPut, "/api/teams/"+id64(team.id), map[string]string{"name": "Default"})
resp.Body.Close()
if resp.StatusCode != http.StatusConflict {
t.Errorf("renaming onto an existing name: expected 409, got %d", resp.StatusCode)
}
}
// The seed runs once. A redeploy must not put the chart's default back over an
// administrator's edit — the rule the dead man's switches already follow.
func TestSettings_SeedDoesNotOverwrite(t *testing.T) {
s := newTS(t)
resp := s.req(t, http.MethodPut, "/api/admin/settings",
map[string]int64{"notify_repeat_seconds": 60})
resp.Body.Close()
// A second start, with the environment still saying 15 minutes.
if err := api.SeedSettings(t.Context(), s.db, testConfig()); err != nil {
t.Fatalf("re-seed: %v", err)
}
var got struct {
Editable map[string]struct {
Seconds int64 `json:"seconds"`
} `json:"editable"`
}
decode(t, s.req(t, http.MethodGet, "/api/admin/settings", nil), &got)
if v := got.Editable["notify_repeat_seconds"].Seconds; v != 60 {
t.Errorf("the edit should survive a restart, got %d", v)
}
}
+11 -7
View File
@@ -11,7 +11,7 @@ import (
func handleStatsAlerts(db *sql.DB) http.HandlerFunc {
return func(w http.ResponseWriter, r *http.Request) {
where, args := statsFilter(r.URL.Query(), "received_at")
where, args := statsFilter(r.URL.Query(), "received_at", callerTeamIDs(r.Context()))
// COALESCE because SUM over zero rows is NULL, not 0, and a count of
// nothing is 0 — without it an empty window is a 500 rather than a
@@ -37,7 +37,7 @@ func handleStatsAlerts(db *sql.DB) http.HandlerFunc {
func handleStatsTop(db *sql.DB) http.HandlerFunc {
return func(w http.ResponseWriter, r *http.Request) {
where, args := statsFilter(r.URL.Query(), "received_at")
where, args := statsFilter(r.URL.Query(), "received_at", callerTeamIDs(r.Context()))
limit := 10
if l := r.URL.Query().Get("limit"); l != "" {
@@ -79,7 +79,7 @@ func handleStatsTop(db *sql.DB) http.HandlerFunc {
func handleStatsByHour(db *sql.DB) http.HandlerFunc {
return func(w http.ResponseWriter, r *http.Request) {
where, args := statsFilter(r.URL.Query(), "received_at")
where, args := statsFilter(r.URL.Query(), "received_at", callerTeamIDs(r.Context()))
rows, err := db.QueryContext(r.Context(), fmt.Sprintf(`
SELECT EXTRACT(HOUR FROM to_timestamp(received_at) AT TIME ZONE 'UTC')::int AS hr,
@@ -119,7 +119,7 @@ func handleStatsByHour(db *sql.DB) http.HandlerFunc {
func handleStatsByDay(db *sql.DB) http.HandlerFunc {
return func(w http.ResponseWriter, r *http.Request) {
where, args := statsFilter(r.URL.Query(), "received_at")
where, args := statsFilter(r.URL.Query(), "received_at", callerTeamIDs(r.Context()))
// Postgres EXTRACT(DOW …) → 0=Sunday … 6=Saturday, the same numbering
// SQLite's strftime('%w') returned, so the frontend needs no change.
@@ -167,7 +167,7 @@ func handleStatsByDay(db *sql.DB) http.HandlerFunc {
// mutated in place and carry no acknowledgement or closure time.
func handleStatsIncidents(db *sql.DB) http.HandlerFunc {
return func(w http.ResponseWriter, r *http.Request) {
where, args := statsFilter(r.URL.Query(), "triggered_at")
where, args := statsFilter(r.URL.Query(), "triggered_at", callerTeamIDs(r.Context()))
// The counts are COALESCEd because SUM over zero rows is NULL, not 0.
// The averages are not: mtta and mttr stay null on purpose, since zero
@@ -206,9 +206,13 @@ func handleStatsIncidents(db *sql.DB) http.HandlerFunc {
// statsFilter builds a WHERE clause and args from optional ?from and ?to query
// params, filtering on timeCol. Archived rows are always excluded, matching the
// default list views.
func statsFilter(q url.Values, timeCol string) (where string, args *sqlArgs) {
//
// teamIDs scopes every figure to the caller's own teams: a report that counted
// other teams' incidents would leak their volume and their names through the
// top-alerts list, and would not be a number about the reader's work anyway.
func statsFilter(q url.Values, timeCol string, teamIDs []int64) (where string, args *sqlArgs) {
args = &sqlArgs{}
clauses := []string{"archived_at IS NULL"}
clauses := []string{"archived_at IS NULL", "team_id = ANY(" + args.add(teamIDs) + ")"}
if from := q.Get("from"); from != "" {
if t, err := time.Parse("2006-01-02", from); err == nil {
clauses = append(clauses, timeCol+" >= "+args.add(t.UTC().Unix()))
+576
View File
@@ -0,0 +1,576 @@
package api
import (
"context"
"database/sql"
"errors"
"net/http"
"strconv"
"strings"
"time"
"git.ryuvia.com/niklas/terdut-server/internal/models"
"github.com/go-chi/chi/v5"
)
// handleListTeams lists the caller's own teams, each with their role in it. An
// administrator listing every team goes through the admin endpoint instead:
// this one answers "what am I part of", which is what the UI's team filter and
// the combined queue are built from.
func handleListTeams(db *sql.DB) http.HandlerFunc {
return func(w http.ResponseWriter, r *http.Request) {
caller, _ := userFromContext(r.Context())
rows, err := db.QueryContext(r.Context(), `
SELECT t.id, t.name, t.created_at, m.role
FROM teams t
JOIN team_members m ON m.team_id = t.id
WHERE m.user_id = $1
ORDER BY t.name`, caller.ID)
if err != nil {
respond(w, http.StatusInternalServerError, errResp("internal error"))
return
}
defer rows.Close()
teams := []models.Team{}
for rows.Next() {
var t models.Team
var created int64
if err := rows.Scan(&t.ID, &t.Name, &created, &t.Role); err != nil {
respond(w, http.StatusInternalServerError, errResp("internal error"))
return
}
t.CreatedAt = time.Unix(created, 0).UTC()
teams = append(teams, t)
}
if err := rows.Err(); err != nil {
respond(w, http.StatusInternalServerError, errResp("internal error"))
return
}
respond(w, http.StatusOK, teams)
}
}
// handleCreateTeam creates a team and makes its creator the first owner. A team
// with no owner would need an administrator to repair before anybody could use
// it, so the two happen in one transaction.
func handleCreateTeam(db *sql.DB) http.HandlerFunc {
return func(w http.ResponseWriter, r *http.Request) {
var req struct {
Name string `json:"name"`
}
if err := decodeJSON(r, &req); err != nil {
respond(w, http.StatusBadRequest, errResp("invalid request body"))
return
}
req.Name = strings.TrimSpace(req.Name)
if req.Name == "" {
respond(w, http.StatusBadRequest, errResp("name is required"))
return
}
caller, _ := userFromContext(r.Context())
tx, err := db.BeginTx(r.Context(), nil)
if err != nil {
respond(w, http.StatusInternalServerError, errResp("internal error"))
return
}
defer tx.Rollback() //nolint:errcheck
var team models.Team
var created int64
if err := tx.QueryRowContext(r.Context(),
"INSERT INTO teams (name) VALUES ($1) RETURNING id, name, created_at",
req.Name).Scan(&team.ID, &team.Name, &created); err != nil {
if isUniqueViolation(err) {
respond(w, http.StatusConflict, errResp("a team with that name already exists"))
return
}
respond(w, http.StatusInternalServerError, errResp("internal error"))
return
}
if _, err := tx.ExecContext(r.Context(),
"INSERT INTO team_members (team_id, user_id, role) VALUES ($1, $2, $3)",
team.ID, caller.ID, models.RoleOwner); err != nil {
respond(w, http.StatusInternalServerError, errResp("internal error"))
return
}
if err := tx.Commit(); err != nil {
respond(w, http.StatusInternalServerError, errResp("internal error"))
return
}
team.CreatedAt = time.Unix(created, 0).UTC()
team.Role = models.RoleOwner
respond(w, http.StatusCreated, team)
}
}
// handleDeleteTeam removes a team and, by cascade, its incidents, alerts,
// schedule and integrations.
//
// Refused while the team still has open incidents: deleting a team is tidying
// up, and tidying up should never be how an unacknowledged page disappears.
// Resolve or archive them first, deliberately.
func handleDeleteTeam(db *sql.DB) http.HandlerFunc {
return func(w http.ResponseWriter, r *http.Request) {
teamID, ok := teamParam(w, r)
if !ok {
return
}
if !requireTeamOwner(w, r, teamID) {
return
}
var open int
if err := db.QueryRowContext(r.Context(),
"SELECT COUNT(*) FROM incidents WHERE team_id = $1 AND resolved_at IS NULL", teamID).
Scan(&open); err != nil {
respond(w, http.StatusInternalServerError, errResp("internal error"))
return
}
if open > 0 {
respond(w, http.StatusConflict, errResp("team still has open incidents"))
return
}
res, err := db.ExecContext(r.Context(), "DELETE FROM teams WHERE id = $1", teamID)
if err != nil {
respond(w, http.StatusInternalServerError, errResp("internal error"))
return
}
if n, _ := res.RowsAffected(); n == 0 {
respond(w, http.StatusNotFound, errResp("not found"))
return
}
w.WriteHeader(http.StatusNoContent)
}
}
// handleListTeamMembers names everybody in a team. Visible to any member: you
// can see who else is on the rota you are on.
func handleListTeamMembers(db *sql.DB) http.HandlerFunc {
return func(w http.ResponseWriter, r *http.Request) {
teamID, ok := teamParam(w, r)
if !ok {
return
}
if !requireTeamMember(w, r, teamID) {
return
}
rows, err := db.QueryContext(r.Context(), `
SELECT m.team_id, m.user_id, u.username, m.role, m.joined_at
FROM team_members m
JOIN users u ON u.id = m.user_id
WHERE m.team_id = $1
ORDER BY u.username`, teamID)
if err != nil {
respond(w, http.StatusInternalServerError, errResp("internal error"))
return
}
defer rows.Close()
members := []models.TeamMember{}
for rows.Next() {
var m models.TeamMember
var joined int64
if err := rows.Scan(&m.TeamID, &m.UserID, &m.Username, &m.Role, &joined); err != nil {
respond(w, http.StatusInternalServerError, errResp("internal error"))
return
}
m.JoinedAt = time.Unix(joined, 0).UTC()
members = append(members, m)
}
if err := rows.Err(); err != nil {
respond(w, http.StatusInternalServerError, errResp("internal error"))
return
}
respond(w, http.StatusOK, members)
}
}
// handleAddTeamMember adds a user to a team, or changes the role of somebody
// already in it.
func handleAddTeamMember(db *sql.DB) http.HandlerFunc {
return func(w http.ResponseWriter, r *http.Request) {
teamID, ok := teamParam(w, r)
if !ok {
return
}
if !requireTeamOwner(w, r, teamID) {
return
}
var req struct {
UserID int64 `json:"user_id"`
Role string `json:"role"`
}
if err := decodeJSON(r, &req); err != nil || req.UserID == 0 {
respond(w, http.StatusBadRequest, errResp("user_id is required"))
return
}
if req.Role == "" {
req.Role = models.RoleMember
}
if req.Role != models.RoleOwner && req.Role != models.RoleMember {
respond(w, http.StatusBadRequest, errResp("role must be owner or member"))
return
}
_, err := db.ExecContext(r.Context(), `
INSERT INTO team_members (team_id, user_id, role)
VALUES ($1, $2, $3)
ON CONFLICT (team_id, user_id) DO UPDATE SET role = excluded.role`,
teamID, req.UserID, req.Role)
if err != nil {
// The only foreign key that can fail here is the user: the team was
// resolved from the caller's own membership.
respond(w, http.StatusNotFound, errResp("user not found"))
return
}
w.WriteHeader(http.StatusNoContent)
}
}
// handleRemoveTeamMember takes a user out of a team.
//
// A team must keep an owner, for the same reason the install must keep an
// administrator: otherwise nobody can configure it, and repairing that needs
// somebody with more access than the team has.
func handleRemoveTeamMember(db *sql.DB) http.HandlerFunc {
return func(w http.ResponseWriter, r *http.Request) {
teamID, ok := teamParam(w, r)
if !ok {
return
}
if !requireTeamOwner(w, r, teamID) {
return
}
userID, err := strconv.ParseInt(chi.URLParam(r, "userID"), 10, 64)
if err != nil {
respond(w, http.StatusBadRequest, errResp("invalid user id"))
return
}
last, err := isLastTeamOwner(r.Context(), db, teamID, userID)
if err != nil {
respond(w, http.StatusInternalServerError, errResp("internal error"))
return
}
if last {
respond(w, http.StatusConflict, errResp("cannot remove the last owner of a team"))
return
}
res, err := db.ExecContext(r.Context(),
"DELETE FROM team_members WHERE team_id = $1 AND user_id = $2", teamID, userID)
if err != nil {
respond(w, http.StatusInternalServerError, errResp("internal error"))
return
}
if n, _ := res.RowsAffected(); n == 0 {
respond(w, http.StatusNotFound, errResp("not a member of this team"))
return
}
w.WriteHeader(http.StatusNoContent)
}
}
func isLastTeamOwner(ctx context.Context, db *sql.DB, teamID, userID int64) (bool, error) {
var last bool
err := db.QueryRowContext(ctx, `
SELECT EXISTS (SELECT 1 FROM team_members
WHERE team_id = $1 AND user_id = $2 AND role = 'owner')
AND NOT EXISTS (SELECT 1 FROM team_members
WHERE team_id = $1 AND user_id <> $2 AND role = 'owner')`,
teamID, userID).Scan(&last)
return last, err
}
// ---------------------------------------------------------------------------
// Integrations
// ---------------------------------------------------------------------------
// handleListIntegrations lists a team's integrations. Never the keys: those
// exist in plaintext only in the response that created them.
func handleListIntegrations(db *sql.DB) http.HandlerFunc {
return func(w http.ResponseWriter, r *http.Request) {
teamID, ok := teamParam(w, r)
if !ok {
return
}
if !requireTeamMember(w, r, teamID) {
return
}
rows, err := db.QueryContext(r.Context(), `
SELECT id, team_id, kind, name, created_at, last_used_at
FROM integrations
WHERE team_id = $1
ORDER BY id`, teamID)
if err != nil {
respond(w, http.StatusInternalServerError, errResp("internal error"))
return
}
defer rows.Close()
integrations := []models.Integration{}
for rows.Next() {
var i models.Integration
var created int64
var lastUsed *int64
if err := rows.Scan(&i.ID, &i.TeamID, &i.Kind, &i.Name, &created, &lastUsed); err != nil {
respond(w, http.StatusInternalServerError, errResp("internal error"))
return
}
i.CreatedAt = time.Unix(created, 0).UTC()
i.LastUsedAt = unixPtr(lastUsed)
integrations = append(integrations, i)
}
if err := rows.Err(); err != nil {
respond(w, http.StatusInternalServerError, errResp("internal error"))
return
}
respond(w, http.StatusOK, integrations)
}
}
// handleCreateIntegration mints an integration key. The key is returned once,
// in this response, and only its hash is kept — the same handling as an API key
// or an acknowledgement token.
func handleCreateIntegration(db *sql.DB, publicURL string) http.HandlerFunc {
return func(w http.ResponseWriter, r *http.Request) {
teamID, ok := teamParam(w, r)
if !ok {
return
}
if !requireTeamOwner(w, r, teamID) {
return
}
var req struct {
Name string `json:"name"`
Kind string `json:"kind"`
}
if err := decodeJSON(r, &req); err != nil {
respond(w, http.StatusBadRequest, errResp("invalid request body"))
return
}
req.Name = strings.TrimSpace(req.Name)
if req.Name == "" {
respond(w, http.StatusBadRequest, errResp("name is required"))
return
}
if req.Kind == "" {
req.Kind = models.IntegrationAlertmanager
}
if req.Kind != models.IntegrationAlertmanager {
respond(w, http.StatusBadRequest, errResp("unsupported integration kind"))
return
}
raw, hash, err := randomToken()
if err != nil {
respond(w, http.StatusInternalServerError, errResp("internal error"))
return
}
var i models.Integration
var created int64
if err := db.QueryRowContext(r.Context(), `
INSERT INTO integrations (team_id, kind, name, key_hash)
VALUES ($1, $2, $3, $4)
RETURNING id, team_id, kind, name, created_at`,
teamID, req.Kind, req.Name, hash).
Scan(&i.ID, &i.TeamID, &i.Kind, &i.Name, &created); err != nil {
respond(w, http.StatusInternalServerError, errResp("internal error"))
return
}
i.CreatedAt = time.Unix(created, 0).UTC()
i.Key = raw
i.URL = strings.TrimSuffix(publicURL, "/") + integrationPath(raw, i.Kind)
respond(w, http.StatusCreated, i)
}
}
func handleDeleteIntegration(db *sql.DB) http.HandlerFunc {
return func(w http.ResponseWriter, r *http.Request) {
teamID, ok := teamParam(w, r)
if !ok {
return
}
if !requireTeamOwner(w, r, teamID) {
return
}
id, err := strconv.ParseInt(chi.URLParam(r, "integrationID"), 10, 64)
if err != nil {
respond(w, http.StatusBadRequest, errResp("invalid integration id"))
return
}
res, err := db.ExecContext(r.Context(),
"DELETE FROM integrations WHERE id = $1 AND team_id = $2", id, teamID)
if err != nil {
respond(w, http.StatusInternalServerError, errResp("internal error"))
return
}
if n, _ := res.RowsAffected(); n == 0 {
respond(w, http.StatusNotFound, errResp("not found"))
return
}
w.WriteHeader(http.StatusNoContent)
}
}
// integrationPath is where a sender of this kind posts. Built in one place so
// the URL handed out at creation and the route the router registers cannot
// drift apart.
func integrationPath(key, kind string) string {
return "/api/integrations/" + key + "/" + kind
}
// teamIDForKey resolves an integration key to its team, and stamps the key's
// last use. An unknown key is not an error worth distinguishing: the caller is
// told nothing beyond "no".
func teamIDForKey(ctx context.Context, db *sql.DB, key string) (int64, error) {
var teamID int64
err := db.QueryRowContext(ctx,
"SELECT team_id FROM integrations WHERE key_hash = $1", hashToken(key)).Scan(&teamID)
if errors.Is(err, sql.ErrNoRows) {
return 0, errUnknownIntegration
}
if err != nil {
return 0, err
}
// Best effort, like an API key's: a failed stamp must not reject an alert.
db.ExecContext(ctx, //nolint:errcheck
"UPDATE integrations SET last_used_at = $1 WHERE key_hash = $2",
time.Now().Unix(), hashToken(key))
return teamID, nil
}
var errUnknownIntegration = errors.New("unknown integration key")
// teamParam reads {teamID} from the path.
func teamParam(w http.ResponseWriter, r *http.Request) (int64, bool) {
id, err := strconv.ParseInt(chi.URLParam(r, "teamID"), 10, 64)
if err != nil {
respond(w, http.StatusBadRequest, errResp("invalid team id"))
return 0, false
}
return id, true
}
// defaultTeamID is the oldest team, which on an upgraded install is the
// "Default" team every pre-teams row was moved into and on a fresh one is the
// team migration 003 creates. Bootstrap puts the first user in it, so somebody
// signing in to a new server lands somewhere rather than in no team at all.
func defaultTeamID(ctx context.Context, db *sql.DB) (int64, error) {
var id int64
err := db.QueryRowContext(ctx, "SELECT id FROM teams ORDER BY id LIMIT 1").Scan(&id)
return id, err
}
// ---------------------------------------------------------------------------
// A team's dead man's switches
// ---------------------------------------------------------------------------
// deadmanResponse is the wire shape of a team's switch configuration. The
// timeout is seconds rather than a duration string, because that is what the
// column holds and what arithmetic is done on; a client renders it.
type deadmanResponse struct {
TeamID int64 `json:"team_id"`
Matchers string `json:"matchers"`
TimeoutSeconds int64 `json:"timeout_seconds"`
Severity string `json:"severity"`
}
func handleGetTeamDeadman(db *sql.DB) http.HandlerFunc {
return func(w http.ResponseWriter, r *http.Request) {
teamID, ok := teamParam(w, r)
if !ok {
return
}
if !requireTeamMember(w, r, teamID) {
return
}
out := deadmanResponse{TeamID: teamID, Severity: "critical"}
err := db.QueryRowContext(r.Context(),
"SELECT matchers, timeout_seconds, severity FROM deadman_configs WHERE team_id = $1",
teamID).Scan(&out.Matchers, &out.TimeoutSeconds, &out.Severity)
if err != nil && !errors.Is(err, sql.ErrNoRows) {
respond(w, http.StatusInternalServerError, errResp("internal error"))
return
}
// A team with no row watches nothing, which is a configuration and not
// an absence: answering 404 would make "off" indistinguishable from
// "this server does not do this".
respond(w, http.StatusOK, out)
}
}
// handleSetTeamDeadman replaces a team's switch configuration.
//
// Validated by parsing: a matcher string that survives ParseDeadmanConfig with
// nothing usable in it is rejected rather than stored, because a switch that
// silently watches nothing is the failure this feature exists to prevent.
func handleSetTeamDeadman(db *sql.DB) http.HandlerFunc {
return func(w http.ResponseWriter, r *http.Request) {
teamID, ok := teamParam(w, r)
if !ok {
return
}
if !requireTeamOwner(w, r, teamID) {
return
}
var req struct {
Matchers string `json:"matchers"`
TimeoutSeconds int64 `json:"timeout_seconds"`
Severity string `json:"severity"`
}
if err := decodeJSON(r, &req); err != nil {
respond(w, http.StatusBadRequest, errResp("invalid request body"))
return
}
req.Matchers = strings.TrimSpace(req.Matchers)
if req.Severity == "" {
req.Severity = "critical"
}
if req.TimeoutSeconds < 0 {
respond(w, http.StatusBadRequest, errResp("timeout_seconds must not be negative"))
return
}
if req.Matchers != "" {
parsed := parseDeadmanQuietly(req.Matchers, time.Duration(req.TimeoutSeconds)*time.Second, req.Severity)
if len(parsed.Matchers) == 0 {
respond(w, http.StatusBadRequest, errResp(
"no usable matchers: each must name an alertname, as in alertname=Watchdog,cluster=prod"))
return
}
}
if _, err := db.ExecContext(r.Context(), `
INSERT INTO deadman_configs (team_id, matchers, timeout_seconds, severity, updated_at)
VALUES ($1, $2, $3, $4, `+nowEpoch+`)
ON CONFLICT (team_id) DO UPDATE SET
matchers = excluded.matchers,
timeout_seconds = excluded.timeout_seconds,
severity = excluded.severity,
updated_at = excluded.updated_at`,
teamID, req.Matchers, req.TimeoutSeconds, req.Severity); err != nil {
respond(w, http.StatusInternalServerError, errResp("internal error"))
return
}
respond(w, http.StatusOK, deadmanResponse{
TeamID: teamID,
Matchers: req.Matchers,
TimeoutSeconds: req.TimeoutSeconds,
Severity: req.Severity,
})
}
}
+350
View File
@@ -0,0 +1,350 @@
package api_test
import (
"bytes"
"encoding/json"
"io"
"net/http"
"testing"
)
// The whole point of #4: two teams sharing one server must not see each other's
// work. These tests build two of them and check the boundary from both sides.
type teamFixture struct {
id int64
key string // integration key: how alerts get in
call func(method, path string, body any) *http.Response
}
// newTeam creates a team with its own member, integration key and API key. The
// admin does the creating, as an install's first user would.
func newTeam(t *testing.T, s *ts, name string) teamFixture {
t.Helper()
var team struct {
ID int64 `json:"id"`
}
decode(t, s.req(t, http.MethodPost, "/api/teams", map[string]string{"name": name}), &team)
var integration struct {
Key string `json:"key"`
URL string `json:"url"`
}
decode(t, s.req(t, http.MethodPost, "/api/teams/"+id64(team.ID)+"/integrations",
map[string]string{"name": name + " alertmanager"}), &integration)
if integration.Key == "" {
t.Fatalf("%s: integration key was not returned", name)
}
// A member of this team and no other.
var user struct {
ID int64 `json:"id"`
}
decode(t, s.req(t, http.MethodPost, "/api/users",
map[string]string{"username": name + "-user", "email": name + "@test.com"}), &user)
resp := s.req(t, http.MethodPost, "/api/teams/"+id64(team.ID)+"/members",
map[string]any{"user_id": user.ID, "role": "owner"})
resp.Body.Close()
if resp.StatusCode != http.StatusNoContent {
t.Fatalf("%s: add member: %d", name, resp.StatusCode)
}
var key struct {
Key string `json:"key"`
}
decode(t, s.req(t, http.MethodPost, "/api/users/"+id64(user.ID)+"/api-keys",
map[string]string{"name": "test"}), &key)
return teamFixture{
id: team.ID,
key: integration.Key,
call: func(method, path string, body any) *http.Response {
t.Helper()
var r io.Reader
if body != nil {
data, _ := json.Marshal(body)
r = bytes.NewReader(data)
}
req, _ := http.NewRequest(method, s.URL+path, r)
req.Header.Set("Authorization", "Bearer "+key.Key)
if body != nil {
req.Header.Set("Content-Type", "application/json")
}
resp, err := http.DefaultClient.Do(req)
if err != nil {
t.Fatalf("%s %s: %v", method, path, err)
}
return resp
},
}
}
// postToIntegration sends one firing alert on a team's integration key, the way
// a real Alertmanager receiver would.
func postToIntegration(t *testing.T, s *ts, key, fingerprint, name string) {
t.Helper()
payload := map[string]any{
"version": "4",
"status": "firing",
"groupKey": "{}:{alertname=\"" + name + "\"}",
"groupLabels": map[string]string{"alertname": name},
"alerts": []map[string]any{
amAlert(fingerprint, name, "firing", "2026-09-20T10:00:00Z", zeroTime, nil),
},
}
data, _ := json.Marshal(payload)
resp, err := http.Post(s.URL+"/api/integrations/"+key+"/alertmanager",
"application/json", bytes.NewReader(data))
if err != nil {
t.Fatalf("post alert: %v", err)
}
defer resp.Body.Close()
if resp.StatusCode != http.StatusOK {
t.Fatalf("post alert: %d", resp.StatusCode)
}
}
func list(t *testing.T, resp *http.Response) []map[string]any {
t.Helper()
var out []map[string]any
decode(t, resp, &out)
return out
}
// An alert posted on one team's key opens an incident in that team and nowhere
// else, and neither team can read the other's queue.
func TestTeams_IncidentsAreScopedToTheReceivingTeam(t *testing.T) {
s := newTS(t)
red := newTeam(t, s, "red")
blue := newTeam(t, s, "blue")
postToIntegration(t, s, red.key, "fp-red", "RedDiskFull")
postToIntegration(t, s, blue.key, "fp-blue", "BlueDiskFull")
redIncidents := list(t, red.call(http.MethodGet, "/api/incidents", nil))
if len(redIncidents) != 1 {
t.Fatalf("red should see exactly its own incident, saw %d", len(redIncidents))
}
if title := redIncidents[0]["title"]; title != "RedDiskFull" {
t.Errorf("red saw %v", title)
}
if teamID := int64(redIncidents[0]["team_id"].(float64)); teamID != red.id {
t.Errorf("red's incident belongs to team %d, want %d", teamID, red.id)
}
blueIncidents := list(t, blue.call(http.MethodGet, "/api/incidents", nil))
if len(blueIncidents) != 1 || blueIncidents[0]["title"] != "BlueDiskFull" {
t.Fatalf("blue should see exactly its own incident, saw %v", blueIncidents)
}
// Reading the other team's incident by id is not found rather than
// forbidden: its existence is the other team's business.
otherID := int64(blueIncidents[0]["id"].(float64))
for _, path := range []string{
"/api/incidents/" + id64(otherID),
"/api/incidents/" + id64(otherID) + "/alerts",
"/api/incidents/" + id64(otherID) + "/timeline",
} {
resp := red.call(http.MethodGet, path, nil)
resp.Body.Close()
if resp.StatusCode != http.StatusNotFound {
t.Errorf("red reading %s: expected 404, got %d", path, resp.StatusCode)
}
}
// And cannot act on it either.
for _, path := range []string{"/acknowledge", "/resolve", "/archive"} {
resp := red.call(http.MethodPost, "/api/incidents/"+id64(otherID)+path, nil)
resp.Body.Close()
if resp.StatusCode != http.StatusNotFound {
t.Errorf("red posting %s: expected 404, got %d", path, resp.StatusCode)
}
}
}
// Alerts, the raw signal record, are scoped the same way.
func TestTeams_AlertsAndStatsAreScoped(t *testing.T) {
s := newTS(t)
red := newTeam(t, s, "red")
blue := newTeam(t, s, "blue")
postToIntegration(t, s, red.key, "fp-red", "RedDiskFull")
postToIntegration(t, s, blue.key, "fp-blue-1", "BlueDiskFull")
postToIntegration(t, s, blue.key, "fp-blue-2", "BlueMemory")
if alerts := list(t, red.call(http.MethodGet, "/api/alerts", nil)); len(alerts) != 1 {
t.Errorf("red should see 1 alert, saw %d", len(alerts))
}
if alerts := list(t, blue.call(http.MethodGet, "/api/alerts", nil)); len(alerts) != 2 {
t.Errorf("blue should see 2 alerts, saw %d", len(alerts))
}
// Statistics count your own work only — otherwise a team's volume, and the
// names of its alerts, leak through the totals.
var stats map[string]any
decode(t, red.call(http.MethodGet, "/api/stats/alerts", nil), &stats)
if total := stats["total"].(float64); total != 1 {
t.Errorf("red's alert stats counted %v alerts, want 1", total)
}
top := list(t, red.call(http.MethodGet, "/api/stats/alerts/top", nil))
for _, row := range top {
if name := row["name"].(string); name != "RedDiskFull" {
t.Errorf("red's top alerts named %q, which is not theirs", name)
}
}
}
// The same fingerprint, the same groupKey and the same date are all legitimate
// in two teams at once: two clusters running the same rules, two rotas.
func TestTeams_SameFingerprintInTwoTeams(t *testing.T) {
s := newTS(t)
red := newTeam(t, s, "red")
blue := newTeam(t, s, "blue")
postToIntegration(t, s, red.key, "fp-shared", "DiskFull")
postToIntegration(t, s, blue.key, "fp-shared", "DiskFull")
for _, team := range []struct {
name string
f teamFixture
}{{"red", red}, {"blue", blue}} {
incidents := list(t, team.f.call(http.MethodGet, "/api/incidents", nil))
if len(incidents) != 1 {
t.Errorf("%s: expected its own incident for the shared fingerprint, saw %d",
team.name, len(incidents))
}
}
// And both rotas can name somebody for the same day.
for _, team := range []struct {
name string
f teamFixture
}{{"red", red}, {"blue", blue}} {
var members []map[string]any
decode(t, team.f.call(http.MethodGet, "/api/teams/"+id64(team.f.id)+"/members", nil), &members)
userID := int64(members[0]["user_id"].(float64))
resp := team.f.call(http.MethodPost, "/api/teams/"+id64(team.f.id)+"/schedule",
map[string]any{"user_id": userID, "dates": []string{"2026-10-01"}})
resp.Body.Close()
if resp.StatusCode != http.StatusCreated {
t.Errorf("%s: taking 2026-10-01 returned %d", team.name, resp.StatusCode)
}
}
}
// An unknown key delivers nothing, and says so rather than accepting silently.
func TestTeams_UnknownIntegrationKeyIsRejected(t *testing.T) {
s := newTS(t)
team := newTeam(t, s, "red")
resp, err := http.Post(s.URL+"/api/integrations/not-a-real-key/alertmanager",
"application/json", bytes.NewReader([]byte(`{"version":"4","status":"firing","alerts":[]}`)))
if err != nil {
t.Fatalf("post: %v", err)
}
resp.Body.Close()
if resp.StatusCode != http.StatusUnauthorized {
t.Errorf("expected 401 for an unknown key, got %d", resp.StatusCode)
}
if incidents := list(t, team.call(http.MethodGet, "/api/incidents", nil)); len(incidents) != 0 {
t.Errorf("a rejected payload opened %d incident(s)", len(incidents))
}
}
// Team configuration is an owner's job; working incidents is a member's.
func TestTeams_MemberCannotConfigureTheTeam(t *testing.T) {
s := newTS(t)
team := newTeam(t, s, "red")
// A plain member of the same team.
var user struct {
ID int64 `json:"id"`
}
decode(t, s.req(t, http.MethodPost, "/api/users",
map[string]string{"username": "plain", "email": "plain@test.com"}), &user)
resp := s.req(t, http.MethodPost, "/api/teams/"+id64(team.id)+"/members",
map[string]any{"user_id": user.ID, "role": "member"})
resp.Body.Close()
var key struct {
Key string `json:"key"`
}
decode(t, s.req(t, http.MethodPost, "/api/users/"+id64(user.ID)+"/api-keys",
map[string]string{"name": "test"}), &key)
call := func(method, path string, body any) *http.Response {
t.Helper()
var r io.Reader
if body != nil {
data, _ := json.Marshal(body)
r = bytes.NewReader(data)
}
req, _ := http.NewRequest(method, s.URL+path, r)
req.Header.Set("Authorization", "Bearer "+key.Key)
if body != nil {
req.Header.Set("Content-Type", "application/json")
}
resp, err := http.DefaultClient.Do(req)
if err != nil {
t.Fatalf("%s %s: %v", method, path, err)
}
return resp
}
base := "/api/teams/" + id64(team.id)
for _, c := range []struct {
name string
method string
path string
body any
}{
{"mint an integration key", http.MethodPost, base + "/integrations",
map[string]string{"name": "mine"}},
{"take a shift", http.MethodPost, base + "/schedule",
map[string]any{"user_id": user.ID, "dates": []string{"2026-11-01"}}},
{"add a member", http.MethodPost, base + "/members",
map[string]any{"user_id": 1}},
{"delete the team", http.MethodDelete, base, nil},
} {
resp := call(c.method, c.path, c.body)
resp.Body.Close()
if resp.StatusCode != http.StatusForbidden {
t.Errorf("%s: expected 403, got %d", c.name, resp.StatusCode)
}
}
// But they can read what the team is doing.
for _, path := range []string{base + "/members", base + "/integrations", base + "/schedule"} {
resp := call(http.MethodGet, path, nil)
resp.Body.Close()
if resp.StatusCode != http.StatusOK {
t.Errorf("reading %s: expected 200, got %d", path, resp.StatusCode)
}
}
}
// A team is not somewhere an outsider can look, whatever they know about it.
func TestTeams_OutsiderSeesNothing(t *testing.T) {
s := newTS(t)
red := newTeam(t, s, "red")
blue := newTeam(t, s, "blue")
base := "/api/teams/" + id64(red.id)
for _, path := range []string{base + "/members", base + "/integrations", base + "/schedule"} {
resp := blue.call(http.MethodGet, path, nil)
resp.Body.Close()
if resp.StatusCode != http.StatusNotFound {
t.Errorf("blue reading %s: expected 404, got %d", path, resp.StatusCode)
}
}
// /api/teams lists your own, never the install's.
teams := list(t, blue.call(http.MethodGet, "/api/teams", nil))
if len(teams) != 1 || teams[0]["name"] != "blue" {
t.Errorf("blue's team list: %v", teams)
}
}
+19
View File
@@ -7,7 +7,9 @@ import (
"os"
"strings"
"testing"
"time"
"git.ryuvia.com/niklas/terdut-server/internal/config"
"git.ryuvia.com/niklas/terdut-server/internal/db"
)
@@ -30,6 +32,23 @@ import (
// tests nothing is worse than one that does not run.
const testDSNEnv = "TERDUT_TEST_DSN"
// testConfig is the environment half of the server's configuration, which the
// admin settings page renders read-only and SeedSettings seeds the editable
// half from. The durations match the defaults config.Load would produce, so a
// test that never touches the settings table behaves as a fresh install does.
func testConfig() config.Config {
return config.Config{
Addr: ":8080",
ArchiveAfter: 7 * 24 * time.Hour,
StaleAfter: 6 * time.Hour,
NotifyRepeat: 15 * time.Minute,
}
}
// defaultTeam is the team migration 003 creates and the bootstrap user owns, as
// a path segment. Every test that does not say otherwise works inside it.
const defaultTeam = "1"
var schemaSeq int
// newTestDB returns a migrated database private to this test, and drops it
+109 -5
View File
@@ -58,7 +58,7 @@ func handleBootstrap(db *sql.DB) http.HandlerFunc {
var userID int64
if err := db.QueryRowContext(r.Context(),
"INSERT INTO users (username, email, password_hash) VALUES ($1, $2, $3) RETURNING id",
"INSERT INTO users (username, email, password_hash, is_admin) VALUES ($1, $2, $3, true) RETURNING id",
req.Username, req.Email, passwordHash).Scan(&userID); err != nil {
respond(w, http.StatusInternalServerError, errResp("internal error"))
return
@@ -77,6 +77,16 @@ func handleBootstrap(db *sql.DB) http.HandlerFunc {
return
}
// The default team exists from migration 003, on a fresh install too.
// Without a membership the first user signs in to a working server with
// no queue, no schedule and nowhere for an integration to hang off.
if teamID, err := defaultTeamID(r.Context(), db); err == nil {
db.ExecContext(r.Context(), //nolint:errcheck
"INSERT INTO team_members (team_id, user_id, role) VALUES ($1, $2, $3) "+
"ON CONFLICT (team_id, user_id) DO NOTHING",
teamID, userID, models.RoleOwner)
}
user, _ := fetchUser(r.Context(), db, userID)
key := models.APIKey{ID: keyID, UserID: userID, Name: "bootstrap", Key: raw, CreatedAt: user.CreatedAt}
respond(w, http.StatusCreated, map[string]any{"user": user, "api_key": key})
@@ -86,7 +96,7 @@ func handleBootstrap(db *sql.DB) http.HandlerFunc {
func handleListUsers(db *sql.DB) http.HandlerFunc {
return func(w http.ResponseWriter, r *http.Request) {
rows, err := db.QueryContext(r.Context(),
"SELECT id, username, email, created_at, ntfy_topic FROM users ORDER BY id")
"SELECT id, username, email, created_at, ntfy_topic, is_admin, disabled_at FROM users ORDER BY id")
if err != nil {
respond(w, http.StatusInternalServerError, errResp("internal error"))
return
@@ -97,11 +107,13 @@ func handleListUsers(db *sql.DB) http.HandlerFunc {
for rows.Next() {
var u models.User
var ts int64
if err := rows.Scan(&u.ID, &u.Username, &u.Email, &ts, &u.NtfyTopic); err != nil {
var disabled *int64
if err := rows.Scan(&u.ID, &u.Username, &u.Email, &ts, &u.NtfyTopic, &u.IsAdmin, &disabled); err != nil {
respond(w, http.StatusInternalServerError, errResp("internal error"))
return
}
u.CreatedAt = time.Unix(ts, 0).UTC()
u.DisabledAt = unixPtr(disabled)
users = append(users, u)
}
respond(w, http.StatusOK, users)
@@ -150,6 +162,9 @@ func handleSetNotifyTarget(db *sql.DB) http.HandlerFunc {
respond(w, http.StatusBadRequest, errResp("invalid user id"))
return
}
if !requireSelfOrAdmin(w, r, id) {
return
}
var req struct {
NtfyTopic string `json:"ntfy_topic"`
}
@@ -190,6 +205,21 @@ func handleDeleteUser(db *sql.DB) http.HandlerFunc {
respond(w, http.StatusBadRequest, errResp("invalid user id"))
return
}
// Deleting yourself is how an install ends up with no administrator at
// all, and it is never what somebody meant to do.
caller, _ := userFromContext(r.Context())
if caller.ID == id {
respond(w, http.StatusConflict, errResp("cannot delete your own account"))
return
}
if last, err := isLastAdmin(r.Context(), db, id); err != nil {
respond(w, http.StatusInternalServerError, errResp("internal error"))
return
} else if last {
respond(w, http.StatusConflict, errResp("cannot delete the last administrator"))
return
}
res, err := db.ExecContext(r.Context(), "DELETE FROM users WHERE id = $1", id)
if err != nil {
respond(w, http.StatusInternalServerError, errResp("internal error"))
@@ -211,6 +241,9 @@ func handleCreateAPIKey(db *sql.DB) http.HandlerFunc {
respond(w, http.StatusBadRequest, errResp("invalid user id"))
return
}
if !requireSelfOrAdmin(w, r, userID) {
return
}
var req struct {
Name string `json:"name"`
@@ -254,6 +287,9 @@ func handleDeleteAPIKey(db *sql.DB) http.HandlerFunc {
respond(w, http.StatusBadRequest, errResp("invalid user id"))
return
}
if !requireSelfOrAdmin(w, r, userID) {
return
}
keyID, err := strconv.ParseInt(chi.URLParam(r, "keyID"), 10, 64)
if err != nil {
respond(w, http.StatusBadRequest, errResp("invalid key id"))
@@ -291,12 +327,80 @@ func randomToken() (raw, hash string, err error) {
func fetchUser(ctx context.Context, db *sql.DB, id int64) (models.User, error) {
var u models.User
var ts int64
var disabled *int64
err := db.QueryRowContext(ctx,
"SELECT id, username, email, created_at, ntfy_topic FROM users WHERE id = $1", id).
Scan(&u.ID, &u.Username, &u.Email, &ts, &u.NtfyTopic)
"SELECT id, username, email, created_at, ntfy_topic, is_admin, disabled_at FROM users WHERE id = $1", id).
Scan(&u.ID, &u.Username, &u.Email, &ts, &u.NtfyTopic, &u.IsAdmin, &disabled)
if err != nil {
return u, err
}
u.CreatedAt = time.Unix(ts, 0).UTC()
u.DisabledAt = unixPtr(disabled)
return u, nil
}
// handleSetAdmin grants or revokes the system administrator flag.
//
// Revoking is guarded twice: an install must keep at least one administrator,
// and you cannot demote yourself. The first stops the flag being lost
// altogether; the second stops the likelier accident, where the only admin
// clears their own flag while tidying up and locks the door behind them.
func handleSetAdmin(db *sql.DB) http.HandlerFunc {
return func(w http.ResponseWriter, r *http.Request) {
id, err := strconv.ParseInt(chi.URLParam(r, "id"), 10, 64)
if err != nil {
respond(w, http.StatusBadRequest, errResp("invalid user id"))
return
}
var req struct {
IsAdmin *bool `json:"is_admin"`
}
if err := decodeJSON(r, &req); err != nil || req.IsAdmin == nil {
respond(w, http.StatusBadRequest, errResp("is_admin is required"))
return
}
if !*req.IsAdmin {
caller, _ := userFromContext(r.Context())
if caller.ID == id {
respond(w, http.StatusConflict, errResp("cannot revoke your own administrator access"))
return
}
if last, err := isLastAdmin(r.Context(), db, id); err != nil {
respond(w, http.StatusInternalServerError, errResp("internal error"))
return
} else if last {
respond(w, http.StatusConflict, errResp("cannot revoke the last administrator"))
return
}
}
res, err := db.ExecContext(r.Context(),
"UPDATE users SET is_admin = $1 WHERE id = $2", *req.IsAdmin, id)
if err != nil {
respond(w, http.StatusInternalServerError, errResp("internal error"))
return
}
if n, _ := res.RowsAffected(); n == 0 {
respond(w, http.StatusNotFound, errResp("user not found"))
return
}
user, err := fetchUser(r.Context(), db, id)
if err != nil {
respond(w, http.StatusInternalServerError, errResp("internal error"))
return
}
respond(w, http.StatusOK, user)
}
}
// isLastAdmin reports whether id is an administrator and no other user is one.
// A non-admin id is never the last one, so removing them is always allowed.
func isLastAdmin(ctx context.Context, db *sql.DB, id int64) (bool, error) {
var last bool
err := db.QueryRowContext(ctx, `
SELECT EXISTS (SELECT 1 FROM users WHERE id = $1 AND is_admin)
AND NOT EXISTS (SELECT 1 FROM users WHERE id <> $1 AND is_admin)`, id).Scan(&last)
return last, err
}
+25
View File
@@ -0,0 +1,25 @@
-- A system administrator role, and the first thing in this server that one user
-- can do and another cannot.
--
-- Until now every authenticated caller could create and delete users, set
-- anybody's password and mint API keys for anybody — auth.go said so in a
-- comment. That was defensible with one operator and a hand-made account; it is
-- not once people sign themselves up (see #7).
--
-- EVERY EXISTING USER BECOMES AN ADMIN. They already hold these powers, so
-- this migration changes nobody's access: it names what is already true, and
-- leaves demotion as a deliberate act somebody performs afterwards. The
-- alternative — promoting only user 1 — would silently strip the others, and
-- could leave an install whose only admin is an account nobody has a password
-- for.
--
-- New users are not admins: the column defaults to false, and the only ways to
-- become one are this backfill, the bootstrap endpoint, or an existing admin
-- granting it.
ALTER TABLE users ADD COLUMN is_admin BOOLEAN NOT NULL DEFAULT false;
UPDATE users SET is_admin = true;
-- The queue's assignment dropdown and the on-call schedule read every user, and
-- the admin screens in #5 will filter on this.
CREATE INDEX users_is_admin_idx ON users(is_admin) WHERE is_admin;
+103
View File
@@ -0,0 +1,103 @@
-- Teams: the unit of tenancy. Everything a person works on now belongs to one.
--
-- Until this migration the install was one shared space — every user saw every
-- alert and every incident, and the Alertmanager webhook was unauthenticated, so
-- anything that could reach the port could open an incident for everybody.
--
-- The shape, in one paragraph: a team owns its incidents, alerts, schedule and
-- integrations. A user belongs to as many teams as they like, with a role in
-- each: an `owner` configures the team, a `member` works its incidents. An
-- integration key is what an alert arrives on, and the key is what says which
-- team the alert belongs to.
--
-- EVERYTHING EXISTING MOVES INTO ONE DEFAULT TEAM, and every existing user
-- becomes an owner of it. That keeps an upgrade a no-op for the people using it:
-- the same queue, the same schedule, the same incidents, with a name on them.
CREATE TABLE teams (
id BIGINT GENERATED BY DEFAULT AS IDENTITY PRIMARY KEY,
name TEXT NOT NULL UNIQUE,
created_at BIGINT NOT NULL DEFAULT FLOOR(EXTRACT(EPOCH FROM now()))::bigint
);
-- role is free text with a CHECK rather than an enum, so adding a third role
-- later is a migration and not a type rewrite.
CREATE TABLE team_members (
team_id BIGINT NOT NULL REFERENCES teams(id) ON DELETE CASCADE,
user_id BIGINT NOT NULL REFERENCES users(id) ON DELETE CASCADE,
role TEXT NOT NULL CHECK (role IN ('owner', 'member')),
joined_at BIGINT NOT NULL DEFAULT FLOOR(EXTRACT(EPOCH FROM now()))::bigint,
PRIMARY KEY (team_id, user_id)
);
CREATE INDEX team_members_user_idx ON team_members(user_id);
-- How alerts get in, and the only thing that says which team they belong to.
-- The key is stored as a SHA-256 hash, like api_keys and the ack tokens: a
-- leaked database gives nobody the ability to post alerts.
CREATE TABLE integrations (
id BIGINT GENERATED BY DEFAULT AS IDENTITY PRIMARY KEY,
team_id BIGINT NOT NULL REFERENCES teams(id) ON DELETE CASCADE,
kind TEXT NOT NULL CHECK (kind IN ('alertmanager')),
name TEXT NOT NULL,
key_hash TEXT NOT NULL UNIQUE,
created_at BIGINT NOT NULL DEFAULT FLOOR(EXTRACT(EPOCH FROM now()))::bigint,
last_used_at BIGINT
);
CREATE INDEX integrations_team_idx ON integrations(team_id);
-- ---------------------------------------------------------------------------
-- The default team, and everything that already exists moving into it.
--
-- Created unconditionally, even on an empty install, so there is always a team
-- for the bootstrap user to land in and for the first integration to hang off.
-- ---------------------------------------------------------------------------
INSERT INTO teams (name) VALUES ('Default');
INSERT INTO team_members (team_id, user_id, role)
SELECT (SELECT id FROM teams WHERE name = 'Default'), id, 'owner' FROM users;
-- ---------------------------------------------------------------------------
-- team_id on everything a team owns.
--
-- Added nullable, backfilled, then made NOT NULL: adding a NOT NULL column with
-- no default to a table with rows is rejected, and a DEFAULT pointing at the
-- default team would quietly keep working after the default team is gone.
-- ---------------------------------------------------------------------------
ALTER TABLE alerts ADD COLUMN team_id BIGINT REFERENCES teams(id) ON DELETE CASCADE;
ALTER TABLE incidents ADD COLUMN team_id BIGINT REFERENCES teams(id) ON DELETE CASCADE;
ALTER TABLE schedule_entries ADD COLUMN team_id BIGINT REFERENCES teams(id) ON DELETE CASCADE;
UPDATE alerts SET team_id = (SELECT id FROM teams WHERE name = 'Default');
UPDATE incidents SET team_id = (SELECT id FROM teams WHERE name = 'Default');
UPDATE schedule_entries SET team_id = (SELECT id FROM teams WHERE name = 'Default');
ALTER TABLE alerts ALTER COLUMN team_id SET NOT NULL;
ALTER TABLE incidents ALTER COLUMN team_id SET NOT NULL;
ALTER TABLE schedule_entries ALTER COLUMN team_id SET NOT NULL;
-- ---------------------------------------------------------------------------
-- The uniqueness rules were all written for one tenant, and every one of them
-- is wrong now: two teams monitoring two clusters legitimately see the same
-- fingerprint, the same groupKey, and want somebody on call on the same day.
-- ---------------------------------------------------------------------------
ALTER TABLE alerts DROP CONSTRAINT alerts_fingerprint_key;
CREATE UNIQUE INDEX alerts_team_fingerprint_idx ON alerts(team_id, fingerprint);
DROP INDEX incidents_open_group_key_idx;
-- Still load-bearing, now per team: at most one OPEN incident per group_key
-- within a team. This is what makes "resolved incident + a new alert occurrence
-- = a new incident" work, and what the webhook's find-or-open lookup relies on.
CREATE UNIQUE INDEX incidents_open_group_key_idx
ON incidents(team_id, group_key) WHERE resolved_at IS NULL;
ALTER TABLE schedule_entries DROP CONSTRAINT schedule_entries_date_key;
CREATE UNIQUE INDEX schedule_entries_team_date_idx ON schedule_entries(team_id, date);
-- The list views all filter by team first.
CREATE INDEX alerts_team_received_idx ON alerts(team_id, received_at DESC);
CREATE INDEX incidents_team_triggered_idx ON incidents(team_id, triggered_at DESC);
@@ -0,0 +1,39 @@
-- Dead man's switches become a team's own configuration.
--
-- They were three environment variables — TERDUT_DEADMAN_MATCHERS, _TIMEOUT and
-- _SEVERITY — which made them one setting for the whole install. That was the
-- last piece of the alerting path a team could not control: a team could take
-- its own alerts on its own key and still not say which of them were
-- heartbeats, or how long a silence had to last before somebody was paged.
--
-- One row per team rather than one row per switch. The unit of monitoring is
-- still the fingerprint, as it always was — two clusters sending the same
-- heartbeat alertname are two independent switches — and the matcher string
-- keeps the format the environment variable used, so a value can be moved from
-- one to the other unchanged.
--
-- No rows are seeded here: a migration cannot read the environment. The server
-- inserts a row per team at startup from its own configuration, and the same
-- values therefore carry forward into the first team's row without anybody
-- retyping them. See seedDeadmanConfigs.
CREATE TABLE deadman_configs (
team_id BIGINT PRIMARY KEY REFERENCES teams(id) ON DELETE CASCADE,
-- ";" separates matchers, "," the label conditions within one, "=" is exact
-- equality: `alertname=Watchdog,cluster=prod; alertname=EdgeHeartbeat`.
-- Every matcher must name an alertname. Empty watches nothing.
matchers TEXT NOT NULL DEFAULT '',
-- Seconds rather than a Go duration string: the column is compared and
-- arithmetic is done on it, and a value that has to be parsed before it can
-- be believed is a value that can be stored unparseable. Zero disables the
-- team's switches entirely.
timeout_seconds BIGINT NOT NULL DEFAULT 0,
-- The severity these incidents open at. They have no member alerts to
-- derive one from, and a heartbeat's own severity label is meaningless —
-- Watchdog ships as "none".
severity TEXT NOT NULL DEFAULT 'critical',
updated_at BIGINT NOT NULL DEFAULT FLOOR(EXTRACT(EPOCH FROM now()))::bigint
);
+35
View File
@@ -0,0 +1,35 @@
-- Settings that an administrator can change without a redeploy, and the flag
-- that takes an account out of use without deleting it.
--
-- Three of the server's tunables were environment variables, which meant
-- changing how long an incident waits before it is paged again required editing
-- a chart, merging it, and waiting for a reconcile. They are behaviour, not
-- infrastructure, and the difference is who needs to change them and how often.
--
-- What stays in the environment: the ntfy URL and token, the database DSN, the
-- listen address and the public URL. Those are where the server is plugged in
-- rather than how it behaves, they are needed before the database is open, and
-- two of them are credentials.
--
-- Key/value rather than a column per setting. A settings table with one row and
-- a column per knob needs a migration for every new knob, and #6 and #7 will
-- both add some. The cost is that values are text and the accessor has to say
-- what type it wanted; settings.go does that in one place.
--
-- No rows are seeded here: a migration cannot read the environment. The server
-- inserts each key from its own configuration at startup, once, so an install
-- that upgrades keeps exactly the behaviour it had. See SeedSettings.
CREATE TABLE settings (
key TEXT PRIMARY KEY,
value TEXT NOT NULL,
updated_at BIGINT NOT NULL DEFAULT FLOOR(EXTRACT(EPOCH FROM now()))::bigint
);
-- Disabling an account rather than deleting it: the person has left, or the
-- credential is suspect, and their incidents, acknowledgements and timeline
-- entries must stay exactly where they are. Deleting a user nulls their
-- acknowledged_by and assigned_to, which quietly rewrites history.
--
-- A disabled user cannot sign in and their API keys stop working, but they are
-- still a name the timeline can show and still a member of their teams.
ALTER TABLE users ADD COLUMN disabled_at BIGINT;
+95
View File
@@ -0,0 +1,95 @@
-- Escalation: page somebody else when the first person does not answer.
--
-- This is the gap the whole multi-tenancy line of work was opened to close.
-- Until now an unacknowledged incident re-paged the same topic every
-- notify_repeat forever, which is a louder version of the same silence: if the
-- person on call is asleep, has no signal, or has left, nothing else happens.
--
-- Shape: one policy per team, an ordered list of levels, each level with a
-- timeout and a set of targets. When a level's timeout passes and the incident
-- is still triggered, the next level is paged. When the last level passes, the
-- chain repeats repeat_count times, and then the team's fallback topic is paged
-- once as the end of the line.
--
-- A team WITHOUT a policy keeps exactly today's behaviour: page the assignee,
-- then remind on the same topic. Escalation is opt-in per team, and the two
-- never both run for one incident -- see enqueueReminders.
CREATE TABLE escalation_policies (
-- One per team for now, hence the team as the key rather than an id with a
-- unique index: routing different alerts to different chains needs the
-- alert to carry something to route ON, which is a separate question.
team_id BIGINT PRIMARY KEY REFERENCES teams(id) ON DELETE CASCADE,
-- How many extra times to run the whole chain after it has been walked
-- once. 0 means walk it once and stop at the fallback.
repeat_count BIGINT NOT NULL DEFAULT 0 CHECK (repeat_count >= 0 AND repeat_count <= 10),
-- Where the last page goes when every level has been tried. Per team now:
-- TERDUT_NTFY_FALLBACK_TOPIC was one topic for the whole install, which in
-- a multi-team server pages the wrong people. Empty means the chain simply
-- ends.
fallback_topic TEXT NOT NULL DEFAULT '',
updated_at BIGINT NOT NULL DEFAULT FLOOR(EXTRACT(EPOCH FROM now()))::bigint
);
CREATE TABLE escalation_levels (
id BIGINT GENERATED BY DEFAULT AS IDENTITY PRIMARY KEY,
team_id BIGINT NOT NULL REFERENCES escalation_policies(team_id) ON DELETE CASCADE,
-- 1-based, dense. The API rewrites the whole ladder on every edit rather
-- than patching one rung, so there is no way to leave a gap.
position BIGINT NOT NULL,
-- How long this level has to produce an acknowledgement before the next one
-- is paged. Seconds, like every other duration in this schema.
timeout_seconds BIGINT NOT NULL CHECK (timeout_seconds > 0),
UNIQUE (team_id, position)
);
-- Who a level pages. Either a named person, or whoever the team's rota says is
-- on call today -- which is the target that keeps working when the rota
-- changes and nobody remembers to edit the policy.
CREATE TABLE escalation_targets (
id BIGINT GENERATED BY DEFAULT AS IDENTITY PRIMARY KEY,
level_id BIGINT NOT NULL REFERENCES escalation_levels(id) ON DELETE CASCADE,
kind TEXT NOT NULL CHECK (kind IN ('user', 'oncall')),
-- Set for kind='user', NULL for kind='oncall'.
user_id BIGINT REFERENCES users(id) ON DELETE CASCADE,
CHECK ((kind = 'user' AND user_id IS NOT NULL) OR (kind = 'oncall' AND user_id IS NULL))
);
CREATE INDEX escalation_targets_level_idx ON escalation_targets(level_id);
-- ---------------------------------------------------------------------------
-- Where an incident is in its chain.
--
-- On the incident rather than in a side table: it is read on every notifier
-- tick alongside the incident's status, and one row per incident is exactly
-- what the state is.
-- ---------------------------------------------------------------------------
-- 0 means no level has been paged yet, which is the state of every incident
-- that existed before escalation and of every incident in a team with no
-- policy. 1 is the first level.
ALTER TABLE incidents ADD COLUMN escalation_level BIGINT NOT NULL DEFAULT 0;
-- When the current level was entered, and therefore what its timeout is
-- measured from. NULL while escalation_level is 0.
ALTER TABLE incidents ADD COLUMN escalation_level_at BIGINT;
-- How many times the chain has been walked in full. Compared against the
-- policy's repeat_count.
ALTER TABLE incidents ADD COLUMN escalation_round BIGINT NOT NULL DEFAULT 0;
-- The notifier's escalation query: incidents still waiting, oldest level first.
CREATE INDEX incidents_escalation_idx
ON incidents(escalation_level_at)
WHERE resolved_at IS NULL AND status = 'triggered';
-- 'escalated' joins the outbox kinds: a page that went out because nobody
-- answered the last one, which is worth telling apart from the first page and
-- from a reminder when reading the timeline or debugging a delivery.
ALTER TABLE notifications DROP CONSTRAINT notifications_kind_check;
ALTER TABLE notifications ADD CONSTRAINT notifications_kind_check
CHECK (kind IN ('triggered', 'reminder', 'resolved', 'escalated'));
+7 -1
View File
@@ -7,7 +7,13 @@ import "time"
// to it — acknowledgement, assignment, notes and closure all live on the
// Incident an alert belongs to.
type Alert struct {
ID int64 `json:"id"`
ID int64 `json:"id"`
// TeamID is the team whose integration received this alert, and TeamName
// rides along so a combined list can label a row without a second request.
TeamID int64 `json:"team_id"`
TeamName string `json:"team_name,omitempty"`
Fingerprint string `json:"fingerprint"`
Name string `json:"name"`
Status string `json:"status"` // "firing" or "resolved"
+13
View File
@@ -11,6 +11,19 @@ import "time"
// the webhook and the sweeper may flip to "resolved" once every member alert has
// stopped firing.
type Incident struct {
// EscalationLevel is which rung of its team's ladder this incident is on,
// 0 for none — either the team has no ladder, or somebody has answered.
// EscalationDueAt is when the current level runs out, so a client can say
// how long is left rather than only what already happened.
EscalationLevel int64 `json:"escalation_level"`
EscalationDueAt *time.Time `json:"escalation_due_at,omitempty"`
// TeamID is the team that owns this incident, fixed when it opens: an
// incident never moves between teams. TeamName rides along so the combined
// queue can badge each row without a second request.
TeamID int64 `json:"team_id"`
TeamName string `json:"team_name,omitempty"`
ID int64 `json:"id"`
GroupKey string `json:"group_key"`
Title string `json:"title"`
+8 -1
View File
@@ -3,7 +3,14 @@ package models
import "time"
type ScheduleEntry struct {
ID int64 `json:"id"`
ID int64 `json:"id"`
// TeamID is whose rota this shift belongs to; TeamName rides along so the
// combined "who is on call" view can label each entry without a second
// request.
TeamID int64 `json:"team_id"`
TeamName string `json:"team_name,omitempty"`
UserID int64 `json:"user_id"`
Username string `json:"username"`
Date string `json:"date"` // YYYY-MM-DD
+54
View File
@@ -0,0 +1,54 @@
package models
import "time"
// Team is the unit of tenancy: it owns its incidents, alerts, schedule and
// integrations, and a user sees exactly the teams they belong to.
type Team struct {
ID int64 `json:"id"`
Name string `json:"name"`
CreatedAt time.Time `json:"created_at"`
// Role is the caller's own role in this team, populated when a team is
// listed for a particular person. Empty when nobody in particular is
// asking, as in the admin listing.
Role string `json:"role,omitempty"`
}
// Team roles. An owner configures the team — its schedule, its integrations and
// who is in it. A member works its incidents.
const (
RoleOwner = "owner"
RoleMember = "member"
)
// TeamMember is one person's membership of one team.
type TeamMember struct {
TeamID int64 `json:"team_id"`
UserID int64 `json:"user_id"`
Username string `json:"username"`
Role string `json:"role"`
JoinedAt time.Time `json:"joined_at"`
}
// Integration is how alerts get in, and the only thing that says which team an
// arriving alert belongs to.
type Integration struct {
ID int64 `json:"id"`
TeamID int64 `json:"team_id"`
Kind string `json:"kind"`
Name string `json:"name"`
CreatedAt time.Time `json:"created_at"`
LastUsedAt *time.Time `json:"last_used_at,omitempty"`
// Key is the raw integration key, shown once when the integration is
// created and never stored. URL is the address to point the sender at,
// likewise only complete at creation time.
Key string `json:"key,omitempty"`
URL string `json:"url,omitempty"`
}
// Integration kinds.
const (
IntegrationAlertmanager = "alertmanager"
)
+11
View File
@@ -12,6 +12,17 @@ type User struct {
// none of their own; incidents assigned to them fall back to the configured
// fallback topic instead.
NtfyTopic *string `json:"ntfy_topic,omitempty"`
// DisabledAt is when the account was taken out of use, or nil. A disabled
// user cannot authenticate by either credential, and keeps their name on
// every acknowledgement and timeline entry they made.
DisabledAt *time.Time `json:"disabled_at,omitempty"`
// IsAdmin is the system administrator flag: managing users and API keys.
// Not omitempty — a client has to be able to tell "false" from "this server
// is too old to have the field", and the web UI decides what to show from
// it.
IsAdmin bool `json:"is_admin"`
}
type APIKey struct {
+67
View File
@@ -285,6 +285,16 @@ input:focus, textarea:focus { outline: none; border-color: var(--accent); box-sh
min-width: 0;
}
.row-meta .labels { overflow: hidden; text-overflow: ellipsis; white-space: nowrap; min-width: 0; max-width: 100%; color: var(--faint); }
/* Which team's queue a row came from. Only rendered for somebody in more than
one team, so it never repeats the same word down the whole list. */
/* Separates the status chips from the team chips in the queue's filter row. */
.chip-sep { width: 1px; align-self: stretch; background: var(--border); margin: 0 2px; }
.row-team {
padding: 1px 6px; border-radius: 4px;
background: var(--surface-2); border: 1px solid var(--border);
color: var(--muted); font-size: 12px; white-space: nowrap;
}
.row.resolved .row-title { color: var(--muted); }
.sev-critical { --sev: var(--crit); }
@@ -620,3 +630,60 @@ kbd {
.toast, .app.detail-open ~ .toast { bottom: 24px; }
.only-desktop { display: block; }
}
/* --- admin ---------------------------------------------------------------
The admin page is three tables of things you act on, so it needs table
styling the rest of the app never did: the queue is a list of links and the
account page is a form. */
.admin-table { width: 100%; border-collapse: collapse; font-size: 14px; }
.admin-table th {
text-align: left; font-weight: 600; color: var(--muted); font-size: 12px;
text-transform: uppercase; letter-spacing: 0.04em;
padding: 4px 8px 4px 0; border-bottom: 1px solid var(--border);
}
.admin-table td { padding: 8px 8px 8px 0; border-bottom: 1px solid var(--border); vertical-align: middle; }
.admin-table tr:last-child td { border-bottom: none; }
.admin-table .num { text-align: right; font-variant-numeric: tabular-nums; }
.admin-table td .btn-sm + .btn-sm { margin-left: 6px; }
/* A disabled account stays readable — it is still the name on old
acknowledgements — but should not look like a working one. */
.disabled-row td { opacity: 0.55; }
.btn-sm.danger { color: var(--crit); border-color: var(--crit-soft); }
.inline-form { display: flex; gap: 8px; margin-top: 12px; }
.inline-form input { flex: 1; min-width: 0; }
.admin-settings .setting-value { width: 5.5em; margin-right: 6px; }
.admin-settings .setting-unit { max-width: 8em; }
.admin-settings button[type="submit"] { margin-top: 12px; }
.small { font-size: 13px; }
/* --- team settings -------------------------------------------------------
Forms with a label above each control, rather than the queue's rows of
links. The escalation ladder is the only nested structure in the app, so it
gets a little indentation to make the levels read as an order. */
.stacked-form { display: flex; flex-direction: column; gap: 10px; margin-top: 12px; align-items: flex-start; }
.stacked-form label { display: flex; align-items: center; gap: 6px; flex-wrap: wrap; font-size: 14px; }
.stacked-form label.checkbox { gap: 8px; }
.stacked-form input.wide { min-width: min(420px, 100%); }
.team-picker { margin-top: 8px; max-width: 100%; }
.ladder-level {
border-left: 3px solid var(--border-strong);
padding: 8px 0 8px 12px; margin: 12px 0;
}
.ladder-head { display: flex; align-items: center; gap: 10px; margin-bottom: 6px; }
.ladder-targets { display: flex; flex-direction: column; gap: 6px; margin-top: 8px; }
.target-row { display: flex; gap: 6px; align-items: center; flex-wrap: wrap; }
/* An integration key is shown exactly once, so it should look like something
to act on rather than another row of text. */
.key-panel {
margin-top: 12px; padding: 12px;
border: 1px solid var(--accent); border-radius: 8px; background: var(--accent-soft);
}
.key-panel pre {
overflow-x: auto; background: var(--surface); border: 1px solid var(--border);
border-radius: 6px; padding: 8px; font-size: 12px;
}
.key-url code { word-break: break-all; }
+13
View File
@@ -59,6 +59,17 @@
<svg viewBox="0 0 24 24" aria-hidden="true"><path d="M6 16V11a6 6 0 0 1 12 0v5l1.5 2h-15z"/><path d="M10 20.5a2 2 0 0 0 4 0"/></svg>
<span class="nav-label">Alerts</span>
</a>
<a class="nav-link" href="/team" data-section="team">
<svg viewBox="0 0 24 24" aria-hidden="true"><circle cx="9" cy="8" r="3"/><circle cx="17" cy="9" r="2.5"/><path d="M3 19a6 6 0 0 1 12 0M15 19a5 5 0 0 1 6-4"/></svg>
<span class="nav-label">Team</span>
</a>
<!-- Hidden unless the signed-in user is a system administrator; app.js
unhides it once /api/me says so. The server refuses every admin
endpoint regardless, so this is a courtesy and not a gate. -->
<a class="nav-link" href="/admin" data-section="admin" id="nav-admin" hidden>
<svg viewBox="0 0 24 24" aria-hidden="true"><path d="M12 3l7 3v6c0 4-3 7-7 9-4-2-7-5-7-9V6z"/></svg>
<span class="nav-label">Admin</span>
</a>
<a class="nav-link" href="/more" data-section="more">
<svg viewBox="0 0 24 24" aria-hidden="true"><circle cx="12" cy="8" r="3.5"/><path d="M5 20a7 7 0 0 1 14 0"/></svg>
<span class="nav-label">Account</span>
@@ -80,6 +91,8 @@
<section id="view-oncall" class="view view-page" data-view="oncall" hidden></section>
<section id="view-alerts" class="view view-page" data-view="alerts" hidden></section>
<section id="view-team" class="view view-page" data-view="team" hidden></section>
<section id="view-admin" class="view view-page" data-view="admin" hidden></section>
<section id="view-more" class="view view-page" data-view="more" hidden></section>
</div>
+308
View File
@@ -0,0 +1,308 @@
// Administration: the teams on this server, the people who can sign in, and
// the settings that change how the server behaves.
//
// Only rendered for a system administrator. The server enforces that on every
// endpoint regardless — hiding a section is a courtesy to the reader, not a
// permission — so this view simply says so rather than pretending to be a
// gate.
import * as api from './api.js';
import { h, clear, spinner, confirm } from './ui.js';
import { state, myID } from './state.js';
const view = () => document.getElementById('view-admin');
let data = null; // { teams, users, settings }
let error = null;
let busy = false;
export function show() {
if (!data) clear(view(), spinner());
refresh();
}
export async function refresh() {
if (!state.me?.user?.is_admin) {
data = null;
render();
return;
}
try {
const [teams, users, settings] = await Promise.all([
api.adminTeams(),
api.users(),
api.adminSettings(),
]);
data = { teams, users, settings };
error = null;
} catch (err) {
error = err.message;
}
render();
}
function render() {
if (!state.me?.user?.is_admin) {
clear(view(), h('div', { class: 'card' },
h('p', { class: 'muted', text: 'Administration is for system administrators. Ask one for access.' })));
return;
}
if (!data) {
clear(view(), error ? h('div', { class: 'load-error', text: error }) : spinner());
return;
}
clear(view(),
error && h('div', { class: 'load-error', text: `Showing older data: ${error}` }),
teamsCard(),
usersCard(),
settingsCard(),
);
}
// --- teams -----------------------------------------------------------------
function teamsCard() {
const rows = data.teams.map((t) =>
h('tr', {},
h('td', {}, h('strong', { text: t.name })),
h('td', { class: 'num', text: String(t.members) }),
h('td', { class: 'num', text: String(t.open_incidents) }),
h('td', {},
h('button', {
class: 'btn-sm',
type: 'button',
text: 'Rename',
onclick: () => renameTeam(t),
}),
// A team with open incidents cannot be deleted, and saying so before
// the click is kinder than a 409 afterwards.
h('button', {
class: 'btn-sm danger',
type: 'button',
text: 'Delete',
disabled: t.open_incidents > 0,
title: t.open_incidents > 0 ? 'Resolve its open incidents first' : '',
onclick: () => deleteTeam(t),
}),
),
));
return h('div', { class: 'card' },
h('h2', { text: 'Teams' }),
h('table', { class: 'admin-table' },
h('thead', {}, h('tr', {},
h('th', { text: 'Name' }),
h('th', { class: 'num', text: 'Members' }),
h('th', { class: 'num', text: 'Open' }),
h('th', { text: '' }))),
h('tbody', {}, rows)),
newTeamForm(),
);
}
function newTeamForm() {
const name = h('input', { name: 'name', type: 'text', placeholder: 'New team name', required: true });
const form = h('form', { class: 'inline-form' }, name,
h('button', { class: 'btn', type: 'submit', text: 'Create' }));
form.addEventListener('submit', async (e) => {
e.preventDefault();
if (busy) return;
busy = true;
try {
await api.createTeam(name.value.trim());
name.value = '';
await refresh();
} catch (err) {
error = err.message;
render();
} finally {
busy = false;
}
});
return form;
}
async function renameTeam(team) {
const next = window.prompt(`Rename ${team.name} to:`, team.name);
if (!next || next === team.name) return;
try {
await api.renameTeam(team.id, next);
} catch (err) {
error = err.message;
}
refresh();
}
async function deleteTeam(team) {
if (!(await confirm({
title: `Delete ${team.name}?`,
text: 'Its alerts, incidents, schedule and integrations go with it. This cannot be undone.',
confirmLabel: 'Delete',
danger: true,
}))) return;
try {
await api.deleteTeam(team.id);
} catch (err) {
error = err.message;
}
refresh();
}
// --- users -----------------------------------------------------------------
function usersCard() {
const rows = data.users.map((u) => {
const self = u.id === myID();
return h('tr', { class: u.disabled_at ? 'disabled-row' : '' },
h('td', {},
h('strong', { text: u.username }),
u.disabled_at && h('span', { class: 'row-team', text: 'disabled' }),
self && h('span', { class: 'you', text: 'you' })),
h('td', { class: 'muted', text: u.email }),
h('td', {}, u.is_admin ? h('span', { class: 'row-team', text: 'admin' }) : null),
h('td', {},
// Neither action is offered for your own account: the server refuses
// both, and an enabled-looking button that always fails is worse than
// no button.
!self && h('button', {
class: 'btn-sm',
type: 'button',
text: u.is_admin ? 'Revoke admin' : 'Make admin',
onclick: () => setAdmin(u, !u.is_admin),
}),
!self && h('button', {
class: 'btn-sm danger',
type: 'button',
text: u.disabled_at ? 'Enable' : 'Disable',
onclick: () => setDisabled(u, !u.disabled_at),
}),
),
);
});
return h('div', { class: 'card' },
h('h2', { text: 'Users' }),
h('p', { class: 'muted small' },
'Disabling an account stops it signing in and stops its API keys, and keeps ',
'its acknowledgements and timeline entries. Deleting a user erases those.'),
h('table', { class: 'admin-table' },
h('thead', {}, h('tr', {},
h('th', { text: 'User' }),
h('th', { text: 'Email' }),
h('th', { text: '' }),
h('th', { text: '' }))),
h('tbody', {}, rows)),
);
}
async function setAdmin(user, next) {
if (next && !(await confirm({
title: `Make ${user.username} an administrator?`,
text: 'They will be able to create and delete users, and grant this to others.',
confirmLabel: 'Make admin',
}))) return;
try {
await api.setUserAdmin(user.id, next);
} catch (err) {
error = err.message;
}
refresh();
}
async function setDisabled(user, next) {
if (next && !(await confirm({
title: `Disable ${user.username}?`,
text: 'They cannot sign in and their API keys stop working. Their history stays.',
confirmLabel: 'Disable',
danger: true,
}))) return;
try {
await api.setUserDisabled(user.id, next);
} catch (err) {
error = err.message;
}
refresh();
}
// --- settings --------------------------------------------------------------
// Seconds are what the API speaks; people think in minutes and hours. The two
// are converted here rather than in the server, which should keep exactly one
// unit.
const UNITS = [
{ label: 'minutes', seconds: 60 },
{ label: 'hours', seconds: 3600 },
{ label: 'days', seconds: 86400 },
];
function bestUnit(seconds) {
for (const u of [...UNITS].reverse()) {
if (seconds > 0 && seconds % u.seconds === 0) return u;
}
return UNITS[0];
}
function settingsCard() {
const editable = data.settings.editable || {};
const inputs = new Map();
const rows = Object.entries(editable).map(([key, s]) => {
const unit = bestUnit(s.seconds);
const value = h('input', {
type: 'number',
min: '0',
value: String(Math.round(s.seconds / unit.seconds)),
class: 'setting-value',
});
const select = h('select', { class: 'setting-unit' },
...UNITS.map((u) => h('option', {
value: String(u.seconds),
text: u.label,
selected: u.seconds === unit.seconds,
})));
inputs.set(key, () => Number(value.value) * Number(select.value));
return h('tr', {},
h('td', {}, h('strong', { text: key.replace(/_seconds$/, '').replace(/_/g, ' ') })),
h('td', { class: 'muted small', text: s.description }),
h('td', {}, value, select),
);
});
const form = h('form', { class: 'admin-settings' },
h('table', { class: 'admin-table' }, h('tbody', {}, rows)),
h('button', { class: 'btn', type: 'submit', text: 'Save settings' }));
form.addEventListener('submit', async (e) => {
e.preventDefault();
if (busy) return;
busy = true;
const body = {};
for (const [key, read] of inputs) body[key] = read();
try {
await api.setAdminSettings(body);
await refresh();
} catch (err) {
error = err.message;
render();
} finally {
busy = false;
}
});
const env = Object.entries(data.settings.from_env || {}).map(([k, v]) =>
h('tr', {},
h('td', {}, h('code', { text: k })),
h('td', { class: 'muted', text: v === '' ? '(unset)' : v })));
return h('div', { class: 'card' },
h('h2', { text: 'Settings' }),
h('p', { class: 'muted small', text: 'Saved changes take effect on the next sweep — no restart.' }),
form,
h('h3', { text: 'From the environment' }),
h('p', { class: 'muted small' },
'Where the server is plugged in, rather than how it behaves. These are set ',
'in the deployment and are read-only here. Credentials are never shown.'),
h('table', { class: 'admin-table' }, h('tbody', {}, env)),
);
}
+45 -9
View File
@@ -87,12 +87,48 @@ export const deleteNote = (id, eventID) => call('DELETE', `/incidents/${id}/note
export const alerts = (query, opts) => call('GET', '/alerts', { query, ...opts });
// schedule
export const schedule = (from, to) => call('GET', '/schedule', { query: { from, to } });
export async function onCallNow() {
try {
return await call('GET', '/schedule/current');
} catch (err) {
if (err instanceof ApiError && err.status === 404) return null;
throw err;
}
}
export const teams = () => call('GET', '/teams');
export const createTeam = (name) => call('POST', '/teams', { body: { name } });
export const renameTeam = (id, name) => call('PUT', `/teams/${id}`, { body: { name } });
export const deleteTeam = (id) => call('DELETE', `/teams/${id}`);
// A team's own settings. Every write is owner-only and every read is
// member-only; the server answers 403 and 404 respectively, so the UI shows
// what the role allows rather than guarding it.
export const teamMembers = (id) => call('GET', `/teams/${id}/members`);
export const addTeamMember = (id, userID, role) =>
call('POST', `/teams/${id}/members`, { body: { user_id: userID, role } });
export const removeTeamMember = (id, userID) => call('DELETE', `/teams/${id}/members/${userID}`);
export const integrations = (id) => call('GET', `/teams/${id}/integrations`);
export const createIntegration = (id, name) =>
call('POST', `/teams/${id}/integrations`, { body: { name } });
export const deleteIntegration = (id, integrationID) =>
call('DELETE', `/teams/${id}/integrations/${integrationID}`);
export const deadman = (id) => call('GET', `/teams/${id}/deadman`);
export const setDeadman = (id, body) => call('PUT', `/teams/${id}/deadman`, { body });
export const escalation = (id) => call('GET', `/teams/${id}/escalation`);
export const setEscalation = (id, body) => call('PUT', `/teams/${id}/escalation`, { body });
export const assignSchedule = (id, userID, dates, replace = false) =>
call('POST', `/teams/${id}/schedule`, { body: { user_id: userID, dates, replace } });
export const unassignSchedule = (id, entryID) => call('DELETE', `/teams/${id}/schedule/${entryID}`);
// Administration. Every one of these is refused with 403 for anybody without
// the flag, so the UI hides the section rather than guarding it.
export const adminTeams = () => call('GET', '/admin/teams');
export const adminSettings = () => call('GET', '/admin/settings');
export const setAdminSettings = (body) => call('PUT', '/admin/settings', { body });
export const setUserAdmin = (id, isAdmin) =>
call('PUT', `/users/${id}/admin`, { body: { is_admin: isAdmin } });
export const setUserDisabled = (id, disabled) =>
call('PUT', `/users/${id}/disabled`, { body: { disabled } });
export const schedule = (teamID, from, to) =>
call('GET', `/teams/${teamID}/schedule`, { query: { from, to } });
// One entry per team the viewer belongs to, for the teams that have somebody
// scheduled today. An empty array means nobody anywhere, which is a real answer
// rather than an error — unlike the pre-teams endpoint, which 404ed.
export const onCallNow = () => call('GET', '/schedule/current');
+10 -2
View File
@@ -3,12 +3,14 @@
import * as api from './api.js';
import * as ui from './ui.js';
import * as poll from './poll.js';
import { state, reset } from './state.js';
import { state, reset, loadTeams } from './state.js';
import * as queue from './queue.js';
import * as incident from './incident.js';
import * as oncall from './oncall.js';
import * as alerts from './alerts.js';
import * as account from './account.js';
import * as team from './team.js';
import * as admin from './admin.js';
const $ = (id) => document.getElementById(id);
@@ -17,6 +19,8 @@ const SECTIONS = {
queue: { title: 'Queue', view: queue },
oncall: { title: 'On-call', view: oncall },
alerts: { title: 'Alerts', view: alerts },
team: { title: 'Team', view: team },
admin: { title: 'Admin', view: admin },
more: { title: 'Account', view: account },
};
@@ -24,7 +28,7 @@ function parseRoute(pathname) {
const m = pathname.match(/^\/incidents\/(\d+)\/?$/);
if (m) return { section: 'queue', incident: Number(m[1]) };
const name = pathname.replace(/^\/|\/$/g, '');
if (name === 'oncall' || name === 'alerts' || name === 'more') return { section: name };
if (name === 'oncall' || name === 'alerts' || name === 'team' || name === 'admin' || name === 'more') return { section: name };
return { section: 'queue', incident: null };
}
@@ -141,6 +145,10 @@ async function boot() {
try {
state.me = await api.me();
await loadTeams();
// The Admin tab exists only for an administrator. Somebody who types /admin
// anyway gets the view's own "ask an administrator" card, not a blank page.
$('nav-admin').hidden = !state.me?.user?.is_admin;
showApp();
} catch (err) {
if (err.status === 401) showLogin();
+13
View File
@@ -95,6 +95,15 @@ function statusBadges() {
if (inc.status !== 'resolved' && isFuture(inc.snoozed_until)) {
out.push(badge(`Snoozed · ${until(inc.snoozed_until)} left`, 'st-snoozed'));
}
// Where it is on the ladder, while it is still climbing. The queue shows
// what happened; this says what happens next, which is the question somebody
// looking at an unacknowledged incident actually has.
if (inc.escalation_level > 0) {
const left = inc.escalation_due_at && isFuture(inc.escalation_due_at)
? ` · next in ${until(inc.escalation_due_at)}`
: ' · next page due';
out.push(badge(`Escalating · level ${inc.escalation_level}${left}`, 'st-triggered'));
}
if (inc.archived_at) out.push(badge('Archived', 'plain'));
return out;
}
@@ -115,6 +124,10 @@ function facts() {
if (inc.status !== 'resolved' && isFuture(inc.snoozed_until)) {
add('Snoozed until', when(inc.snoozed_until));
}
if (inc.escalation_level > 0 && inc.escalation_due_at) {
add('Escalates next', when(inc.escalation_due_at),
h('span', { class: 'sub', text: ` · level ${inc.escalation_level}` }));
}
if (inc.resolved_at) {
const how = inc.resolution_source === 'manual' ? 'by hand' : 'alerts stopped firing';
add('Resolved', when(inc.resolved_at), h('span', { class: 'sub', text: ` · ${how}` }));
+39 -11
View File
@@ -1,10 +1,14 @@
// On-call: who is on duty now, the week around it, and your own next shifts.
// Read-only for now; the TUI edits the schedule.
//
// One team's rota at a time — the viewer's first team, since a viewer in one
// team has nothing to choose between. "On call now" is the exception and shows
// every team the viewer is in, because somebody on two rotas wants both.
import * as api from './api.js';
import { h, clear, icon, spinner } from './ui.js';
import { isoDate, mondayOf, addDays, isoWeek, initial } from './format.js';
import { myID } from './state.js';
import { myID, currentTeam } from './state.js';
const view = () => document.getElementById('view-oncall');
@@ -24,10 +28,17 @@ export async function refresh() {
const start = weekStart;
const today = new Date();
try {
const team = currentTeam();
if (!team) {
data = { now: [], week: [], upcoming: [] };
error = null;
render();
return;
}
const [now, week, upcoming] = await Promise.all([
api.onCallNow(),
api.schedule(isoDate(start), isoDate(addDays(start, 6))),
api.schedule(isoDate(today), isoDate(addDays(today, 60))),
api.schedule(team.id, isoDate(start), isoDate(addDays(start, 6))),
api.schedule(team.id, isoDate(today), isoDate(addDays(today, 60))),
]);
if (start !== weekStart) return;
data = { now, week, upcoming };
@@ -60,15 +71,32 @@ function you(userID) {
return userID === myID() ? h('span', { class: 'you', text: 'you' }) : null;
}
// One card per team with somebody on call, and a single empty card when there
// is nobody anywhere. The team's name is shown only when the viewer is in more
// than one, so the common case reads exactly as it did before teams existed.
function nowCard() {
const n = data.now;
return h('div', { class: 'card now-card' },
h('div', { class: `avatar ${n ? '' : 'none'}`, text: n ? initial(n.username) : '–' }),
h('div', {},
h('div', { class: 'now-label', text: 'On call now' }),
h('div', { class: 'now-name' }, n ? n.username : 'Nobody', n && you(n.user_id)),
),
);
const entries = data.now || [];
const showTeam = entries.length > 1;
if (entries.length === 0) {
return h('div', { class: 'card now-card' },
h('div', { class: 'avatar none', text: '–' }),
h('div', {},
h('div', { class: 'now-label', text: 'On call now' }),
h('div', { class: 'now-name', text: 'Nobody' }),
),
);
}
return h('div', {}, ...entries.map((n) =>
h('div', { class: 'card now-card' },
h('div', { class: 'avatar', text: initial(n.username) }),
h('div', {},
h('div', {
class: 'now-label',
text: showTeam ? `On call now · ${n.team_name}` : 'On call now',
}),
h('div', { class: 'now-name' }, n.username, you(n.user_id)),
),
)));
}
function weekCard() {
+62 -3
View File
@@ -26,12 +26,32 @@ const EMPTY = {
};
let filter = loadFilter();
let teamFilter = loadTeamFilter(); // '' for every team the viewer is in
let items = null; // null while loading
let error = null;
let selected = null;
let cursor = -1; // keyboard position in the list
let built = false;
function loadTeamFilter() {
try {
return sessionStorage.getItem('terdut.queue.team') || '';
} catch {
return '';
}
}
function setTeamFilter(id) {
teamFilter = id;
try {
sessionStorage.setItem('terdut.queue.team', id);
} catch {
/* storage unavailable */
}
renderChips();
refresh({ fresh: true });
}
function loadFilter() {
try {
const f = sessionStorage.getItem('terdut.queue.filter');
@@ -64,7 +84,11 @@ export async function refresh({ fresh = false } = {}) {
const requested = filter;
try {
// The open list is already fetched for the badges; no need to ask twice.
const result = filter === 'open' && !fresh ? state.open : await api.incidents(f.query);
// The cached open queue covers every team, so it can only be reused when
// no team filter is applied.
const query = teamFilter ? { ...f.query, team_id: teamFilter } : f.query;
const cached = filter === 'open' && !fresh && !teamFilter;
const result = cached ? state.open : await api.incidents(query);
if (requested !== filter) return;
items = result;
error = null;
@@ -88,7 +112,7 @@ function setFilter(id) {
function renderChips() {
const el = document.getElementById('queue-filters');
clear(el, FILTERS.map((f) =>
const chips = FILTERS.map((f) =>
h('button', {
class: 'chip',
type: 'button',
@@ -97,7 +121,34 @@ function renderChips() {
onclick: () => setFilter(f.id),
text: f.label,
}),
));
);
// Somebody in one team has nothing to choose between, so the row of team
// chips appears only when there is more than one. The default is all of
// them: the combined queue is the point.
if (state.teams.length > 1) {
chips.push(h('span', { class: 'chip-sep' }));
chips.push(h('button', {
class: 'chip',
type: 'button',
role: 'tab',
'aria-selected': String(teamFilter === ''),
onclick: () => setTeamFilter(''),
text: 'All teams',
}));
for (const team of state.teams) {
chips.push(h('button', {
class: 'chip',
type: 'button',
role: 'tab',
'aria-selected': String(teamFilter === String(team.id)),
onclick: () => setTeamFilter(String(team.id)),
text: team.name,
}));
}
}
clear(el, chips);
}
function renderList() {
@@ -142,6 +193,13 @@ function row(inc, index) {
// The server already puts the group labels in the title; show only the rest.
const labels = labelSummary(Object.fromEntries(
Object.entries(inc.group_labels || {}).filter(([k, v]) => !inc.title.includes(`${k}=${v}`))));
// The team is shown only to somebody who is in more than one. For everybody
// else it is the same word on every row, which is noise rather than
// information.
const team = state.teams.length > 1 && inc.team_name
? h('span', { class: 'row-team', text: inc.team_name })
: null;
return h('a', {
class: `row ${severityClass(inc.severity)} ${resolved ? 'resolved' : ''} ${index === cursor ? 'kbd-focus' : ''}`,
href: `/incidents/${inc.id}`,
@@ -153,6 +211,7 @@ function row(inc, index) {
h('div', { class: 'row-meta' },
status,
assignee,
team,
labels && h('span', { class: 'labels', text: labels }),
),
);
+13
View File
@@ -6,8 +6,15 @@ import * as api from './api.js';
export const state = {
me: null, // { user, has_password }
open: [], // the default queue: open, not snoozed
teams: [], // the teams the viewer belongs to, each with their role
};
// The team whose schedule and settings the views act on. A viewer in one team —
// which is everybody until somebody makes a second — never has to choose.
export function currentTeam() {
return state.teams[0] || null;
}
export function myID() {
return state.me ? state.me.user.id : null;
}
@@ -24,8 +31,14 @@ export async function users() {
return usersCache;
}
export async function loadTeams() {
state.teams = await api.teams();
return state.teams;
}
export function reset() {
state.me = null;
state.open = [];
state.teams = [];
usersCache = null;
}
+453
View File
@@ -0,0 +1,453 @@
// Team settings: the rota, who is in the team, where its alerts come from,
// what it escalates through, and which of its alerts are heartbeats.
//
// Everything here was API-only until now, which meant a team owner had to use
// curl to set up escalation — the feature this whole line of work exists for.
//
// The server decides what a role may do: an owner's edits succeed, a member's
// are refused with 403, and a non-member gets 404 for the lot. This view hides
// the controls a member cannot use, because a form that always fails is worse
// than no form, but it is not the thing enforcing anything.
import * as api from './api.js';
import { h, clear, spinner, confirm } from './ui.js';
import { state, currentTeam, users as allUsers } from './state.js';
import { isoDate, addDays } from './format.js';
const view = () => document.getElementById('view-team');
let teamID = null;
let data = null; // { team, members, integrations, escalation, deadman, schedule, users }
let error = null;
let freshKey = null; // an integration key, shown once, until the view is left
export function show() {
if (!data) clear(view(), spinner());
refresh();
}
function selectedTeam() {
const teams = state.teams || [];
return teams.find((t) => t.id === teamID) || currentTeam();
}
export async function refresh() {
const team = selectedTeam();
if (!team) {
data = null;
render();
return;
}
teamID = team.id;
try {
// A member may read all of this; only the writes are owner-only.
const [members, integrations, escalation, deadman, schedule, users] = await Promise.all([
api.teamMembers(team.id),
api.integrations(team.id),
api.escalation(team.id),
api.deadman(team.id),
api.schedule(team.id, isoDate(new Date()), isoDate(addDays(new Date(), 30))),
allUsers(),
]);
data = { team, members, integrations, escalation, deadman, schedule, users };
error = null;
} catch (err) {
error = err.message;
}
render();
}
function isOwner() {
return data?.team?.role === 'owner' || state.me?.user?.is_admin;
}
function render() {
if (!data) {
clear(view(), error
? h('div', { class: 'load-error', text: error })
: h('div', { class: 'card' }, h('p', { class: 'muted', text: 'You are not in a team yet.' })));
return;
}
clear(view(),
error && h('div', { class: 'load-error', text: `Showing older data: ${error}` }),
teamPicker(),
!isOwner() && h('div', { class: 'card' },
h('p', { class: 'muted small', text: 'You are a member of this team. Only an owner can change its settings.' })),
scheduleCard(),
escalationCard(),
integrationsCard(),
deadmanCard(),
membersCard(),
);
}
// Only shown to somebody in more than one team, like the queue's filter chips.
function teamPicker() {
if ((state.teams || []).length < 2) {
return h('div', { class: 'card' }, h('h2', { text: data.team.name }));
}
const select = h('select', { class: 'team-picker' },
...state.teams.map((t) => h('option', {
value: String(t.id), text: t.name, selected: t.id === teamID,
})));
select.addEventListener('change', () => {
teamID = Number(select.value);
data = null;
freshKey = null;
show();
});
return h('div', { class: 'card' }, h('h2', { text: 'Team' }), select);
}
// --- schedule --------------------------------------------------------------
// The rota is one person per UTC day. The on-call page shows it; this is where
// it is set, which until now was the TUI's job and the TUI cannot do it any
// more.
function scheduleCard() {
const rows = (data.schedule || []).map((e) =>
h('tr', {},
h('td', { text: e.date }),
h('td', {}, h('strong', { text: e.username })),
h('td', {}, isOwner() && h('button', {
class: 'btn-sm danger', type: 'button', text: 'Clear',
onclick: () => act(() => api.unassignSchedule(teamID, e.id)),
})),
));
return h('div', { class: 'card' },
h('h2', { text: 'On-call rota' }),
h('p', { class: 'muted small', text: 'One person per UTC day, for the next 30 days.' }),
rows.length
? h('table', { class: 'admin-table' }, h('tbody', {}, rows))
: h('p', { class: 'muted', text: 'Nobody is scheduled.' }),
isOwner() && assignForm(),
);
}
function assignForm() {
const who = memberSelect();
const from = h('input', { type: 'date', required: true, value: isoDate(new Date()) });
const days = h('input', { type: 'number', min: '1', max: '31', value: '1', class: 'setting-value' });
const replace = h('input', { type: 'checkbox' });
const form = h('form', { class: 'stacked-form' },
h('label', {}, 'Who ', who),
h('label', {}, 'From ', from),
h('label', {}, 'Days ', days),
// Taking a day somebody else holds has to be asked for, the same rule the
// API enforces: a plain assignment that silently moved a shift would move
// who gets paged without telling either of them.
h('label', { class: 'checkbox' }, replace, ' Take days somebody else holds'),
h('button', { class: 'btn', type: 'submit', text: 'Assign' }));
form.addEventListener('submit', (e) => {
e.preventDefault();
const start = new Date(from.value + 'T00:00:00Z');
const dates = [];
for (let i = 0; i < Number(days.value || 1); i++) dates.push(isoDate(addDays(start, i)));
act(() => api.assignSchedule(teamID, Number(who.value), dates, replace.checked));
});
return form;
}
function memberSelect(selected) {
return h('select', {},
...(data.members || []).map((m) => h('option', {
value: String(m.user_id), text: m.username, selected: m.user_id === selected,
})));
}
// --- escalation ------------------------------------------------------------
// The ladder is edited as a whole and sent as a whole, because the API replaces
// it wholesale: the levels are an order, and patching one rung would leave the
// numbering of the others undecided.
let draft = null;
function escalationCard() {
const esc = data.escalation;
if (!draft) {
draft = {
repeat_count: esc.repeat_count || 0,
fallback_topic: esc.fallback_topic || '',
levels: (esc.levels || []).map((l) => ({
timeout_seconds: l.timeout_seconds,
targets: (l.targets || []).map((t) => ({ kind: t.kind, user_id: t.user_id })),
})),
};
}
const body = [];
if (!draft.levels.length) {
body.push(h('p', { class: 'muted' },
'No ladder. An unacknowledged incident re-pages the same person every ',
'reminder interval and nobody else is woken.'));
}
draft.levels.forEach((level, i) => {
body.push(h('div', { class: 'ladder-level' },
h('div', { class: 'ladder-head' },
h('strong', { text: `Level ${i + 1}` }),
isOwner() && h('button', {
class: 'btn-sm danger', type: 'button', text: 'Remove',
onclick: () => { draft.levels.splice(i, 1); render(); },
})),
h('label', {}, 'Wait ', minutesInput(level.timeout_seconds, (secs) => {
level.timeout_seconds = secs;
}), ' before the next level'),
h('div', { class: 'ladder-targets' },
...level.targets.map((t, ti) => targetRow(level, t, ti)),
isOwner() && h('button', {
class: 'btn-sm', type: 'button', text: '+ target',
onclick: () => { level.targets.push({ kind: 'oncall' }); render(); },
})),
));
});
if (isOwner()) {
body.push(h('button', {
class: 'btn-sm', type: 'button', text: '+ level',
onclick: () => {
draft.levels.push({ timeout_seconds: 300, targets: [{ kind: 'oncall' }] });
render();
},
}));
const repeat = h('input', {
type: 'number', min: '0', max: '10', class: 'setting-value',
value: String(draft.repeat_count),
oninput: (e) => { draft.repeat_count = Number(e.target.value); },
});
const fallback = h('input', {
type: 'text', value: draft.fallback_topic, placeholder: 'terdut-oncall-all',
oninput: (e) => { draft.fallback_topic = e.target.value; },
});
body.push(h('label', {}, 'Repeat the whole ladder ', repeat, ' more times'));
body.push(h('label', {}, 'Then page this ntfy topic once ', fallback));
body.push(h('button', {
class: 'btn', type: 'button', text: 'Save ladder',
onclick: () => act(() => api.setEscalation(teamID, draft), { resetDraft: true }),
}));
}
return h('div', { class: 'card' },
h('h2', { text: 'Escalation' }),
h('p', { class: 'muted small' },
'When a level’s wait passes and nobody has acknowledged, the next level is ',
'paged. Acknowledging or resolving stops it; snoozing pauses it.'),
...body,
);
}
function targetRow(level, target, index) {
const kind = h('select', {},
h('option', { value: 'oncall', text: 'Whoever is on call', selected: target.kind === 'oncall' }),
h('option', { value: 'user', text: 'A specific person', selected: target.kind === 'user' }));
kind.addEventListener('change', () => {
target.kind = kind.value;
target.user_id = kind.value === 'user' ? (data.members[0] || {}).user_id : undefined;
render();
});
const who = target.kind === 'user'
? memberSelect(target.user_id)
: null;
if (who) {
who.addEventListener('change', () => { target.user_id = Number(who.value); });
}
return h('div', { class: 'target-row' }, kind, who,
isOwner() && h('button', {
class: 'btn-sm danger', type: 'button', text: '×',
title: 'Remove this target',
onclick: () => { level.targets.splice(index, 1); render(); },
}));
}
function minutesInput(seconds, onChange) {
const input = h('input', {
type: 'number', min: '1', class: 'setting-value',
value: String(Math.max(1, Math.round(seconds / 60))),
oninput: (e) => onChange(Number(e.target.value) * 60),
});
return h('span', {}, input, ' minutes');
}
// --- integrations ----------------------------------------------------------
function integrationsCard() {
const rows = (data.integrations || []).map((i) =>
h('tr', {},
h('td', {}, h('strong', { text: i.name })),
h('td', { class: 'muted small', text: i.kind }),
h('td', { class: 'muted small', text: i.last_used_at ? 'in use' : 'never used' }),
h('td', {}, isOwner() && h('button', {
class: 'btn-sm danger', type: 'button', text: 'Revoke',
onclick: async () => {
if (!(await confirm({
title: `Revoke ${i.name}?`,
text: 'Anything posting with this key stops delivering immediately.',
confirmLabel: 'Revoke',
danger: true,
}))) return;
act(() => api.deleteIntegration(teamID, i.id));
},
})),
));
return h('div', { class: 'card' },
h('h2', { text: 'Alert sources' }),
h('p', { class: 'muted small' },
'Alerts arrive on an integration key, which says both that the sender may ',
'post and which team the alerts belong to.'),
rows.length
? h('table', { class: 'admin-table' }, h('tbody', {}, rows))
: h('p', { class: 'muted', text: 'No alert source yet, so nothing can reach this team.' }),
freshKey && newKeyPanel(),
isOwner() && !freshKey && newIntegrationForm(),
);
}
// The key is returned exactly once. Say so, show it large, and give the
// Alertmanager snippet with it already in place — the next thing anybody does
// with it is paste it into a config.
function newKeyPanel() {
const url = freshKey.url || `${location.origin}/api/integrations/${freshKey.key}/alertmanager`;
const snippet = `receivers:
- name: terdut
webhook_configs:
- url: ${url}
send_resolved: true`;
return h('div', { class: 'key-panel' },
h('strong', { text: 'Copy this now — it is not shown again.' }),
h('pre', { class: 'key-url' }, h('code', { text: url })),
h('button', {
class: 'btn-sm', type: 'button', text: 'Copy URL',
onclick: () => navigator.clipboard?.writeText(url),
}),
h('p', { class: 'muted small', text: 'Alertmanager receiver:' }),
h('pre', {}, h('code', { text: snippet })),
h('button', {
class: 'btn-sm', type: 'button', text: 'Done',
onclick: () => { freshKey = null; render(); },
}),
);
}
function newIntegrationForm() {
const name = h('input', { type: 'text', placeholder: 'prod alertmanager', required: true });
const form = h('form', { class: 'inline-form' }, name,
h('button', { class: 'btn', type: 'submit', text: 'Add' }));
form.addEventListener('submit', async (e) => {
e.preventDefault();
try {
freshKey = await api.createIntegration(teamID, name.value.trim());
await refresh();
} catch (err) {
error = err.message;
render();
}
});
return form;
}
// --- dead man's switches ---------------------------------------------------
function deadmanCard() {
const d = data.deadman || {};
const matchers = h('input', {
type: 'text', value: d.matchers || '', placeholder: 'alertname=Watchdog',
class: 'wide',
});
const timeout = h('input', {
type: 'number', min: '0', class: 'setting-value',
value: String(Math.round((d.timeout_seconds || 0) / 60)),
});
const severity = h('select', {},
...['critical', 'error', 'warning', 'info'].map((s) =>
h('option', { value: s, text: s, selected: (d.severity || 'critical') === s })));
const form = h('form', { class: 'stacked-form' },
h('label', {}, 'Heartbeat alerts ', matchers),
h('label', {}, 'Declare dead after ', timeout, ' minutes of silence'),
h('label', {}, 'Open the incident at severity ', severity),
h('button', { class: 'btn', type: 'submit', text: 'Save switches' }));
form.addEventListener('submit', (e) => {
e.preventDefault();
act(() => api.setDeadman(teamID, {
matchers: matchers.value.trim(),
timeout_seconds: Number(timeout.value) * 60,
severity: severity.value,
}));
});
return h('div', { class: 'card' },
h('h2', { text: 'Dead man’s switches' }),
h('p', { class: 'muted small' },
'Alerts whose ABSENCE is the signal. Receiving one opens nothing; going ',
'quiet for longer than the timeout opens an incident. ',
h('code', { text: 'alertname=Watchdog,cluster=prod; alertname=EdgeHeartbeat' }),
' — semicolons separate switches, commas separate conditions, and every ',
'switch must name an alertname. Leave empty to watch nothing.'),
isOwner() ? form : h('p', { class: 'muted', text: d.matchers || 'Nothing watched.' }),
);
}
// --- members ---------------------------------------------------------------
function membersCard() {
const rows = (data.members || []).map((m) =>
h('tr', {},
h('td', {}, h('strong', { text: m.username })),
h('td', { class: 'muted small', text: m.role }),
h('td', {}, isOwner() && h('button', {
class: 'btn-sm', type: 'button',
text: m.role === 'owner' ? 'Make member' : 'Make owner',
onclick: () => act(() =>
api.addTeamMember(teamID, m.user_id, m.role === 'owner' ? 'member' : 'owner')),
}), isOwner() && h('button', {
class: 'btn-sm danger', type: 'button', text: 'Remove',
onclick: () => act(() => api.removeTeamMember(teamID, m.user_id)),
})),
));
const inTeam = new Set((data.members || []).map((m) => m.user_id));
const candidates = (data.users || []).filter((u) => !inTeam.has(u.id) && !u.disabled_at);
const pick = h('select', {},
...candidates.map((u) => h('option', { value: String(u.id), text: u.username })));
const role = h('select', {},
h('option', { value: 'member', text: 'member' }),
h('option', { value: 'owner', text: 'owner' }));
const form = h('form', { class: 'inline-form' }, pick, role,
h('button', { class: 'btn', type: 'submit', text: 'Add' }));
form.addEventListener('submit', (e) => {
e.preventDefault();
act(() => api.addTeamMember(teamID, Number(pick.value), role.value));
});
return h('div', { class: 'card' },
h('h2', { text: 'Members' }),
h('table', { class: 'admin-table' }, h('tbody', {}, rows)),
isOwner() && candidates.length > 0 && form,
);
}
// --- plumbing --------------------------------------------------------------
// act runs a write and reloads. Errors are shown rather than thrown away: a
// 409 from the last-owner guard or the schedule's conflict rule is the server
// explaining itself, and the reader needs to see it.
async function act(fn, { resetDraft = false } = {}) {
try {
await fn();
error = null;
if (resetDraft) draft = null;
} catch (err) {
error = err.message;
}
if (!resetDraft) draft = null;
await refresh();
}
-198
View File
@@ -1,198 +0,0 @@
//go:build migrate
// Command sqlite-to-postgres copies a terdut SQLite database into a freshly
// migrated Postgres one. It exists for exactly one upgrade — the one that moved
// this server off SQLite — and should be deleted once the installs that need it
// have run it. The modernc.org/sqlite dependency goes with it.
//
// Build-tagged so the dependency stays out of the server binary and out of a
// plain `go build ./...`:
//
// go run -tags migrate ./scripts/sqlite-to-postgres.go \
// -sqlite /path/to/terdut.db \
// -dsn 'postgres://terdut:secret@localhost:5432/terdut?sslmode=disable'
//
// The Postgres side must already have the schema: start the new server once
// against an empty database, let it migrate, stop it, then run this. The copy
// refuses to touch a database that already has rows, so a second run cannot
// double-insert.
//
// Ids are preserved, which is what keeps every foreign key — incident_alerts,
// incident_events, notifications, the ack tokens — pointing at the same rows it
// pointed at before. The identity sequences are moved past the copied ids at the
// end, so the first row the server writes afterwards does not collide.
package main
import (
"database/sql"
"flag"
"fmt"
"log"
"strings"
_ "github.com/jackc/pgx/v5/stdlib"
_ "modernc.org/sqlite"
)
// tables are copied parents first: every foreign key points at a table earlier
// in this list.
var tables = []struct {
name string
columns []string
// jsonb marks columns that were TEXT in SQLite and are jsonb in Postgres,
// so the insert can cast them.
jsonb []string
// sequence is the identity sequence to advance afterwards, empty when the
// table has no generated id.
sequence string
}{
{name: "users", columns: []string{"id", "username", "email", "created_at", "ntfy_topic", "password_hash"}, sequence: "users_id_seq"},
{name: "api_keys", columns: []string{"id", "user_id", "key_hash", "name", "created_at", "last_used_at"}, sequence: "api_keys_id_seq"},
{name: "sessions", columns: []string{"id", "token_hash", "user_id", "created_at", "last_seen_at", "expires_at", "user_agent"}, sequence: "sessions_id_seq"},
{name: "alerts", columns: []string{"id", "fingerprint", "name", "status", "labels", "annotations", "starts_at", "ends_at", "generator_url", "received_at", "archived_at", "resolution_source"}, jsonb: []string{"labels", "annotations"}, sequence: "alerts_id_seq"},
{name: "schedule_entries", columns: []string{"id", "user_id", "date", "created_at"}, sequence: "schedule_entries_id_seq"},
{name: "incidents", columns: []string{"id", "group_key", "title", "group_labels", "status", "severity", "triggered_at", "acknowledged_by", "acknowledged_at", "assigned_to", "snoozed_until", "resolved_at", "resolution_source", "archived_at"}, jsonb: []string{"group_labels"}, sequence: "incidents_id_seq"},
{name: "incident_alerts", columns: []string{"incident_id", "alert_id", "added_at"}},
{name: "incident_events", columns: []string{"id", "incident_id", "type", "user_id", "alert_id", "detail", "created_at"}, sequence: "incident_events_id_seq"},
{name: "notifications", columns: []string{"id", "incident_id", "user_id", "topic", "kind", "created_at", "send_after", "attempts", "sent_at", "last_error"}, sequence: "notifications_id_seq"},
{name: "incident_ack_tokens", columns: []string{"token_hash", "incident_id", "user_id", "created_at", "expires_at"}},
}
func main() {
sqlitePath := flag.String("sqlite", "", "path to the existing terdut SQLite database")
dsn := flag.String("dsn", "", "Postgres DSN of the migrated, empty database")
flag.Parse()
if *sqlitePath == "" || *dsn == "" {
log.Fatal("both -sqlite and -dsn are required")
}
src, err := sql.Open("sqlite", *sqlitePath)
if err != nil {
log.Fatalf("open sqlite: %v", err)
}
defer src.Close()
dst, err := sql.Open("pgx", *dsn)
if err != nil {
log.Fatalf("open postgres: %v", err)
}
defer dst.Close()
if err := dst.Ping(); err != nil {
log.Fatalf("ping postgres: %v", err)
}
if err := assertEmpty(dst); err != nil {
log.Fatalf("%v", err)
}
// One transaction for the whole copy: a run that dies half way leaves the
// target as it found it, rather than a partial database someone has to
// recognise as partial.
tx, err := dst.Begin()
if err != nil {
log.Fatalf("begin: %v", err)
}
defer tx.Rollback() //nolint:errcheck
for _, t := range tables {
n, err := copyTable(src, tx, t.name, t.columns, t.jsonb)
if err != nil {
log.Fatalf("copy %s: %v", t.name, err)
}
log.Printf("%-20s %d row(s)", t.name, n)
}
for _, t := range tables {
if t.sequence == "" {
continue
}
if err := advanceSequence(tx, t.sequence, t.name); err != nil {
log.Fatalf("advance %s: %v", t.sequence, err)
}
}
if err := tx.Commit(); err != nil {
log.Fatalf("commit: %v", err)
}
log.Print("done")
}
// assertEmpty refuses a target that already holds data, so running this twice
// cannot duplicate anything.
func assertEmpty(dst *sql.DB) error {
for _, t := range tables {
var n int64
if err := dst.QueryRow("SELECT COUNT(*) FROM " + t.name).Scan(&n); err != nil {
return fmt.Errorf("count %s (has the new server migrated this database?): %w", t.name, err)
}
if n > 0 {
return fmt.Errorf("%s already has %d row(s): the target must be empty", t.name, n)
}
}
return nil
}
func copyTable(src *sql.DB, tx *sql.Tx, table string, columns, jsonb []string) (int64, error) {
rows, err := src.Query("SELECT " + strings.Join(columns, ", ") + " FROM " + table)
if err != nil {
return 0, err
}
defer rows.Close()
insert := "INSERT INTO " + table + " (" + strings.Join(columns, ", ") + ") VALUES (" +
strings.Join(valuePlaceholders(columns, jsonb), ", ") + ")"
stmt, err := tx.Prepare(insert)
if err != nil {
return 0, err
}
defer stmt.Close()
var copied int64
for rows.Next() {
// Scanning into any lets the SQLite driver decide each column's Go type
// and hands it straight back to pgx, which is all this needs: the column
// types match on both sides, apart from the JSON casts above.
values := make([]any, len(columns))
targets := make([]any, len(columns))
for i := range values {
targets[i] = &values[i]
}
if err := rows.Scan(targets...); err != nil {
return copied, err
}
if _, err := stmt.Exec(values...); err != nil {
return copied, fmt.Errorf("insert row %d: %w", copied+1, err)
}
copied++
}
return copied, rows.Err()
}
// valuePlaceholders numbers the placeholders, casting the columns that became
// jsonb: pgx sends a Go string as text, and Postgres will not assign text to a
// jsonb column without being told.
func valuePlaceholders(columns, jsonb []string) []string {
isJSON := make(map[string]bool, len(jsonb))
for _, c := range jsonb {
isJSON[c] = true
}
out := make([]string, len(columns))
for i, c := range columns {
out[i] = fmt.Sprintf("$%d", i+1)
if isJSON[c] {
out[i] += "::jsonb"
}
}
return out
}
// advanceSequence puts an identity sequence past the largest copied id. Without
// it the first insert after the migration would reuse id 1.
func advanceSequence(tx *sql.Tx, sequence, table string) error {
_, err := tx.Exec(fmt.Sprintf(
`SELECT setval('%s', COALESCE((SELECT MAX(id) FROM %s), 0) + 1, false)`,
sequence, table))
return err
}