Page the next person when nobody answers #16

2026-09-20T16:38:11Z

niklas commented

2026-09-20 16:38:11 +00:00

Closes #6 — and closes the thing this whole line of work was opened for. Until now an unacknowledged incident re-paged the same topic every notify_repeat forever, which is a louder version of the same silence: if the person on call is asleep, out of signal, or has left, nothing else happened.

The ladder

Each level has a timeout and targets; a target is a named person or whoever the rota says is on call today — the kind that keeps working when the rota changes and nobody remembers to edit the policy.

level 1   5m    oncall            the rota gets first refusal
level 2   5m    user:bob          then a named second
                                  then repeat_count more rounds
                                  then the team's fallback topic, once

The incident stays open throughout: running out of people to wake is not somebody answering.

Decisions worth arguing with

Acknowledging or resolving stops it; snoozing pauses it. Continuing to wake people after "I have this" is how a tool teaches people to mute it. Snooze is a deliberate "not now", so the ladder holds and resumes when the snooze ends rather than carrying on without the person who asked for quiet.
Reminders and escalation never both run. A team with a ladder gets escalation; a team without keeps today's behaviour exactly. Two pages for one silence is the surest route to a muted tool.
An unreachable level is entered anyway, recorded as nobody reachable, and the ladder moves on. Stalling on a rung that cannot ring would be this feature's own failure mode wearing its clothes.
Each target gets its own outbox row, and so its own Acknowledge token. The button must acknowledge as the person holding the phone, not as whoever was paged first — the question I flagged before starting.
The API replaces the ladder wholesale. The levels are an order; patching one rung has to answer what happens to the numbering of the others.

It rides the notifier's existing 30-second tick and outbox rather than adding a second scheduler, and runs before delivery so a level due on a tick is paged on that tick.

Verified

make fmt lint test helm-lint green with -race against Postgres 17, plus eleven new tests. They back-date escalation_level_at the way the dead-man tests back-date received_at — the sweeper reads a stored timestamp, so moving the timestamp is moving the clock.

Also driven against a live server with a real ntfy stand-in: alice paged → nobody answers → bob paged → nobody answers → fallback topic paged once, with the timeline reading level 2: bob then escalation exhausted: paged terdut-oncall-all. Then a second incident acknowledged before its timeout, which woke nobody else.

Three of my own test bugs surfaced on the way and are worth noting because they were all "the test was wrong, not the code": reminders need a RepeatEvery in the config to run at all, snooze takes a duration string rather than minutes, and notifyTS already puts the admin on call.

Not in here

No UI. The team-settings screens for escalation, integrations and dead-man switches are all still missing, and they're one piece of work rather than three.

https://claude.ai/code/session_01RHPj4ggeFdEjKKfm4SHbD7

Closes #6 — and closes the thing this whole line of work was opened for. Until now an unacknowledged incident re-paged the same topic every `notify_repeat` forever, which is a louder version of the same silence: if the person on call is asleep, out of signal, or has left, nothing else happened. ### The ladder Each level has a timeout and targets; a target is a named person or **whoever the rota says is on call today** — the kind that keeps working when the rota changes and nobody remembers to edit the policy. ``` level 1 5m oncall the rota gets first refusal level 2 5m user:bob then a named second then repeat_count more rounds then the team's fallback topic, once ``` The incident stays open throughout: **running out of people to wake is not somebody answering.** ### Decisions worth arguing with - **Acknowledging or resolving stops it; snoozing pauses it.** Continuing to wake people after "I have this" is how a tool teaches people to mute it. Snooze is a deliberate "not now", so the ladder holds and resumes when the snooze ends rather than carrying on without the person who asked for quiet. - **Reminders and escalation never both run.** A team with a ladder gets escalation; a team without keeps today's behaviour exactly. Two pages for one silence is the surest route to a muted tool. - **An unreachable level is entered anyway**, recorded as `nobody reachable`, and the ladder moves on. Stalling on a rung that cannot ring would be this feature's own failure mode wearing its clothes. - **Each target gets its own outbox row, and so its own Acknowledge token.** The button must acknowledge as the person holding the phone, not as whoever was paged first — the question I flagged before starting. - **The API replaces the ladder wholesale.** The levels are an order; patching one rung has to answer what happens to the numbering of the others. It rides the notifier's existing 30-second tick and outbox rather than adding a second scheduler, and runs before delivery so a level due on a tick is paged on that tick. ### Verified `make fmt lint test helm-lint` green with `-race` against Postgres 17, plus eleven new tests. They back-date `escalation_level_at` the way the dead-man tests back-date `received_at` — the sweeper reads a stored timestamp, so moving the timestamp is moving the clock. **Also driven against a live server with a real ntfy stand-in:** alice paged → nobody answers → bob paged → nobody answers → fallback topic paged once, with the timeline reading `level 2: bob` then `escalation exhausted: paged terdut-oncall-all`. Then a second incident acknowledged before its timeout, which woke nobody else. Three of my own test bugs surfaced on the way and are worth noting because they were all "the test was wrong, not the code": reminders need a `RepeatEvery` in the config to run at all, snooze takes a duration string rather than minutes, and `notifyTS` already puts the admin on call. ### Not in here **No UI.** The team-settings screens for escalation, integrations and dead-man switches are all still missing, and they're one piece of work rather than three. https://claude.ai/code/session_01RHPj4ggeFdEjKKfm4SHbD7

niklas added 1 commit 2026-09-20 16:38:12 +00:00

Page the next person when nobody answers

CI / chart (pull_request) Successful in 1s

Details

CI / security (pull_request) Successful in 16s

Details

CI / test (pull_request) Successful in 2m21s

Details

3183e7e5c5

Closes #6, and closes the thing this whole line of work was opened for.
Until now an unacknowledged incident re-paged the same topic every
notify_repeat forever, which is a louder version of the same silence: if
the person on call is asleep, out of signal or has left the company,
nothing else happened.

A team can now configure an ordered ladder. Each level has a timeout and
a set of targets; a target is a named person or whoever the team's rota
says is on call today. That second kind is the one that keeps working
when the rota changes and nobody remembers to edit the policy. When a
level's timeout passes with the incident still triggered, the next level
is paged; off the end the chain repeats repeat_count times and then the
team's fallback topic is paged once. The incident stays open throughout,
because running out of people to wake is not somebody answering.

Escalation rides the notifier's existing 30-second tick and its outbox
rather than adding a second scheduler, and runs before delivery so a
level that comes due on a tick is paged on that tick. Each target gets
its own outbox row and therefore its own Acknowledge token: the button in
a notification must acknowledge as the person holding the phone, not as
whoever was paged first.

Acknowledging or resolving takes the incident off the ladder. Snoozing
pauses it -- a deliberate "not now" holds the ladder where it is and it
resumes when the snooze runs out, rather than carrying on without the
person who asked for quiet.

Reminders and escalation never both run. A team with a ladder gets
escalation; a team without keeps today's behaviour exactly. Both would
mean two pages for one silence, which is how a tool gets muted.

A level whose targets cannot be reached -- no topic, a disabled account,
an empty rota -- is entered anyway, recorded as "nobody reachable", and
the ladder moves on. Stalling on a rung that cannot ring would be the
failure this feature exists to prevent, wearing the feature's clothes. A
policy with such a level cannot be created, but an older row could hold
one.

The API replaces the ladder wholesale rather than patching a rung,
because the levels are an order: editing one has to answer what happens
to the numbering of the others, and a whole-ladder PUT makes that the
client's decision and the edit atomic.

Verified against a live server as well as in tests: alice paged, nobody
answers, bob paged, nobody answers, the fallback topic paged once and the
timeline reading "level 2: bob" then "escalation exhausted: paged
terdut-oncall-all" -- and a second incident acknowledged before its
timeout, which woke nobody else.

No UI yet. The team-settings screens for escalation, integrations and
dead man's switches are all still missing, and they are one piece of work
rather than three.

Claude-Session: https://claude.ai/code/session_01RHPj4ggeFdEjKKfm4SHbD7

niklas merged commit 4d62c1130b into main

2026-09-20 16:43:03 +00:00

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: niklas/terdut-server#16