Page the next person when nobody answers
Closes #6, and closes the thing this whole line of work was opened for. Until now an unacknowledged incident re-paged the same topic every notify_repeat forever, which is a louder version of the same silence: if the person on call is asleep, out of signal or has left the company, nothing else happened. A team can now configure an ordered ladder. Each level has a timeout and a set of targets; a target is a named person or whoever the team's rota says is on call today. That second kind is the one that keeps working when the rota changes and nobody remembers to edit the policy. When a level's timeout passes with the incident still triggered, the next level is paged; off the end the chain repeats repeat_count times and then the team's fallback topic is paged once. The incident stays open throughout, because running out of people to wake is not somebody answering. Escalation rides the notifier's existing 30-second tick and its outbox rather than adding a second scheduler, and runs before delivery so a level that comes due on a tick is paged on that tick. Each target gets its own outbox row and therefore its own Acknowledge token: the button in a notification must acknowledge as the person holding the phone, not as whoever was paged first. Acknowledging or resolving takes the incident off the ladder. Snoozing pauses it -- a deliberate "not now" holds the ladder where it is and it resumes when the snooze runs out, rather than carrying on without the person who asked for quiet. Reminders and escalation never both run. A team with a ladder gets escalation; a team without keeps today's behaviour exactly. Both would mean two pages for one silence, which is how a tool gets muted. A level whose targets cannot be reached -- no topic, a disabled account, an empty rota -- is entered anyway, recorded as "nobody reachable", and the ladder moves on. Stalling on a rung that cannot ring would be the failure this feature exists to prevent, wearing the feature's clothes. A policy with such a level cannot be created, but an older row could hold one. The API replaces the ladder wholesale rather than patching a rung, because the levels are an order: editing one has to answer what happens to the numbering of the others, and a whole-ladder PUT makes that the client's decision and the edit atomic. Verified against a live server as well as in tests: alice paged, nobody answers, bob paged, nobody answers, the fallback topic paged once and the timeline reading "level 2: bob" then "escalation exhausted: paged terdut-oncall-all" -- and a second incident acknowledged before its timeout, which woke nobody else. No UI yet. The team-settings screens for escalation, integrations and dead man's switches are all still missing, and they are one piece of work rather than three. Claude-Session: https://claude.ai/code/session_01RHPj4ggeFdEjKKfm4SHbD7
This commit is contained in:
@@ -43,6 +43,10 @@ const (
|
||||
notifyTriggered = "triggered"
|
||||
notifyReminder = "reminder"
|
||||
notifyResolved = "resolved"
|
||||
|
||||
// notifyEscalated is a page that went out because nobody answered the last
|
||||
// one. Told apart from a reminder because it goes to somebody else.
|
||||
notifyEscalated = "escalated"
|
||||
)
|
||||
|
||||
// Timeline event types the notifier writes, so an incident's history says who
|
||||
@@ -120,6 +124,9 @@ func StartNotifier(ctx context.Context, db *sql.DB, cfg NotifyConfig) {
|
||||
// Exported so tests can drive a pass without waiting on the ticker.
|
||||
func NotifySweep(ctx context.Context, db *sql.DB, cfg NotifyConfig) {
|
||||
enqueueReminders(ctx, db, cfg)
|
||||
// Escalation before delivery, so a level that comes due on this tick is
|
||||
// paged on this tick rather than waiting for the next one.
|
||||
escalate(ctx, db, cfg)
|
||||
deliverPending(ctx, db, cfg)
|
||||
}
|
||||
|
||||
@@ -159,7 +166,12 @@ func enqueueReminders(ctx context.Context, db *sql.DB, cfg NotifyConfig) {
|
||||
AND i.resolved_at IS NULL
|
||||
AND i.archived_at IS NULL
|
||||
AND i.status = 'triggered'
|
||||
AND (i.snoozed_until IS NULL OR i.snoozed_until <= $2)`,
|
||||
AND (i.snoozed_until IS NULL OR i.snoozed_until <= $2)
|
||||
-- A team with an escalation ladder gets escalation instead. Both
|
||||
-- would mean two pages for one silence, which is how people learn to
|
||||
-- mute a tool.
|
||||
AND NOT EXISTS (
|
||||
SELECT 1 FROM escalation_levels el WHERE el.team_id = i.team_id)`,
|
||||
now.Add(-repeat).Unix(), now.Unix())
|
||||
if err != nil {
|
||||
log.Printf("notifier: find reminders: %v", err)
|
||||
|
||||
Reference in New Issue
Block a user