Escalation policies: page the next person when an incident is not acknowledged

niklas commented

2026-09-20 05:59:16 +00:00

Owner

Part of #1. Depends on teams (policies are per team and target team schedules).

Today

An incoming alert opens an incident assigned to that day's on-call user (currentOnCall, internal/api/incident_store.go:118).
internal/api/notifier.go ticks every 30s over a notifications outbox, with exponential backoff from 30s to 15m and at most 8 attempts.
enqueueReminders re-pages the same topic every TERDUT_NOTIFY_REPEAT (15m) while the incident is still triggered.
With nobody on call, the page goes to a fallback topic.
Nobody else is ever paged. That is the gap.

Scope

Model

escalation_policies (one per team, at least to start) and escalation_levels: position, timeout, targets.
A target is a specific user or "whoever is on call in schedule X".
Policy-level repeat_count: when the chain runs out, run it again N times, then page the team's fallback topic.
The fallback topic moves from the global TERDUT_NTFY_FALLBACK_TOPIC to a per-team setting.
Incident state for escalation: current level, when the level started, how many times the chain has repeated.

Behaviour

Build on the existing outbox and 30s tick — no second scheduler.
When a level's timeout passes and the incident is still triggered, advance to the next level and enqueue notifications for its targets.
Acknowledge or resolve stops escalation. Snooze pauses it, and un-snoozing resumes from the current level.
A target with no ntfy topic is skipped, and that is recorded.
incident_events gains escalated (with the level and who was paged), shown in the timeline UI.
The existing per-incident ack token (incident_ack_tokens, notify_ack.go) has to work for every escalation target, not just the first one paged.
Keep or retire the old reminder behaviour — a policy with one level and a repeat is the same thing, so decide explicitly.

UI

A team-owner page to edit the policy: ordered levels, targets, timeouts, repeat count.
The incident view shows the current level and when the next escalation is due.

Done when

An unacknowledged incident pages level 1, then level 2 after its timeout, stops the moment someone acknowledges, and the timeline shows each step. Covered by tests with a fake clock rather than by waiting.

Part of #1. Depends on teams (policies are per team and target team schedules). ### Today - An incoming alert opens an incident assigned to that day's on-call user (`currentOnCall`, `internal/api/incident_store.go:118`). - `internal/api/notifier.go` ticks every 30s over a `notifications` outbox, with exponential backoff from 30s to 15m and at most 8 attempts. - `enqueueReminders` re-pages **the same topic** every `TERDUT_NOTIFY_REPEAT` (15m) while the incident is still `triggered`. - With nobody on call, the page goes to a fallback topic. - Nobody else is ever paged. That is the gap. ### Scope **Model** - [ ] `escalation_policies` (one per team, at least to start) and `escalation_levels`: position, timeout, targets. - [ ] A target is a specific user or "whoever is on call in schedule X". - [ ] Policy-level `repeat_count`: when the chain runs out, run it again N times, then page the team's fallback topic. - [ ] The fallback topic moves from the global `TERDUT_NTFY_FALLBACK_TOPIC` to a per-team setting. - [ ] Incident state for escalation: current level, when the level started, how many times the chain has repeated. **Behaviour** - [ ] Build on the existing outbox and 30s tick — no second scheduler. - [ ] When a level's timeout passes and the incident is still `triggered`, advance to the next level and enqueue notifications for its targets. - [ ] Acknowledge or resolve stops escalation. Snooze pauses it, and un-snoozing resumes from the current level. - [ ] A target with no ntfy topic is skipped, and that is recorded. - [ ] `incident_events` gains `escalated` (with the level and who was paged), shown in the timeline UI. - [ ] The existing per-incident ack token (`incident_ack_tokens`, `notify_ack.go`) has to work for every escalation target, not just the first one paged. - [ ] Keep or retire the old reminder behaviour — a policy with one level and a repeat is the same thing, so decide explicitly. **UI** - [ ] A team-owner page to edit the policy: ordered levels, targets, timeouts, repeat count. - [ ] The incident view shows the current level and when the next escalation is due. ### Done when An unacknowledged incident pages level 1, then level 2 after its timeout, stops the moment someone acknowledges, and the timeline shows each step. Covered by tests with a fake clock rather than by waiting.