Record notification delivery on the incident timeline
Release / test (push) Failing after 8s
Release / build (amd64, darwin) (push) Has been skipped
Release / build (amd64, linux) (push) Has been skipped
Release / build (arm64, darwin) (push) Has been skipped
Release / build (arm64, linux) (push) Has been skipped
Release / docker (push) Has been skipped
Release / chart (push) Has been skipped
Release / release (push) Has been skipped

An incident's history went quiet after "Incident opened": nothing said
that anybody had been paged, reminded, or told it resolved. Delivery
lived only in the notifications outbox, which no API exposes, so when a
page failed to arrive there was nothing in the product that said whether
it had been sent.

The notifier now writes two event types. A notified event once ntfy
accepts the publish, carrying the kind in detail and the paged user in
user_id — absent when the page went to the shared fallback topic, which
belongs to nobody. And a notify_failed event when a notification
exhausts its retries, which is the one worth having: without it a page
that never landed leaves the timeline identical to one that did.

Both are written from the delivery result rather than at enqueue. A
queued notification is an intention, and the timeline is append-only, so
claiming somebody was told before ntfy accepted it would be a lie that
stays there. A failed timeline write is logged rather than returned, so
it cannot make a delivered row look unsent and send the page twice.

The topic is deliberately in neither: it is a shared secret with the
ntfy server, and every API key can read the timeline.

No migration — incident_events.type is free text, unlike
notifications.kind.
This commit is contained in:
Niklas Ye
2026-08-07 13:31:03 +02:00
parent 17ee290d90
commit 4224dbe96c
3 changed files with 191 additions and 2 deletions
+16 -2
View File
@@ -235,6 +235,12 @@ Delivery is a queue, not an inline call: the webhook writes a row and a
background notifier sends it within 30 seconds, retrying with exponential
backoff up to 8 attempts. Nothing about ingestion blocks on ntfy being reachable.
Every delivery is recorded on the incident's timeline: a `notified` event once
ntfy accepts the publish, and a `notify_failed` event when a notification
exhausts its retries. Written from the result rather than at enqueue, so the
timeline says what actually happened — and a page that never landed is visible
instead of looking the same as one that did.
### Stale alert expiry
A resolved webhook is the only signal that an alert has stopped firing, so a
@@ -353,8 +359,16 @@ name: degrade unknown values to "resolved, reason unknown".
Types written today: `triggered`, `alert_added`, `alert_resolved`,
`acknowledged`, `unacknowledged`, `assigned`, `snoozed`, `unsnoozed`, `resolved`,
`note`. On an `assigned` event `user_id` is the **assignee**, not the actor. New
types may be added; render unknown ones generically rather than dropping them.
`note`, `notified`, `notify_failed`. On an `assigned` event `user_id` is the
**assignee**, not the actor. New types may be added; render unknown ones
generically rather than dropping them.
On `notified` and `notify_failed`, `detail` carries the notification kind
(`triggered` | `reminder` | `resolved`), and on a failure the reason after it.
`user_id` is who was paged — absent means the page went to the shared fallback
topic and so belongs to nobody. The topic itself is never written to the
timeline: it is a shared secret with the ntfy server, and every API key can read
this.
### Alerts