Record notification delivery on the incident timeline
Release / test (push) Failing after 8s
Release / build (amd64, darwin) (push) Has been skipped
Release / build (amd64, linux) (push) Has been skipped
Release / build (arm64, darwin) (push) Has been skipped
Release / build (arm64, linux) (push) Has been skipped
Release / docker (push) Has been skipped
Release / chart (push) Has been skipped
Release / release (push) Has been skipped
Release / test (push) Failing after 8s
Release / build (amd64, darwin) (push) Has been skipped
Release / build (amd64, linux) (push) Has been skipped
Release / build (arm64, darwin) (push) Has been skipped
Release / build (arm64, linux) (push) Has been skipped
Release / docker (push) Has been skipped
Release / chart (push) Has been skipped
Release / release (push) Has been skipped
An incident's history went quiet after "Incident opened": nothing said that anybody had been paged, reminded, or told it resolved. Delivery lived only in the notifications outbox, which no API exposes, so when a page failed to arrive there was nothing in the product that said whether it had been sent. The notifier now writes two event types. A notified event once ntfy accepts the publish, carrying the kind in detail and the paged user in user_id — absent when the page went to the shared fallback topic, which belongs to nobody. And a notify_failed event when a notification exhausts its retries, which is the one worth having: without it a page that never landed leaves the timeline identical to one that did. Both are written from the delivery result rather than at enqueue. A queued notification is an intention, and the timeline is append-only, so claiming somebody was told before ntfy accepted it would be a lie that stays there. A failed timeline write is logged rather than returned, so it cannot make a delivered row look unsent and send the page twice. The topic is deliberately in neither: it is a shared secret with the ntfy server, and every API key can read the timeline. No migration — incident_events.type is free text, unlike notifications.kind.
This commit is contained in:
@@ -235,6 +235,12 @@ Delivery is a queue, not an inline call: the webhook writes a row and a
|
||||
background notifier sends it within 30 seconds, retrying with exponential
|
||||
backoff up to 8 attempts. Nothing about ingestion blocks on ntfy being reachable.
|
||||
|
||||
Every delivery is recorded on the incident's timeline: a `notified` event once
|
||||
ntfy accepts the publish, and a `notify_failed` event when a notification
|
||||
exhausts its retries. Written from the result rather than at enqueue, so the
|
||||
timeline says what actually happened — and a page that never landed is visible
|
||||
instead of looking the same as one that did.
|
||||
|
||||
### Stale alert expiry
|
||||
|
||||
A resolved webhook is the only signal that an alert has stopped firing, so a
|
||||
@@ -353,8 +359,16 @@ name: degrade unknown values to "resolved, reason unknown".
|
||||
|
||||
Types written today: `triggered`, `alert_added`, `alert_resolved`,
|
||||
`acknowledged`, `unacknowledged`, `assigned`, `snoozed`, `unsnoozed`, `resolved`,
|
||||
`note`. On an `assigned` event `user_id` is the **assignee**, not the actor. New
|
||||
types may be added; render unknown ones generically rather than dropping them.
|
||||
`note`, `notified`, `notify_failed`. On an `assigned` event `user_id` is the
|
||||
**assignee**, not the actor. New types may be added; render unknown ones
|
||||
generically rather than dropping them.
|
||||
|
||||
On `notified` and `notify_failed`, `detail` carries the notification kind
|
||||
(`triggered` | `reminder` | `resolved`), and on a failure the reason after it.
|
||||
`user_id` is who was paged — absent means the page went to the shared fallback
|
||||
topic and so belongs to nobody. The topic itself is never written to the
|
||||
timeline: it is a shared secret with the ntfy server, and every API key can read
|
||||
this.
|
||||
|
||||
### Alerts
|
||||
|
||||
|
||||
Reference in New Issue
Block a user