niklas/terdut-server
1
0
Fork 0
Code Issues 2 Pull Requests Actions Packages Projects Releases 34 Wiki Activity

A dead man's switch incident cannot recover when its heartbeat moves to another team, and nothing says the switch is orphaned #22

New Issue
Open
opened 2026-09-25 09:55:35 +00:00 by niklas · 3 comments
niklas commented 2026-09-25 09:55:35 +00:00
Owner
Copy Link
Copy Source

Found in production on 2026-09-25 while checking why an incident had been open for four days. Nothing was changed; this records what was observed and read from the code. Deployed version: v0.20.1 (43f0044), chart 1.12.1.

Observed

Incident #9 ("No heartbeat from Watchdog", severity critical, team 1 "Default") opened on 2026-09-21 17:43:30Z and is still triggered on 2026-09-25, never acknowledged. In that time it has sent 1 triggered and 352 reminder notifications, all sent on the first attempt, every 15 minutes, to one topic. Meanwhile the Watchdog heartbeat is arriving normally: the alert was last received 2026-09-25 09:50Z.

The reason both are true is that the heartbeat moved to a different team:

integrations
  id 1  team 1 Default  "kube-prometheus-stack alertmanager"  created 09-20 13:43:02  last_used 09-21 17:17:40
  id 2  team 2 Ryuvia   "prod-alertmanager"                   created 09-21 16:56:19  last_used 09-25 (now)

alerts (name = Watchdog, same fingerprint 972f3dc166677cf2)
  id 1    team 1  status resolved  resolution_source deadman  ends_at 09-21 17:43:30  received_at 09-21 17:17:40
  id 920  team 2  status firing                                                       received_at 09-25 09:50:26

deadman_configs:  team 1  alertname=Watchdog 900 s critical   |   team 2  alertname=Watchdog 900 s critical
incidents:        id 9  team 1  group_key deadman:972f3dc166677cf2  triggered, resolved_at NULL

The sender's Alertmanager receiver was repointed from integration 1's key to integration 2's on 2026-09-21 (Ryuvia/charts 963f53e, 17:15Z). That is a deliberate change, not a fault. Team 1's switch then correctly declared the heartbeat dead, because no more heartbeats reached team 1.

Why it never recovers (from reading internal/api/deadman.go, not from running a test)

  • sweepDeadman runs per team. For team 2, row 920 is fresh and firing, so it calls deadmanRecovered, which selects ... WHERE team_id = $1 AND group_key = $2 AND resolved_at IS NULL with team 2's id. The open incident is in team 1, so it finds sql.ErrNoRows and returns. That is correct for isolated teams.
  • For team 1, row 1 is resolved, so it calls deadmanDied, which returns immediately because an incident is already open. Nothing ever sends team 1 another heartbeat, so deadmanRecovered is never called for it.

So the only way out of this state is a person resolving the incident. The code comment in deadmanRecovered says "recovery is the only automatic way out", and here recovery is structurally unreachable.

What is working as designed (so this is not filed as a reminder bug)

The 352 reminders are the intended behaviour. enqueueReminders stops on acknowledged, snoozed, resolved or archived, and its own comment says there is no reminder cap because snooze is the mute button. Reminders apply only to teams with no escalation ladder (NOT EXISTS (SELECT 1 FROM escalation_levels ...)). That is why team 2's four incidents on the same night got only a triggered and a resolved notification, and team 1's has been reminded ever since. None of that is wrong.

The only thing the reminders did here was make an orphaned switch loud. They did not say why it was silent.

The gap

Nothing in the system distinguishes "this heartbeat source died" from "this heartbeat source now reports somewhere else". Both look like a switch that stopped chirping. The switch's team keeps a permanent open incident, the operator's only signal is a reminder every 15 minutes, and the remedy (resolve the incident, remove the matcher from the old team's config) is not suggested anywhere.

Options, none chosen

Recorded so the next attempt does not start from scratch. No option has been tried.

  1. Surface it. Show, on the incident or the team's integrations list, that the integration that fed the switch has not been used since T. Cheapest, and changes no behaviour.
  2. Cross-team recovery by fingerprint. The code comment says "the unit of monitoring is the fingerprint". Recovering when the same fingerprint is alive in another team would fix this case, but it leaks state across teams that the rest of the design keeps isolated, and one team's heartbeat could then close another team's incident. Probably wrong, listed so it is decided rather than assumed.
  3. Expire an unrecovered switch incident when its integration has been unused longer than N. Needs a policy for N and for what "expire" means (resolve, or archive).
  4. Leave as is and treat the migration step "remove the old team's switch" as an operator task, documented.

A smaller observation from the same incident

The sweeper ticks every 15 minutes (sweepInterval = 15 * time.Minute in archiver.go) and the switch's timeout is 900 s. Detection therefore takes between 15 and 30 minutes after the last heartbeat. Here: last heartbeat 17:17:40, incident 17:43:30, that is 25 minutes 50 seconds ("last heartbeat 25m ago" in the event detail). The Timeout comment says it must be shorter than Alertmanager's repeat_interval; that comparison omits the sweep interval. One data point plus the arithmetic; not a defect by itself, but a configured "900 s" does not mean 900 s.

What I have not done

  • Not reproduced in a test. The cross-team scenario above comes from production data and reading the code, not from a failing test in deadman_test.go.
  • Not resolved incident #9 and not edited either team's deadman config. Both are the owner's data and were left alone.
  • Not read the notification topic or any key; the counts above come from aggregate queries.

Closing condition

Close when a decision is recorded here and verified: for whichever option is chosen, either a test that reproduces the two-team scenario and shows the intended outcome, or, if option 4, the documented step. Not on merge alone.

Refs Ryuvia/charts 963f53e (the receiver change), Ryuvia/talos#1 (unrelated; the night's incidents that led here).

Found in production on 2026-09-25 while checking why an incident had been open for four days. Nothing was changed; this records what was observed and read from the code. Deployed version: v0.20.1 (`43f0044`), chart 1.12.1. ## Observed Incident **#9** ("No heartbeat from Watchdog", severity critical, team 1 "Default") opened on 2026-09-21 17:43:30Z and is still `triggered` on 2026-09-25, never acknowledged. In that time it has sent **1 `triggered` and 352 `reminder` notifications**, all `sent` on the first attempt, every 15 minutes, to one topic. Meanwhile the Watchdog heartbeat is arriving normally: the alert was last received 2026-09-25 09:50Z. The reason both are true is that the heartbeat **moved to a different team**: ``` integrations id 1 team 1 Default "kube-prometheus-stack alertmanager" created 09-20 13:43:02 last_used 09-21 17:17:40 id 2 team 2 Ryuvia "prod-alertmanager" created 09-21 16:56:19 last_used 09-25 (now) alerts (name = Watchdog, same fingerprint 972f3dc166677cf2) id 1 team 1 status resolved resolution_source deadman ends_at 09-21 17:43:30 received_at 09-21 17:17:40 id 920 team 2 status firing received_at 09-25 09:50:26 deadman_configs: team 1 alertname=Watchdog 900 s critical | team 2 alertname=Watchdog 900 s critical incidents: id 9 team 1 group_key deadman:972f3dc166677cf2 triggered, resolved_at NULL ``` The sender's Alertmanager receiver was repointed from integration 1's key to integration 2's on 2026-09-21 (Ryuvia/charts `963f53e`, 17:15Z). That is a deliberate change, not a fault. Team 1's switch then correctly declared the heartbeat dead, because no more heartbeats reached team 1. ## Why it never recovers (from reading `internal/api/deadman.go`, not from running a test) - `sweepDeadman` runs per team. For team 2, row 920 is fresh and firing, so it calls `deadmanRecovered`, which selects `... WHERE team_id = $1 AND group_key = $2 AND resolved_at IS NULL` with **team 2's** id. The open incident is in team 1, so it finds `sql.ErrNoRows` and returns. That is correct for isolated teams. - For team 1, row 1 is `resolved`, so it calls `deadmanDied`, which returns immediately because an incident is already open. Nothing ever sends team 1 another heartbeat, so `deadmanRecovered` is never called for it. So the only way out of this state is a person resolving the incident. The code comment in `deadmanRecovered` says "recovery is the only automatic way out", and here recovery is structurally unreachable. ## What is working as designed (so this is not filed as a reminder bug) The 352 reminders are the intended behaviour. `enqueueReminders` stops on acknowledged, snoozed, resolved or archived, and its own comment says there is no reminder cap because **snooze is the mute button**. Reminders apply only to teams with **no escalation ladder** (`NOT EXISTS (SELECT 1 FROM escalation_levels ...)`). That is why team 2's four incidents on the same night got only a `triggered` and a `resolved` notification, and team 1's has been reminded ever since. None of that is wrong. The only thing the reminders did here was make an orphaned switch loud. They did not say *why* it was silent. ## The gap Nothing in the system distinguishes "this heartbeat source died" from "this heartbeat source now reports somewhere else". Both look like a switch that stopped chirping. The switch's team keeps a permanent open incident, the operator's only signal is a reminder every 15 minutes, and the remedy (resolve the incident, remove the matcher from the old team's config) is not suggested anywhere. ## Options, none chosen Recorded so the next attempt does not start from scratch. **No option has been tried.** 1. **Surface it.** Show, on the incident or the team's integrations list, that the integration that fed the switch has not been used since T. Cheapest, and changes no behaviour. 2. **Cross-team recovery by fingerprint.** The code comment says "the unit of monitoring is the fingerprint". Recovering when the same fingerprint is alive in *another* team would fix this case, but it leaks state across teams that the rest of the design keeps isolated, and one team's heartbeat could then close another team's incident. Probably wrong, listed so it is decided rather than assumed. 3. **Expire an unrecovered switch incident when its integration has been unused longer than N.** Needs a policy for N and for what "expire" means (resolve, or archive). 4. **Leave as is** and treat the migration step "remove the old team's switch" as an operator task, documented. ## A smaller observation from the same incident The sweeper ticks every 15 minutes (`sweepInterval = 15 * time.Minute` in `archiver.go`) and the switch's timeout is 900 s. Detection therefore takes between 15 and 30 minutes after the last heartbeat. Here: last heartbeat 17:17:40, incident 17:43:30, that is 25 minutes 50 seconds ("last heartbeat 25m ago" in the event detail). The `Timeout` comment says it must be shorter than Alertmanager's `repeat_interval`; that comparison omits the sweep interval. One data point plus the arithmetic; not a defect by itself, but a configured "900 s" does not mean 900 s. ## What I have not done - Not reproduced in a test. The cross-team scenario above comes from production data and reading the code, not from a failing test in `deadman_test.go`. - Not resolved incident #9 and not edited either team's deadman config. Both are the owner's data and were left alone. - Not read the notification topic or any key; the counts above come from aggregate queries. ## Closing condition Close when a decision is recorded here and verified: for whichever option is chosen, either a test that reproduces the two-team scenario and shows the intended outcome, or, if option 4, the documented step. Not on merge alone. Refs Ryuvia/charts `963f53e` (the receiver change), Ryuvia/talos#1 (unrelated; the night's incidents that led here).
niklas commented 2026-09-25 10:20:54 +00:00
Author
Owner
Copy Link
Copy Source

Cleanup applied to production on 2026-09-25 (the orphaned instance only; the design gap is unchanged)

The stuck incident from this issue was cleared by hand at 10:05:04Z, on request. This comment records what was done and how, so the state in the database is explained. It does not address anything in "The gap" or "Options" above, and the issue stays open.

What changed

incidents        id 9 (team 1, group_key deadman:972f3dc166677cf2)
                   triggered -> resolved, resolution_source 'manual', resolved_at 09-25 10:05:04
                   escalation_level reset to 0
incident_events  one 'resolved' event added to incident 9, user_id NULL
deadman_configs  team 1: matchers 'alertname=Watchdog' -> '' (timeout 900 s and severity critical kept)

Team 2's deadman config (alertname=Watchdog, 900 s, critical) and its Watchdog alert (row 920) were not touched; that is the heartbeat that matters now. Team 1 therefore watches nothing.

How, and what that cost

Applied with SQL in one guarded transaction, not through the API. No API key or session was available, and creating one in the api_keys table was a bigger step than the change itself, so it was not done. The transaction only ran if the incident was exactly id 9 / team 1 / its deadman group key / still open, and the config was exactly alertname=Watchdog; otherwise it would have raised and changed nothing. The three incident changes copy what handleIncidentResolve does (UPDATE ... resolution_source = 'manual', stopEscalation, logEvent(evResolved)); the config change is what handleSetTeamDeadman does with an empty matchers.

Two costs of going around the app, stated so nobody has to work them out later:

  • The resolved event has user_id NULL, so the timeline does not say who resolved it. The handler would have recorded the caller.
  • No notification was queued. This is a correction to what I first said in conversation, that a database edit would bypass the app's notification handling. Reading the handler showed a manual resolve does not queue an all-clear notification either (only deadmanRecovered does), so nothing that the app would have sent was skipped.

Verified afterwards

  • The 10:14:52Z reminder tick queued nothing: incident 9 still has 354 notifications (1 triggered, 352 reminder plus the ones after) and no queued reminder line was logged after the change. Reminders had been going every 15 minutes since 09-21.
  • The sweeper's first pass after the change was 10:17:50Z. That time is computed from the container start (2026-09-22T20:47:50Z plus 15-minute steps), not observed in a log, because a no-op sweep logs nothing. After it: no open incidents, none opened since 10:05Z, team 1's matcher still empty, team 2's Watchdog firing with received_at 10:18:26Z.

Still open

Everything in the issue above: nothing tells an operator that a switch is orphaned when its heartbeat moves to another team, and recovery only looks in the team that received the heartbeat. Team 1's stale Watchdog row (id 1, resolved) is harmless and was left as is. If the heartbeat ever moves again, the same thing will happen.

## Cleanup applied to production on 2026-09-25 (the orphaned instance only; the design gap is unchanged) The stuck incident from this issue was cleared by hand at **10:05:04Z**, on request. This comment records what was done and how, so the state in the database is explained. **It does not address anything in "The gap" or "Options" above, and the issue stays open.** ### What changed ``` incidents id 9 (team 1, group_key deadman:972f3dc166677cf2) triggered -> resolved, resolution_source 'manual', resolved_at 09-25 10:05:04 escalation_level reset to 0 incident_events one 'resolved' event added to incident 9, user_id NULL deadman_configs team 1: matchers 'alertname=Watchdog' -> '' (timeout 900 s and severity critical kept) ``` Team 2's deadman config (`alertname=Watchdog`, 900 s, critical) and its Watchdog alert (row 920) were **not touched**; that is the heartbeat that matters now. Team 1 therefore watches nothing. ### How, and what that cost **Applied with SQL in one guarded transaction, not through the API.** No API key or session was available, and creating one in the `api_keys` table was a bigger step than the change itself, so it was not done. The transaction only ran if the incident was exactly id 9 / team 1 / its deadman group key / still open, and the config was exactly `alertname=Watchdog`; otherwise it would have raised and changed nothing. The three incident changes copy what `handleIncidentResolve` does (`UPDATE ... resolution_source = 'manual'`, `stopEscalation`, `logEvent(evResolved)`); the config change is what `handleSetTeamDeadman` does with an empty `matchers`. Two costs of going around the app, stated so nobody has to work them out later: - **The `resolved` event has `user_id NULL`, so the timeline does not say who resolved it.** The handler would have recorded the caller. - **No notification was queued.** This is a correction to what I first said in conversation, that a database edit would bypass the app's notification handling. Reading the handler showed a manual resolve does not queue an all-clear notification either (only `deadmanRecovered` does), so nothing that the app would have sent was skipped. ### Verified afterwards - The 10:14:52Z reminder tick queued nothing: incident 9 still has **354 notifications** (1 `triggered`, 352 `reminder` plus the ones after) and no `queued reminder` line was logged after the change. Reminders had been going every 15 minutes since 09-21. - The sweeper's first pass after the change was **10:17:50Z**. That time is computed from the container start (2026-09-22T20:47:50Z plus 15-minute steps), **not observed in a log**, because a no-op sweep logs nothing. After it: no open incidents, none opened since 10:05Z, team 1's matcher still empty, team 2's Watchdog firing with `received_at` 10:18:26Z. ### Still open Everything in the issue above: nothing tells an operator that a switch is orphaned when its heartbeat moves to another team, and recovery only looks in the team that received the heartbeat. Team 1's stale Watchdog row (id 1, `resolved`) is harmless and was left as is. If the heartbeat ever moves again, the same thing will happen.
niklas commented 2026-09-25 10:21:10 +00:00
Author
Owner
Copy Link
Copy Source

Correction to the comment above: it says incident 9 has 354 notifications "(1 triggered, 352 reminder plus the ones after)". The exact breakdown, read from notifications, is 1 triggered and 353 reminder (last reminder 2026-09-25 09:59:52Z, none since the resolve). The 352 in the issue body was the count when first looked at, before the reminders at 09:44, 09:59 and one more had been sent; the total of 354 is right.

Correction to the comment above: it says incident 9 has 354 notifications "(1 `triggered`, 352 `reminder` plus the ones after)". The exact breakdown, read from `notifications`, is **1 `triggered` and 353 `reminder`** (last reminder 2026-09-25 09:59:52Z, none since the resolve). The 352 in the issue body was the count when first looked at, before the reminders at 09:44, 09:59 and one more had been sent; the total of 354 is right.
niklas commented 2026-09-25 10:21:26 +00:00
Author
Owner
Copy Link
Copy Source

Second correction, to my own correction above. I said the 352 in the issue body predates "the reminders at 09:44, 09:59 and one more". That is wrong. The 352 was the reminder count when I queried at about 09:53Z, which already included the 09:44:52 reminder. Exactly one reminder was sent after that query (09:59:52Z), which is what took it from 352 to 353. Final state: 1 triggered + 353 reminder = 354, none sent since the incident was resolved at 10:05:04Z.

Second correction, to my own correction above. I said the 352 in the issue body predates "the reminders at 09:44, 09:59 and one more". That is wrong. The 352 was the reminder count when I queried at about 09:53Z, which already included the 09:44:52 reminder. **Exactly one** reminder was sent after that query (09:59:52Z), which is what took it from 352 to 353. Final state: 1 `triggered` + 353 `reminder` = 354, none sent since the incident was resolved at 10:05:04Z.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
auth
Authentication, roles, permissions
 bug
Something is broken
 database
Schema, migrations, storage
 enhancement
New feature or improvement
 epic
Tracking issue with sub-issues
 helm
Chart, deployment, cluster resources
 notifications
Paging, escalation, ntfy delivery
 onboarding
Sign-up, invites, first-run experience
 teams
Multi-tenancy and team scoping
 ui
Web frontend
No Label
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
niklas (Niklas Ye)
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: niklas/terdut-server#22
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?