diff --git a/README.md b/README.md index 50fa583..836b9fa 100644 --- a/README.md +++ b/README.md @@ -513,19 +513,27 @@ nothing unless something downstream notices it stop. That is what `TERDUT_DEADMAN_MATCHERS` defaults to. **Switches belong to a team**, which decides which of its own alerts are -heartbeats and how long a silence has to last. An owner sets them through -`PUT /api/teams/{teamID}/deadman`; a missed heartbeat opens an incident in the -team whose integration received it. +heartbeats and how long a silence has to last. Each **switch** is a row of its +own — a name, one matcher, a timeout and a severity — so switches in one team +can have different deadlines. An owner adds and removes them on **Team → +Switches**, which lists each with a status (**healthy**, **dead**, or +**dormant** until its first heartbeat), when it was last heard from, and when it +last opened an incident; a matcher that several clusters satisfy is broken down +per cluster. The API is `POST`/`DELETE /api/teams/{teamID}/deadman/switches`. A +missed heartbeat opens an incident in the team whose integration received it. +Removing a switch stops the watching; an incident it already opened stays open +until somebody resolves it. -The environment variables are the starting point, not the setting: at startup -every team **without** a configuration of its own is given one from them, and an -owner's later edit is never overwritten by a redeploy. A team created after -that starts watching nothing until its owner says otherwise — inheriting an -install-wide heartbeat would page a new team about a source it has never heard -of. +The environment variables are the starting point, not the setting: the **first** +time the server starts, every team is given a switch per default matcher from +them, once. After that a team's switches are its own — an owner's edit or +deletion is never put back by a redeploy. A team created later starts watching +nothing until its owner says otherwise — inheriting an install-wide heartbeat +would page a new team about a source it has never heard of. A matcher is a set of exact label conditions, one of which must be the -`alertname`, in the same format the environment variable uses: +`alertname`, in the format the environment variable uses (one matcher per switch; the +variable takes several, separated by `;`): ``` alertname=Watchdog,cluster=prod; alertname=EdgeHeartbeat @@ -718,8 +726,9 @@ administrator who is not in the team gets the same `404` as anybody else. | `DELETE` | `/api/teams/{teamID}/invites/{inviteID}` | **owner** | Revoke a link before it expires | | `GET` | `/api/teams/{teamID}/escalation` | member | The team's [escalation ladder](#escalation) `{repeat_count, fallback_topic, levels[]}`. Empty levels means the team has none | | `PUT` | `/api/teams/{teamID}/escalation` | **owner** | Replace it wholesale. `400` for a level with no targets or no timeout — a rung that pages nobody is a silence with a number on it | -| `GET` | `/api/teams/{teamID}/deadman` | member | The team's [dead man's switch](#dead-mans-switch) configuration `{matchers, timeout_seconds, severity}` | -| `PUT` | `/api/teams/{teamID}/deadman` | **owner** | Replace it. `400` when no matcher names an `alertname`, because a switch that silently watches nothing is the failure this feature exists to prevent | +| `GET` | `/api/teams/{teamID}/deadman/switches` | member | The team's [dead man's switches](#dead-mans-switch), each `{id, name, matcher, timeout_seconds, severity, status, last_heartbeat_at, last_triggered_at, open_incident_id, sources[]}`. `status` is `healthy`, `dead` or `dormant`; `sources` has one entry per heartbeat fingerprint. Empty when the team watches nothing | +| `POST` | `/api/teams/{teamID}/deadman/switches` | **owner** | Add one: `{name?, matcher, timeout_seconds, severity?}`. `400` when the matcher names no `alertname` or holds several, or the timeout is not positive — a switch that silently watches nothing is the failure this feature exists to prevent | +| `DELETE` | `/api/teams/{teamID}/deadman/switches/{switchID}` | **owner** | Stop watching. An incident it opened stays open. `404` for a switch of another team | ### Notifications @@ -964,8 +973,8 @@ What changes, and will need attention: **Dead man's switches moved too.** `TERDUT_DEADMAN_MATCHERS`, `_TIMEOUT` and `_SEVERITY` are no longer the setting; they are the default each existing team -is seeded with at startup, after which an owner edits them per team through -`PUT /api/teams/{teamID}/deadman` and a redeploy never overwrites that. +is seeded with at startup, after which an owner manages them per team through +`/api/teams/{teamID}/deadman/switches` and a redeploy never overwrites that. Nothing else about an incident changes, and incidents never move between teams: an alert belongs to whichever team's key it arrived on. diff --git a/internal/api/alertmanager.go b/internal/api/alertmanager.go index 4722a46..3f0f403 100644 --- a/internal/api/alertmanager.go +++ b/internal/api/alertmanager.go @@ -124,7 +124,7 @@ func ingest(ctx context.Context, db *sql.DB, notify NotifyConfig, teamID int64, // Which arriving alerts are heartbeats is the team's own answer, read // inside the transaction so an owner editing it mid-payload cannot split // one webhook across two interpretations. - deadman, err := deadmanConfigForTeam(ctx, tx, teamID) + deadman, err := deadmanSetForTeam(ctx, tx, teamID) if err != nil { return err } @@ -186,7 +186,7 @@ func ingest(ctx context.Context, db *sql.DB, notify NotifyConfig, teamID int64, // upsertAlerts stores each alert of a payload and reports what changed. Payloads // the ordering guard rejected are left out entirely. -func upsertAlerts(ctx context.Context, tx *sql.Tx, deadman DeadmanConfig, teamID int64, alerts []amAlert) ([]ingested, error) { +func upsertAlerts(ctx context.Context, tx *sql.Tx, deadman deadmanSet, teamID int64, alerts []amAlert) ([]ingested, error) { now := time.Now().Unix() accepted := make([]ingested, 0, len(alerts)) diff --git a/internal/api/api_test.go b/internal/api/api_test.go index 0943e06..0e22904 100644 --- a/internal/api/api_test.go +++ b/internal/api/api_test.go @@ -88,27 +88,25 @@ func newDeadmanTS(t *testing.T, deadman api.DeadmanConfig, notify ...api.NotifyC return s } -// setTeamDeadman configures the default team's switches over the API, rendering -// the matchers back into the string form the endpoint takes. +// setTeamDeadman gives the default team one switch per configured matcher, over +// the API, the way an owner would add them. func setTeamDeadman(t *testing.T, s *ts, cfg api.DeadmanConfig) { t.Helper() - matchers := make([]string, 0, len(cfg.Matchers)) for _, m := range cfg.Matchers { parts := []string{"alertname=" + m.Name} for k, v := range m.Labels { parts = append(parts, k+"="+v) } sort.Strings(parts[1:]) - matchers = append(matchers, strings.Join(parts, ",")) - } - resp := s.req(t, http.MethodPut, "/api/teams/"+defaultTeam+"/deadman", map[string]any{ - "matchers": strings.Join(matchers, "; "), - "timeout_seconds": int64(cfg.Timeout.Seconds()), - "severity": cfg.Severity, - }) - defer resp.Body.Close() - if resp.StatusCode != http.StatusOK { - t.Fatalf("configure the team's dead man's switches: %d", resp.StatusCode) + resp := s.req(t, http.MethodPost, "/api/teams/"+defaultTeam+"/deadman/switches", map[string]any{ + "matcher": strings.Join(parts, ","), + "timeout_seconds": int64(cfg.Timeout.Seconds()), + "severity": cfg.Severity, + }) + resp.Body.Close() + if resp.StatusCode != http.StatusCreated { + t.Fatalf("add a dead man's switch: %d", resp.StatusCode) + } } } diff --git a/internal/api/deadman.go b/internal/api/deadman.go index 50b1d7a..441c0fa 100644 --- a/internal/api/deadman.go +++ b/internal/api/deadman.go @@ -4,6 +4,8 @@ import ( "context" "database/sql" "encoding/json" + "errors" + "fmt" "log" "sort" "strings" @@ -39,6 +41,17 @@ func (m DeadmanMatcher) String() string { return m.Name + " (" + strings.Join(parts, ", ") + ")" } +// config renders the matcher in the form parseDeadmanMatcher reads, which is +// what a switch row stores: `alertname=Watchdog,cluster=prod`. +func (m DeadmanMatcher) config() string { + parts := make([]string, 0, len(m.Labels)) + for k, v := range m.Labels { + parts = append(parts, k+"="+v) + } + sort.Strings(parts) + return strings.Join(append([]string{"alertname=" + m.Name}, parts...), ",") +} + // matches reports whether an alert's labels satisfy every condition. func (m DeadmanMatcher) matches(labels map[string]string) bool { if labels["alertname"] != m.Name { @@ -52,12 +65,10 @@ func (m DeadmanMatcher) matches(labels map[string]string) bool { return true } -// DeadmanConfig inverts the handling of the alerts it matches: receiving one -// opens nothing, and the absence of one opens an incident. -// -// The unit of monitoring is the fingerprint, not the matcher — two clusters -// sending the same heartbeat alertname are two independent switches, so one -// healthy cluster cannot mask a dead one. +// DeadmanConfig is the server-wide default a team's switches are seeded from: +// the environment's matchers, timeout and severity. Switches themselves are rows +// of a team's own — see DeadmanSwitch — and this is only how a fresh install +// starts out. type DeadmanConfig struct { Matchers []DeadmanMatcher @@ -76,41 +87,81 @@ type DeadmanConfig struct { // enabled reports whether there is anything to watch. func (c DeadmanConfig) enabled() bool { return c.Timeout > 0 && len(c.Matchers) > 0 } -// match returns the first matcher an alert satisfies. -func (c DeadmanConfig) match(labels map[string]string) (DeadmanMatcher, bool) { - if !c.enabled() { - return DeadmanMatcher{}, false - } - for _, m := range c.Matchers { - if m.matches(labels) { - return m, true - } - } - return DeadmanMatcher{}, false +// DeadmanSwitch inverts the handling of the alerts it matches: receiving one +// opens nothing, and the absence of one opens an incident. +// +// The unit of monitoring is the fingerprint, not the switch — two clusters +// sending the same heartbeat alertname are two independent heartbeats under one +// switch, so one healthy cluster cannot mask a dead one. +type DeadmanSwitch struct { + ID int64 + Name string + Matcher DeadmanMatcher + + // Timeout is how long a heartbeat may go unheard before it is declared dead. + Timeout time.Duration + + // Severity is what the incident opens at. + Severity string } -// isDeadman is match without the matcher, for the ingest path. -func (c DeadmanConfig) isDeadman(labels map[string]string) bool { - _, ok := c.match(labels) +// deadmanSet is one team's switches. +type deadmanSet []DeadmanSwitch + +// match returns the first switch an alert satisfies. +func (d deadmanSet) match(labels map[string]string) (DeadmanSwitch, bool) { + for _, sw := range d { + if sw.Matcher.matches(labels) { + return sw, true + } + } + return DeadmanSwitch{}, false +} + +// isDeadman is match without the switch, for the ingest path. +func (d deadmanSet) isDeadman(labels map[string]string) bool { + _, ok := d.match(labels) return ok } // names lists the distinct alertnames worth loading from the database. -func (c DeadmanConfig) names() []string { +func (d deadmanSet) names() []string { seen := map[string]bool{} - out := make([]string, 0, len(c.Matchers)) - for _, m := range c.Matchers { - if !seen[m.Name] { - seen[m.Name] = true - out = append(out, m.Name) + out := make([]string, 0, len(d)) + for _, sw := range d { + if !seen[sw.Matcher.Name] { + seen[sw.Matcher.Name] = true + out = append(out, sw.Matcher.Name) } } return out } +// parseDeadmanMatcher reads one matcher from its configured form: "," separates +// the conditions and "=" is exact label equality — `alertname=Watchdog,cluster=prod`. +// The error says what is wrong with it, in words a form can show. +func parseDeadmanMatcher(entry string) (DeadmanMatcher, error) { + m := DeadmanMatcher{Labels: map[string]string{}} + for _, cond := range strings.Split(strings.TrimSpace(entry), ",") { + k, v, ok := strings.Cut(cond, "=") + k, v = strings.TrimSpace(k), strings.TrimSpace(v) + if !ok || k == "" || v == "" { + return DeadmanMatcher{}, fmt.Errorf("%q is not label=value", strings.TrimSpace(cond)) + } + if k == "alertname" { + m.Name = v + continue + } + m.Labels[k] = v + } + if m.Name == "" { + return DeadmanMatcher{}, errors.New("no alertname condition") + } + return m, nil +} + // ParseDeadmanConfig reads the matcher list from its configured form: -// ";" separates matchers, "," separates the conditions within one, and "=" is -// exact label equality — `alertname=Watchdog,cluster=prod; alertname=Heartbeat`. +// ";" separates matchers, and each is parsed as parseDeadmanMatcher does. // // A malformed or alertname-less entry is dropped rather than fatal, following // config.duration's rule that one bad tuning knob should not take the server @@ -125,28 +176,9 @@ func ParseDeadmanConfig(matchers string, timeout time.Duration, severity string) if entry == "" { continue } - - m := DeadmanMatcher{Labels: map[string]string{}} - malformed := false - for _, cond := range strings.Split(entry, ",") { - k, v, ok := strings.Cut(cond, "=") - k, v = strings.TrimSpace(k), strings.TrimSpace(v) - if !ok || k == "" || v == "" { - log.Printf("deadman: ignoring matcher %q: %q is not label=value", entry, strings.TrimSpace(cond)) - malformed = true - break - } - if k == "alertname" { - m.Name = v - continue - } - m.Labels[k] = v - } - if malformed { - continue - } - if m.Name == "" { - log.Printf("deadman: ignoring matcher %q: no alertname condition", entry) + m, err := parseDeadmanMatcher(entry) + if err != nil { + log.Printf("deadman: ignoring matcher %q: %v", entry, err) continue } cfg.Matchers = append(cfg.Matchers, m) @@ -162,23 +194,35 @@ func ParseDeadmanConfig(matchers string, timeout time.Duration, severity string) for _, m := range cfg.Matchers { rendered = append(rendered, m.String()) } - log.Printf("deadman: watching %s, timeout %s, severity %s", + log.Printf("deadman: default for new teams: %s, timeout %s, severity %s", strings.Join(rendered, "; "), timeout, severity) } return cfg } -// deadmanAlert is one switch: the alert row carrying its last heartbeat. +// deadmanAlert is one heartbeat: the alert row carrying its last sighting, and +// the switch that claimed it. type deadmanAlert struct { id int64 teamID int64 fingerprint string labels map[string]string - matcher DeadmanMatcher + sw DeadmanSwitch resolved bool receivedAt int64 } +// dead is the one rule for a silent heartbeat, shared by the sweeper that pages +// on it and the status the Switches page shows, so the page cannot disagree +// with the pager. +// +// An explicit resolved from Alertmanager is a stronger death signal than mere +// absence: the sender is telling us the heartbeat stopped, so there is nothing +// left to wait out. +func (a deadmanAlert) dead(now time.Time) bool { + return a.resolved || a.receivedAt < now.Add(-a.sw.Timeout).Unix() +} + // groupKey is the switch's identity as an incident. Per fingerprint, so each // source is tracked on its own. func (a deadmanAlert) groupKey() string { return deadmanGroupPrefix + a.fingerprint } @@ -189,13 +233,13 @@ func (a deadmanAlert) groupKey() string { return deadmanGroupPrefix + a.fingerpr // It returns the ids of the alerts it owns, because the generic staleness // expiry must leave them alone — staleAfter and ends_at would otherwise resolve // a heartbeat long before its own, much tighter, timeout ever fired. -// Each team is swept against its own configuration: its own matchers, its own -// timeout, its own severity. A team watching nothing is skipped entirely, which -// is most of them. +// Each team is swept against its own switches, each with its own matcher, +// timeout and severity. A team watching nothing is skipped entirely, which is +// most of them. func sweepDeadman(ctx context.Context, db *sql.DB, notify NotifyConfig) map[int64]bool { owned := map[int64]bool{} - configs, err := deadmanConfigs(ctx, db) + configs, err := deadmanSets(ctx, db) if err != nil { log.Printf("deadman: load configs: %v", err) return owned @@ -203,39 +247,35 @@ func sweepDeadman(ctx context.Context, db *sql.DB, notify NotifyConfig) map[int6 now := time.Now() for teamID, cfg := range configs { - switches, err := deadmanAlerts(ctx, db, teamID, cfg) + heartbeats, err := deadmanAlerts(ctx, db, teamID, cfg) if err != nil { - log.Printf("deadman: load switches for team %d: %v", teamID, err) + log.Printf("deadman: load heartbeats for team %d: %v", teamID, err) continue } - cutoff := now.Add(-cfg.Timeout).Unix() - for _, sw := range switches { - owned[sw.id] = true + for _, hb := range heartbeats { + owned[hb.id] = true - // An explicit resolved from Alertmanager is a stronger death signal - // than mere absence: the sender is telling us the heartbeat - // stopped, so there is nothing left to wait out. - if sw.resolved || sw.receivedAt < cutoff { - if err := deadmanDied(ctx, db, cfg, notify, sw, now); err != nil { - log.Printf("deadman: open incident for %s: %v", sw.matcher.Name, err) + if hb.dead(now) { + if err := deadmanDied(ctx, db, notify, hb, now); err != nil { + log.Printf("deadman: open incident for %s: %v", hb.sw.Matcher.Name, err) } continue } - if err := deadmanRecovered(ctx, db, sw); err != nil { - log.Printf("deadman: resolve incident for %s: %v", sw.matcher.Name, err) + if err := deadmanRecovered(ctx, db, hb); err != nil { + log.Printf("deadman: resolve incident for %s: %v", hb.sw.Matcher.Name, err) } } } return owned } -// deadmanAlerts loads every alert row that a matcher claims. The candidate query +// deadmanAlerts loads every alert row that one of a team's switches claims. The candidate query // is narrowed by alertname so it rides alerts_name_idx; the rest of the matching // happens in Go, which keeps one implementation of the rules. The rows are read // in full before the caller writes, so the writes do not run against an open // cursor over the same table. -func deadmanAlerts(ctx context.Context, db *sql.DB, teamID int64, cfg DeadmanConfig) ([]deadmanAlert, error) { +func deadmanAlerts(ctx context.Context, db *sql.DB, teamID int64, cfg deadmanSet) ([]deadmanAlert, error) { names := cfg.names() args := &sqlArgs{} nameList := make([]any, len(names)) @@ -263,11 +303,11 @@ func deadmanAlerts(ctx context.Context, db *sql.DB, teamID int64, cfg DeadmanCon } json.Unmarshal([]byte(labelsJSON), &a.labels) //nolint:errcheck - m, ok := cfg.match(a.labels) + sw, ok := cfg.match(a.labels) if !ok { continue } - a.matcher = m + a.sw = sw a.resolved = status == "resolved" out = append(out, a) } @@ -284,16 +324,16 @@ func deadmanAlerts(ctx context.Context, db *sql.DB, teamID int64, cfg DeadmanCon // incidentForGroup), and a source that is gone for good is a one-time page // rather than a nag. Only a heartbeat that comes back and dies again earns a new // incident. -func deadmanDied(ctx context.Context, db *sql.DB, cfg DeadmanConfig, notify NotifyConfig, sw deadmanAlert, now time.Time) error { +func deadmanDied(ctx context.Context, db *sql.DB, notify NotifyConfig, hb deadmanAlert, now time.Time) error { var lastTriggered, open int64 if err := db.QueryRowContext(ctx, ` SELECT COALESCE(MAX(triggered_at), 0), COUNT(*) FILTER (WHERE resolved_at IS NULL) FROM incidents WHERE team_id = $1 AND group_key = $2`, - sw.teamID, sw.groupKey()).Scan(&lastTriggered, &open); err != nil { + hb.teamID, hb.groupKey()).Scan(&lastTriggered, &open); err != nil { return err } - if open > 0 || sw.receivedAt <= lastTriggered { + if open > 0 || hb.receivedAt <= lastTriggered { return nil } @@ -306,18 +346,18 @@ func deadmanDied(ctx context.Context, db *sql.DB, cfg DeadmanConfig, notify Noti // A heartbeat nobody has heard from is not firing, and saying otherwise in // the alert list would be a lie. An Alertmanager-sourced resolution keeps its // own source: it told us the truth first. - if !sw.resolved { + if !hb.resolved { if _, err := tx.ExecContext(ctx, ` UPDATE alerts SET status = 'resolved', resolution_source = $1, ends_at = COALESCE(ends_at, `+nowEpoch+`) - WHERE id = $2 AND status = 'firing'`, resolutionDeadman, sw.id); err != nil { + WHERE id = $2 AND status = 'firing'`, resolutionDeadman, hb.id); err != nil { return err } } - severity := cfg.Severity + severity := hb.sw.Severity var sev *string if severity != "" { sev = &severity @@ -325,14 +365,14 @@ func deadmanDied(ctx context.Context, db *sql.DB, cfg DeadmanConfig, notify Noti // The incident opens in the team whose integration received the heartbeat: // the switch belongs to whoever is watching that source, not to the install. - incidentID, err := openIncident(ctx, tx, notify, sw.teamID, sw.groupKey(), - "No heartbeat from "+sw.matcher.String(), sw.labels, sev) + incidentID, err := openIncident(ctx, tx, notify, hb.teamID, hb.groupKey(), + "No heartbeat from "+hb.sw.Matcher.String(), hb.labels, sev) if err != nil { return err } - alertID := sw.id - detail := "last heartbeat " + humanDuration(now.Sub(time.Unix(sw.receivedAt, 0))) + " ago" + alertID := hb.id + detail := "last heartbeat " + humanDuration(now.Sub(time.Unix(hb.receivedAt, 0))) + " ago" if err := logEvent(ctx, tx, incidentID, evDeadmanSilent, nil, &alertID, &detail); err != nil { return err } @@ -340,7 +380,7 @@ func deadmanDied(ctx context.Context, db *sql.DB, cfg DeadmanConfig, notify Noti if err := tx.Commit(); err != nil { return err } - log.Printf("deadman: %s went silent, opened incident %d", sw.matcher.String(), incidentID) + log.Printf("deadman: %s went silent, opened incident %d", hb.sw.Matcher.String(), incidentID) return nil } @@ -350,12 +390,12 @@ func deadmanDied(ctx context.Context, db *sql.DB, cfg DeadmanConfig, notify Noti // member alerts (linking the heartbeat would have the settled-incident cascade // close it on the very same sweep that opened it), so the alert-driven cascade // ignores it entirely and recovery is the only automatic way out. -func deadmanRecovered(ctx context.Context, db *sql.DB, sw deadmanAlert) error { +func deadmanRecovered(ctx context.Context, db *sql.DB, hb deadmanAlert) error { var incidentID int64 switch err := db.QueryRowContext(ctx, ` SELECT id FROM incidents WHERE team_id = $1 AND group_key = $2 AND resolved_at IS NULL`, - sw.teamID, sw.groupKey()).Scan(&incidentID); { + hb.teamID, hb.groupKey()).Scan(&incidentID); { case err == sql.ErrNoRows: return nil case err != nil: @@ -387,116 +427,256 @@ func deadmanRecovered(ctx context.Context, db *sql.DB, sw deadmanAlert) error { if err := tx.Commit(); err != nil { return err } - log.Printf("deadman: %s is back, resolved incident %d", sw.matcher.String(), incidentID) + log.Printf("deadman: %s is back, resolved incident %d", hb.sw.Matcher.String(), incidentID) return nil } // --------------------------------------------------------------------------- -// Per-team configuration +// A team's switches // --------------------------------------------------------------------------- -// deadmanConfigForTeam reads one team's switches. A team with no row, or with -// nothing configured, gets a disabled config — which is the right answer rather -// than an error: most teams watch no heartbeat at all. -func deadmanConfigForTeam(ctx context.Context, q querier, teamID int64) (DeadmanConfig, error) { - var matchers, severity string - var timeout int64 - err := q.QueryRowContext(ctx, - "SELECT matchers, timeout_seconds, severity FROM deadman_configs WHERE team_id = $1", - teamID).Scan(&matchers, &timeout, &severity) - if err == sql.ErrNoRows { - return DeadmanConfig{}, nil - } - if err != nil { - return DeadmanConfig{}, err - } - return parseDeadmanQuietly(matchers, time.Duration(timeout)*time.Second, severity), nil -} +const deadmanSwitchColumns = "id, team_id, name, matcher, timeout_seconds, severity" -// deadmanConfigs reads every team's switches in one query, for the sweeper. -func deadmanConfigs(ctx context.Context, db *sql.DB) (map[int64]DeadmanConfig, error) { - rows, err := db.QueryContext(ctx, - "SELECT team_id, matchers, timeout_seconds, severity FROM deadman_configs") - if err != nil { - return nil, err - } +// scanDeadmanSwitches reads switch rows into per-team sets. A row whose matcher +// no longer parses is skipped rather than fatal: the API refuses to store one, +// so it can only mean a hand edit, and one bad row must not stop the others +// from being watched. +func scanDeadmanSwitches(rows *sql.Rows) (map[int64]deadmanSet, error) { defer rows.Close() - - out := map[int64]DeadmanConfig{} + out := map[int64]deadmanSet{} for rows.Next() { + var sw DeadmanSwitch var teamID, timeout int64 - var matchers, severity string - if err := rows.Scan(&teamID, &matchers, &timeout, &severity); err != nil { + var matcher string + if err := rows.Scan(&sw.ID, &teamID, &sw.Name, &matcher, &timeout, &sw.Severity); err != nil { return nil, err } - cfg := parseDeadmanQuietly(matchers, time.Duration(timeout)*time.Second, severity) - if cfg.enabled() { - out[teamID] = cfg + m, err := parseDeadmanMatcher(matcher) + if err != nil { + log.Printf("deadman: switch %d has an unusable matcher %q: %v", sw.ID, matcher, err) + continue } + sw.Matcher = m + sw.Timeout = time.Duration(timeout) * time.Second + out[teamID] = append(out[teamID], sw) } return out, rows.Err() } -// SeedDeadmanConfigs gives every team without a row the server's environment -// configuration, so the install that upgrades into per-team switches keeps -// watching exactly what it was watching before. +// deadmanSetForTeam reads one team's switches. A team with none gets an empty +// set — which is the right answer rather than an error: most teams watch no +// heartbeat at all. +func deadmanSetForTeam(ctx context.Context, q querier, teamID int64) (deadmanSet, error) { + rows, err := q.QueryContext(ctx, + "SELECT "+deadmanSwitchColumns+" FROM deadman_switches WHERE team_id = $1 ORDER BY id", teamID) + if err != nil { + return nil, err + } + sets, err := scanDeadmanSwitches(rows) + return sets[teamID], err +} + +// deadmanSets reads every team's switches in one query, for the sweeper. +func deadmanSets(ctx context.Context, db *sql.DB) (map[int64]deadmanSet, error) { + rows, err := db.QueryContext(ctx, + "SELECT "+deadmanSwitchColumns+" FROM deadman_switches ORDER BY id") + if err != nil { + return nil, err + } + return scanDeadmanSwitches(rows) +} + +// deadmanSeededKey is the settings row that records the environment defaults +// were handed out. Without it, a team that deleted its last switch would get +// the default back on the next restart. +const deadmanSeededKey = "deadman_seeded" + +// SeedDeadmanConfigs gives every team the server's environment defaults as +// switches, exactly once per install, so a fresh install watches Watchdog +// without anybody setting it up. // -// Idempotent, and never overwrites: once a team has a row it owns its own -// configuration, and a redeploy must not quietly put the environment's value -// back over an owner's edit. +// Once seeded it never runs again: a team's switches are its own, and a redeploy +// must not quietly put the environment's value back over an owner's edit or +// deletion. Installs that upgraded from per-team configuration were already +// seeded, which migration 009 records. // -// A team created after startup gets no row and therefore watches nothing until -// its owner says otherwise. That is deliberate: inheriting an install-wide -// heartbeat would page a new team about a source it has never heard of, and a -// switch nobody chose is the kind that gets muted rather than fixed. +// A team created after that gets none and watches nothing until its owner says +// otherwise. That is deliberate: inheriting an install-wide heartbeat would page +// a new team about a source it has never heard of, and a switch nobody chose is +// the kind that gets muted rather than fixed. func SeedDeadmanConfigs(ctx context.Context, db *sql.DB, cfg DeadmanConfig) error { - matchers := make([]string, 0, len(cfg.Matchers)) + if !cfg.enabled() { + return nil + } + + tx, err := db.BeginTx(ctx, nil) + if err != nil { + return err + } + defer tx.Rollback() //nolint:errcheck + + res, err := tx.ExecContext(ctx, + "INSERT INTO settings (key, value) VALUES ($1, '1') ON CONFLICT (key) DO NOTHING", + deadmanSeededKey) + if err != nil { + return err + } + if n, _ := res.RowsAffected(); n == 0 { + return nil + } + for _, m := range cfg.Matchers { - parts := []string{"alertname=" + m.Name} - for k, v := range m.Labels { - parts = append(parts, k+"="+v) + if _, err := tx.ExecContext(ctx, ` + INSERT INTO deadman_switches (team_id, name, matcher, timeout_seconds, severity) + SELECT id, $1, $1, $2, $3 FROM teams`, + m.config(), int64(cfg.Timeout.Seconds()), cfg.Severity); err != nil { + return err } - sort.Strings(parts[1:]) - matchers = append(matchers, strings.Join(parts, ",")) } - - _, err := db.ExecContext(ctx, ` - INSERT INTO deadman_configs (team_id, matchers, timeout_seconds, severity) - SELECT id, $1, $2, $3 FROM teams - ON CONFLICT (team_id) DO NOTHING`, - strings.Join(matchers, "; "), int64(cfg.Timeout.Seconds()), cfg.Severity) - return err + return tx.Commit() } -// parseDeadmanQuietly is ParseDeadmanConfig without the startup logging: a -// team's configuration is read on every sweep and every webhook, and logging it -// each time would bury everything else. -func parseDeadmanQuietly(matchers string, timeout time.Duration, severity string) DeadmanConfig { - cfg := DeadmanConfig{Timeout: timeout, Severity: severity} - for _, entry := range strings.Split(matchers, ";") { - entry = strings.TrimSpace(entry) - if entry == "" { - continue - } - m := DeadmanMatcher{Labels: map[string]string{}} - malformed := false - for _, cond := range strings.Split(entry, ",") { - k, v, ok := strings.Cut(cond, "=") - k, v = strings.TrimSpace(k), strings.TrimSpace(v) - if !ok || k == "" || v == "" { - malformed = true - break - } - if k == "alertname" { - m.Name = v - continue - } - m.Labels[k] = v - } - if malformed || m.Name == "" { - continue - } - cfg.Matchers = append(cfg.Matchers, m) - } - return cfg +// --------------------------------------------------------------------------- +// Status +// --------------------------------------------------------------------------- + +const ( + switchHealthy = "healthy" + switchDead = "dead" + switchDormant = "dormant" +) + +// deadmanSource is one heartbeat under a switch: a fingerprint that matched. +type deadmanSource struct { + Fingerprint string `json:"fingerprint"` + Labels map[string]string `json:"labels"` + Status string `json:"status"` + LastHeartbeatAt time.Time `json:"last_heartbeat_at"` + LastTriggeredAt *time.Time `json:"last_triggered_at"` + IncidentID *int64 `json:"incident_id"` +} + +// deadmanSwitchStatus is a switch as the Switches page shows it. +type deadmanSwitchStatus struct { + ID int64 `json:"id"` + Name string `json:"name"` + Matcher string `json:"matcher"` + TimeoutSeconds int64 `json:"timeout_seconds"` + Severity string `json:"severity"` + + // Status is dead when any source is, dormant when none has ever been heard + // from, healthy otherwise — a live cluster must not hide a dead one. + Status string `json:"status"` + LastHeartbeatAt *time.Time `json:"last_heartbeat_at"` + LastTriggeredAt *time.Time `json:"last_triggered_at"` + OpenIncidentID *int64 `json:"open_incident_id"` + Sources []deadmanSource `json:"sources"` +} + +// deadmanStatuses reports every switch of a team with what its heartbeats are +// doing. The liveness verdict is deadmanAlert.dead, the sweeper's own. +func deadmanStatuses(ctx context.Context, db *sql.DB, teamID int64, set deadmanSet, now time.Time) ([]deadmanSwitchStatus, error) { + out := make([]deadmanSwitchStatus, 0, len(set)) + if len(set) == 0 { + return out, nil + } + + heartbeats, err := deadmanAlerts(ctx, db, teamID, set) + if err != nil { + return nil, err + } + + // One query for every switch's incident history, keyed the way the sweeper + // keys it. + type history struct { + triggeredAt int64 + openID int64 + } + incidents := map[string]history{} + rows, err := db.QueryContext(ctx, ` + SELECT group_key, MAX(triggered_at), COALESCE(MAX(id) FILTER (WHERE resolved_at IS NULL), 0) + FROM incidents + WHERE team_id = $1 AND group_key LIKE $2 + GROUP BY group_key`, teamID, deadmanGroupPrefix+"%") + if err != nil { + return nil, err + } + defer rows.Close() + for rows.Next() { + var key string + var h history + if err := rows.Scan(&key, &h.triggeredAt, &h.openID); err != nil { + return nil, err + } + incidents[key] = h + } + if err := rows.Err(); err != nil { + return nil, err + } + + bySwitch := map[int64][]deadmanAlert{} + for _, hb := range heartbeats { + bySwitch[hb.sw.ID] = append(bySwitch[hb.sw.ID], hb) + } + + later := func(cur *time.Time, unix int64) *time.Time { + t := time.Unix(unix, 0).UTC() + if cur == nil || t.After(*cur) { + return &t + } + return cur + } + + for _, sw := range set { + st := deadmanSwitchStatus{ + ID: sw.ID, Name: sw.Name, Matcher: sw.Matcher.config(), + TimeoutSeconds: int64(sw.Timeout.Seconds()), Severity: sw.Severity, + Status: switchDormant, Sources: []deadmanSource{}, + } + + for _, hb := range bySwitch[sw.ID] { + src := deadmanSource{ + Fingerprint: hb.fingerprint, + Labels: hb.labels, + Status: switchHealthy, + LastHeartbeatAt: time.Unix(hb.receivedAt, 0).UTC(), + } + if hb.dead(now) { + src.Status = switchDead + } + if h, ok := incidents[hb.groupKey()]; ok { + t := time.Unix(h.triggeredAt, 0).UTC() + src.LastTriggeredAt = &t + st.LastTriggeredAt = later(st.LastTriggeredAt, h.triggeredAt) + if h.openID != 0 { + id := h.openID + src.IncidentID = &id + if st.OpenIncidentID == nil || id > *st.OpenIncidentID { + st.OpenIncidentID = &id + } + } + } + st.LastHeartbeatAt = later(st.LastHeartbeatAt, hb.receivedAt) + st.Sources = append(st.Sources, src) + + switch { + case src.Status == switchDead: + st.Status = switchDead + case st.Status == switchDormant: + st.Status = switchHealthy + } + } + + // Dead ones first, then by fingerprint: what needs attention leads, and + // the order does not shuffle between refreshes. + sort.Slice(st.Sources, func(i, j int) bool { + a, b := st.Sources[i], st.Sources[j] + if (a.Status == switchDead) != (b.Status == switchDead) { + return a.Status == switchDead + } + return a.Fingerprint < b.Fingerprint + }) + out = append(out, st) + } + return out, nil } diff --git a/internal/api/deadman_test.go b/internal/api/deadman_test.go index 4b69aef..4d6d172 100644 --- a/internal/api/deadman_test.go +++ b/internal/api/deadman_test.go @@ -1,6 +1,7 @@ package api_test import ( + "context" "net/http" "strings" "testing" @@ -483,13 +484,13 @@ func TestDeadman_ConfigurationIsPerTeam(t *testing.T) { unwatched := newTeam(t, s, "unwatched") // Only the first team calls Watchdog a heartbeat. - resp := s.req(t, http.MethodPut, "/api/teams/"+id64(watched.id)+"/deadman", map[string]any{ - "matchers": "alertname=Watchdog", + resp := s.req(t, http.MethodPost, "/api/teams/"+id64(watched.id)+"/deadman/switches", map[string]any{ + "matcher": "alertname=Watchdog", "timeout_seconds": 3600, "severity": "critical", }) resp.Body.Close() - if resp.StatusCode != http.StatusOK { + if resp.StatusCode != http.StatusCreated { t.Fatalf("configure the watched team: %d", resp.StatusCode) } @@ -550,9 +551,9 @@ func TestDeadman_ConfigurationIsOwnerOnly(t *testing.T) { decode(t, s.req(t, http.MethodPost, "/api/users/"+id64(user.ID)+"/api-keys", map[string]string{"name": "test"}), &key) - req, _ := http.NewRequest(http.MethodPut, - s.URL+"/api/teams/"+id64(team.id)+"/deadman", - strings.NewReader(`{"matchers":"alertname=Watchdog","timeout_seconds":60}`)) + req, _ := http.NewRequest(http.MethodPost, + s.URL+"/api/teams/"+id64(team.id)+"/deadman/switches", + strings.NewReader(`{"matcher":"alertname=Watchdog","timeout_seconds":60}`)) req.Header.Set("Authorization", "Bearer "+key.Key) req.Header.Set("Content-Type", "application/json") resp, err := http.DefaultClient.Do(req) @@ -564,7 +565,7 @@ func TestDeadman_ConfigurationIsOwnerOnly(t *testing.T) { t.Errorf("a member editing the switches: expected 403, got %d", resp.StatusCode) } - read, _ := http.NewRequest(http.MethodGet, s.URL+"/api/teams/"+id64(team.id)+"/deadman", nil) + read, _ := http.NewRequest(http.MethodGet, s.URL+"/api/teams/"+id64(team.id)+"/deadman/switches", nil) read.Header.Set("Authorization", "Bearer "+key.Key) got, err := http.DefaultClient.Do(read) if err != nil { @@ -577,16 +578,165 @@ func TestDeadman_ConfigurationIsOwnerOnly(t *testing.T) { } // A matcher with no alertname watches nothing, silently, which is the failure -// this feature exists to prevent — so it is refused at the door. -func TestDeadman_UnusableMatchersAreRejected(t *testing.T) { +// this feature exists to prevent — so it is refused at the door, along with the +// other things that would make a switch unable to fire. +func TestDeadman_UnusableSwitchesAreRejected(t *testing.T) { s, _ := deadmanTS(t, deadmanCfg()) - resp := s.req(t, http.MethodPut, "/api/teams/"+defaultTeam+"/deadman", map[string]any{ - "matchers": "cluster=prod", - "timeout_seconds": 900, - }) - resp.Body.Close() - if resp.StatusCode != http.StatusBadRequest { - t.Errorf("expected 400 for a matcher with no alertname, got %d", resp.StatusCode) + for name, body := range map[string]map[string]any{ + "no alertname": {"matcher": "cluster=prod", "timeout_seconds": 900}, + "malformed": {"matcher": "alertname=Watchdog,garbage", "timeout_seconds": 900}, + "several": {"matcher": "alertname=A; alertname=B", "timeout_seconds": 900}, + "zero timeout": {"matcher": "alertname=Watchdog", "timeout_seconds": 0}, + "bad severity": {"matcher": "alertname=Watchdog", "timeout_seconds": 900, "severity": "loud"}, + "empty matcher": {"matcher": "", "timeout_seconds": 900}, + } { + resp := s.req(t, http.MethodPost, "/api/teams/"+defaultTeam+"/deadman/switches", body) + resp.Body.Close() + if resp.StatusCode != http.StatusBadRequest { + t.Errorf("%s: expected 400, got %d", name, resp.StatusCode) + } + } +} + +// --------------------------------------------------------------------------- +// The switch list +// --------------------------------------------------------------------------- + +// listSwitches reads the default team's switches as the Switches page does. +func listSwitches(t *testing.T, s *ts) []map[string]any { + t.Helper() + return list(t, s.req(t, http.MethodGet, "/api/teams/"+defaultTeam+"/deadman/switches", nil)) +} + +// A switch is healthy while its heartbeat is fresh, dead once it is silent, and +// dormant until the first one arrives. +func TestDeadman_ListReportsStatus(t *testing.T) { + s, _ := deadmanTS(t, api.ParseDeadmanConfig("alertname=Watchdog; alertname=NeverSent", time.Hour, "critical")) + + got := listSwitches(t, s) + if len(got) != 2 { + t.Fatalf("expected 2 switches, got %d", len(got)) + } + for _, sw := range got { + if sw["status"] != "dormant" || sw["last_heartbeat_at"] != nil || sw["last_triggered_at"] != nil { + t.Errorf("a switch nobody has heard from should be dormant and blank, got %v", sw) + } + } + + heartbeat(t, s, "fp-watchdog", nil) + got = listSwitches(t, s) + if got[0]["status"] != "healthy" || got[0]["last_heartbeat_at"] == nil { + t.Errorf("a fresh heartbeat should be healthy with a timestamp, got %v", got[0]) + } + if got[1]["status"] != "dormant" { + t.Errorf("the other switch is still dormant, got %v", got[1]["status"]) + } + + silence(t, s, "fp-watchdog", 2*time.Hour) + sweep(t, s, noArchive) + got = listSwitches(t, s) + if got[0]["status"] != "dead" { + t.Fatalf("a silent heartbeat should be dead, got %v", got[0]["status"]) + } + if got[0]["last_triggered_at"] == nil || got[0]["open_incident_id"] == nil { + t.Errorf("a dead switch should show when it triggered and its open incident, got %v", got[0]) + } +} + +// One matcher, several clusters: the switch is as bad as its worst heartbeat and +// each heartbeat is listed on its own. +func TestDeadman_ListBreaksDownByFingerprint(t *testing.T) { + s, _ := deadmanTS(t, deadmanCfg()) + + heartbeat(t, s, "fp-a", map[string]string{"cluster": "a"}) + heartbeat(t, s, "fp-b", map[string]string{"cluster": "b"}) + silence(t, s, "fp-b", 2*time.Hour) + + sw := listSwitches(t, s)[0] + if sw["status"] != "dead" { + t.Errorf("one dead cluster makes the switch dead, got %v", sw["status"]) + } + sources := sw["sources"].([]any) + if len(sources) != 2 { + t.Fatalf("expected 2 sources, got %d", len(sources)) + } + first, second := sources[0].(map[string]any), sources[1].(map[string]any) + if first["fingerprint"] != "fp-b" || first["status"] != "dead" || second["status"] != "healthy" { + t.Errorf("the dead source should lead, got %v then %v", first, second) + } +} + +// Every switch keeps its own deadline. +func TestDeadman_TimeoutsArePerSwitch(t *testing.T) { + s, _ := deadmanTS(t, deadmanCfg()) + resp := s.req(t, http.MethodPost, "/api/teams/"+defaultTeam+"/deadman/switches", map[string]any{ + "matcher": "alertname=Edge", "timeout_seconds": 300, + }) + resp.Body.Close() + + heartbeat(t, s, "fp-watchdog", nil) + postWebhook(t, s, []map[string]any{ + amAlert("fp-edge", "Edge", "firing", "2026-05-20T10:00:00Z", zeroTime, nil), + }, `{}:{alertname="Edge"}`) + + // Ten minutes of silence: past the Edge switch's five, inside Watchdog's hour. + silence(t, s, "fp-watchdog", 10*time.Minute) + silence(t, s, "fp-edge", 10*time.Minute) + + got := listSwitches(t, s) + if got[0]["status"] != "healthy" || got[1]["status"] != "dead" { + t.Errorf("want Watchdog healthy and Edge dead, got %v and %v", got[0]["status"], got[1]["status"]) + } +} + +// Deleting is an owner's, is scoped to the team, and leaves what the switch +// already opened alone. +func TestDeadman_DeleteIsScopedToTheTeam(t *testing.T) { + s, _ := deadmanTS(t, deadmanCfg()) + other := newTeam(t, s, "other") + + id := int64(listSwitches(t, s)[0]["id"].(float64)) + + // Another team's owner cannot reach it. + resp := other.call(http.MethodDelete, "/api/teams/"+id64(other.id)+"/deadman/switches/"+id64(id), nil) + resp.Body.Close() + if resp.StatusCode != http.StatusNotFound { + t.Errorf("deleting another team's switch: expected 404, got %d", resp.StatusCode) + } + if got := len(listSwitches(t, s)); got != 1 { + t.Fatalf("the switch should have survived, %d left", got) + } + + resp = s.req(t, http.MethodDelete, "/api/teams/"+defaultTeam+"/deadman/switches/"+id64(id), nil) + resp.Body.Close() + if resp.StatusCode != http.StatusNoContent { + t.Fatalf("deleting: expected 204, got %d", resp.StatusCode) + } + if got := len(listSwitches(t, s)); got != 0 { + t.Errorf("expected no switches, got %d", got) + } +} + +// The environment's defaults are handed out once and then belong to the teams. +func TestDeadman_SeedRunsOnce(t *testing.T) { + s := newTS(t) + cfg := api.ParseDeadmanConfig("alertname=Watchdog", time.Hour, "critical") + + if err := api.SeedDeadmanConfigs(context.Background(), s.db, cfg); err != nil { + t.Fatalf("seed: %v", err) + } + if got := len(listSwitches(t, s)); got != 1 { + t.Fatalf("the first seed should add the default, got %d switches", got) + } + + // The owner deletes it; a restart must not put it back. + id := int64(listSwitches(t, s)[0]["id"].(float64)) + s.req(t, http.MethodDelete, "/api/teams/"+defaultTeam+"/deadman/switches/"+id64(id), nil).Body.Close() + if err := api.SeedDeadmanConfigs(context.Background(), s.db, cfg); err != nil { + t.Fatalf("seed again: %v", err) + } + if got := len(listSwitches(t, s)); got != 0 { + t.Errorf("a second seed resurrected %d switch(es)", got) } } diff --git a/internal/api/router.go b/internal/api/router.go index fd55d25..bf1cba6 100644 --- a/internal/api/router.go +++ b/internal/api/router.go @@ -144,8 +144,9 @@ func NewRouter(db *sql.DB, notify NotifyConfig, cfg config.Config) http.Handler // A team's own dead man's switches: which of its alerts are heartbeats, // and how long a silence has to last before somebody is paged. - r.Get("/api/teams/{teamID}/deadman", handleGetTeamDeadman(db)) - r.Put("/api/teams/{teamID}/deadman", handleSetTeamDeadman(db)) + r.Get("/api/teams/{teamID}/deadman/switches", handleListTeamDeadman(db)) + r.Post("/api/teams/{teamID}/deadman/switches", handleCreateTeamDeadman(db)) + r.Delete("/api/teams/{teamID}/deadman/switches/{switchID}", handleDeleteTeamDeadman(db)) // Integrations: where a team's alerts come in, and the key that says so. r.Get("/api/teams/{teamID}/integrations", handleListIntegrations(db)) diff --git a/internal/api/teams.go b/internal/api/teams.go index 20de78f..5d59872 100644 --- a/internal/api/teams.go +++ b/internal/api/teams.go @@ -539,17 +539,24 @@ func defaultTeamID(ctx context.Context, db *sql.DB) (int64, error) { // A team's dead man's switches // --------------------------------------------------------------------------- -// deadmanResponse is the wire shape of a team's switch configuration. The -// timeout is seconds rather than a duration string, because that is what the -// column holds and what arithmetic is done on; a client renders it. -type deadmanResponse struct { - TeamID int64 `json:"team_id"` - Matchers string `json:"matchers"` +// deadmanSwitchRequest is what creating a switch takes. The timeout is seconds, +// because that is what the column holds and what arithmetic is done on; a client +// renders it. +type deadmanSwitchRequest struct { + Name string `json:"name"` + Matcher string `json:"matcher"` TimeoutSeconds int64 `json:"timeout_seconds"` Severity string `json:"severity"` } -func handleGetTeamDeadman(db *sql.DB) http.HandlerFunc { +// deadmanSeverities are the severities an incident can open at. +var deadmanSeverities = map[string]bool{"critical": true, "error": true, "warning": true, "info": true} + +// handleListTeamDeadman lists a team's switches with what each one's heartbeats +// are doing. A team with none gets an empty list, which is a configuration and +// not an absence: answering 404 would make "off" indistinguishable from "this +// server does not do this". +func handleListTeamDeadman(db *sql.DB) http.HandlerFunc { return func(w http.ResponseWriter, r *http.Request) { teamID, ok := teamParam(w, r) if !ok { @@ -559,27 +566,26 @@ func handleGetTeamDeadman(db *sql.DB) http.HandlerFunc { return } - out := deadmanResponse{TeamID: teamID, Severity: "critical"} - err := db.QueryRowContext(r.Context(), - "SELECT matchers, timeout_seconds, severity FROM deadman_configs WHERE team_id = $1", - teamID).Scan(&out.Matchers, &out.TimeoutSeconds, &out.Severity) - if err != nil && !errors.Is(err, sql.ErrNoRows) { + set, err := deadmanSetForTeam(r.Context(), db, teamID) + if err != nil { + respond(w, http.StatusInternalServerError, errResp("internal error")) + return + } + out, err := deadmanStatuses(r.Context(), db, teamID, set, time.Now()) + if err != nil { respond(w, http.StatusInternalServerError, errResp("internal error")) return } - // A team with no row watches nothing, which is a configuration and not - // an absence: answering 404 would make "off" indistinguishable from - // "this server does not do this". respond(w, http.StatusOK, out) } } -// handleSetTeamDeadman replaces a team's switch configuration. +// handleCreateTeamDeadman adds one switch. // -// Validated by parsing: a matcher string that survives ParseDeadmanConfig with -// nothing usable in it is rejected rather than stored, because a switch that -// silently watches nothing is the failure this feature exists to prevent. -func handleSetTeamDeadman(db *sql.DB) http.HandlerFunc { +// Validated by parsing: a matcher with no alertname is rejected rather than +// stored, because a switch that silently watches nothing is the failure this +// feature exists to prevent. +func handleCreateTeamDeadman(db *sql.DB) http.HandlerFunc { return func(w http.ResponseWriter, r *http.Request) { teamID, ok := teamParam(w, r) if !ok { @@ -589,50 +595,87 @@ func handleSetTeamDeadman(db *sql.DB) http.HandlerFunc { return } - var req struct { - Matchers string `json:"matchers"` - TimeoutSeconds int64 `json:"timeout_seconds"` - Severity string `json:"severity"` - } + var req deadmanSwitchRequest if err := decodeJSON(r, &req); err != nil { respond(w, http.StatusBadRequest, errResp("invalid request body")) return } - req.Matchers = strings.TrimSpace(req.Matchers) + req.Matcher = strings.TrimSpace(req.Matcher) + req.Name = strings.TrimSpace(req.Name) if req.Severity == "" { req.Severity = "critical" } - if req.TimeoutSeconds < 0 { - respond(w, http.StatusBadRequest, errResp("timeout_seconds must not be negative")) + if !deadmanSeverities[req.Severity] { + respond(w, http.StatusBadRequest, errResp("severity must be critical, error, warning or info")) return } - if req.Matchers != "" { - parsed := parseDeadmanQuietly(req.Matchers, time.Duration(req.TimeoutSeconds)*time.Second, req.Severity) - if len(parsed.Matchers) == 0 { - respond(w, http.StatusBadRequest, errResp( - "no usable matchers: each must name an alertname, as in alertname=Watchdog,cluster=prod")) - return - } + if req.TimeoutSeconds <= 0 { + respond(w, http.StatusBadRequest, errResp("timeout_seconds must be positive")) + return + } + if strings.Contains(req.Matcher, ";") { + respond(w, http.StatusBadRequest, errResp("one matcher per switch: add another switch instead of separating with ;")) + return + } + m, err := parseDeadmanMatcher(req.Matcher) + if err != nil { + respond(w, http.StatusBadRequest, errResp( + "unusable matcher ("+err.Error()+"): each must name an alertname, as in alertname=Watchdog,cluster=prod")) + return + } + if req.Name == "" { + req.Name = m.config() + } + if len(req.Name) > 100 { + respond(w, http.StatusBadRequest, errResp("name is too long")) + return } - if _, err := db.ExecContext(r.Context(), ` - INSERT INTO deadman_configs (team_id, matchers, timeout_seconds, severity, updated_at) - VALUES ($1, $2, $3, $4, `+nowEpoch+`) - ON CONFLICT (team_id) DO UPDATE SET - matchers = excluded.matchers, - timeout_seconds = excluded.timeout_seconds, - severity = excluded.severity, - updated_at = excluded.updated_at`, - teamID, req.Matchers, req.TimeoutSeconds, req.Severity); err != nil { + var id int64 + if err := db.QueryRowContext(r.Context(), ` + INSERT INTO deadman_switches (team_id, name, matcher, timeout_seconds, severity) + VALUES ($1, $2, $3, $4, $5) RETURNING id`, + teamID, req.Name, m.config(), req.TimeoutSeconds, req.Severity).Scan(&id); err != nil { respond(w, http.StatusInternalServerError, errResp("internal error")) return } - respond(w, http.StatusOK, deadmanResponse{ - TeamID: teamID, - Matchers: req.Matchers, - TimeoutSeconds: req.TimeoutSeconds, - Severity: req.Severity, + respond(w, http.StatusCreated, deadmanSwitchStatus{ + ID: id, Name: req.Name, Matcher: m.config(), + TimeoutSeconds: req.TimeoutSeconds, Severity: req.Severity, + Status: switchDormant, Sources: []deadmanSource{}, }) } } + +// handleDeleteTeamDeadman removes a switch. An incident it already opened stays +// open until somebody resolves it: deleting the switch says "stop watching", not +// "the problem is gone". +func handleDeleteTeamDeadman(db *sql.DB) http.HandlerFunc { + return func(w http.ResponseWriter, r *http.Request) { + teamID, ok := teamParam(w, r) + if !ok { + return + } + if !requireTeamOwner(w, r, teamID) { + return + } + switchID, err := strconv.ParseInt(chi.URLParam(r, "switchID"), 10, 64) + if err != nil { + respond(w, http.StatusBadRequest, errResp("invalid switch id")) + return + } + + res, err := db.ExecContext(r.Context(), + "DELETE FROM deadman_switches WHERE id = $1 AND team_id = $2", switchID, teamID) + if err != nil { + respond(w, http.StatusInternalServerError, errResp("internal error")) + return + } + if n, _ := res.RowsAffected(); n == 0 { + respond(w, http.StatusNotFound, errResp("switch not found")) + return + } + w.WriteHeader(http.StatusNoContent) + } +} diff --git a/internal/db/migrations/009_deadman_switches.sql b/internal/db/migrations/009_deadman_switches.sql new file mode 100644 index 0000000..e693d46 --- /dev/null +++ b/internal/db/migrations/009_deadman_switches.sql @@ -0,0 +1,54 @@ +-- Dead man's switches become rows of their own. +-- +-- 004 kept a team's switches in one string with one timeout and one severity, +-- which was enough to configure them and not enough to show them: there was no +-- thing to list, nothing to hang a status on, and every switch in a team had to +-- share a deadline. A row per switch gives each its own name, matcher, timeout +-- and severity, and gives the Team → Switches page something to be a list of. +-- +-- The matcher keeps the syntax the string used, one matcher per row: +-- `alertname=Watchdog,cluster=prod`. The unit of monitoring is still the +-- fingerprint, so a matcher that many clusters satisfy is still one switch row +-- watching several independent heartbeats. +CREATE TABLE deadman_switches ( + id BIGSERIAL PRIMARY KEY, + team_id BIGINT NOT NULL REFERENCES teams(id) ON DELETE CASCADE, + + -- What the owner calls it. Defaults to the matcher when they do not say. + name TEXT NOT NULL, + + -- "," separates the label conditions, "=" is exact equality, and alertname is + -- mandatory: it is what keeps the sweeper's candidate query on an index. + matcher TEXT NOT NULL, + + -- Seconds of silence before the switch is declared dead. Never zero: a switch + -- that cannot fire is deleted, not disabled. + timeout_seconds BIGINT NOT NULL CHECK (timeout_seconds > 0), + + -- The severity its incidents open at. See 004 for why they carry their own. + severity TEXT NOT NULL DEFAULT 'critical', + + created_at BIGINT NOT NULL DEFAULT FLOOR(EXTRACT(EPOCH FROM now()))::bigint +); + +CREATE INDEX deadman_switches_team_idx ON deadman_switches (team_id); + +-- Carry every team's configuration over, one row per matcher. A team whose +-- timeout was zero had switches turned off, which is now "no rows". +INSERT INTO deadman_switches (team_id, name, matcher, timeout_seconds, severity) +SELECT c.team_id, btrim(m), btrim(m), c.timeout_seconds, c.severity +FROM deadman_configs c, + LATERAL regexp_split_to_table(c.matchers, ';') AS m +WHERE c.timeout_seconds > 0 + AND btrim(m) <> '' +ORDER BY c.team_id; + +-- The server seeds environment defaults into teams once, and remembers that it +-- did. An install that had a row per team was already seeded; without this +-- marker the first start after upgrading would seed teams that had switched +-- theirs off. +INSERT INTO settings (key, value) +SELECT 'deadman_seeded', '1' +WHERE EXISTS (SELECT 1 FROM deadman_configs); + +DROP TABLE deadman_configs; diff --git a/internal/web/static/app.css b/internal/web/static/app.css index 422d241..efb3c47 100644 --- a/internal/web/static/app.css +++ b/internal/web/static/app.css @@ -359,7 +359,11 @@ input:focus, textarea:focus { outline: none; border-color: var(--accent); box-sh .badge.st-triggered, .badge.st-firing { background: var(--crit-soft); color: var(--crit); } .badge.st-acknowledged { background: var(--warn-soft); color: var(--warn); } .badge.st-snoozed { background: var(--snooze-soft); color: var(--snooze); } -.badge.st-resolved { background: var(--ok-soft); color: var(--ok); } +.badge.st-resolved, .badge.st-healthy { background: var(--ok-soft); color: var(--ok); } +.badge.st-dead { background: var(--crit-soft); color: var(--crit); } +/* Dormant is the plain badge on purpose: nothing has gone wrong and nothing has + gone right, which is what the muted default already says. */ +.badge.st-dormant { background: var(--surface-2); color: var(--muted); } .badge.sev-critical { background: var(--crit-soft); color: var(--crit); } .badge.sev-warning { background: var(--warn-soft); color: var(--warn); } .badge.sev-info { background: var(--info-soft); color: var(--info); } @@ -691,6 +695,18 @@ kbd { .disabled-row td { opacity: 0.55; } .btn-sm.danger { color: var(--crit); border-color: var(--crit-soft); } +/* --- dead man's switches ------------------------------------------------- + Six columns do not fit a phone, so the table scrolls inside its card rather + than the page. A heartbeat under a switch with several is indented, the way + the escalation ladder indents its levels. */ +.card-head { display: flex; align-items: center; justify-content: space-between; gap: 12px; flex-wrap: wrap; } +.table-scroll { overflow-x: auto; margin-top: 12px; } +.switch-table th, .switch-table td { white-space: nowrap; } +.switch-table td:nth-child(2) { white-space: normal; min-width: 12em; } +.switch-table .source-row td { border-bottom-style: dashed; } +.switch-table .source-row td:first-child { padding-left: 16px; } +.source-labels { display: flex; flex-wrap: wrap; gap: 4px; align-items: center; } + .inline-form { display: flex; gap: 8px; margin-top: 12px; } .inline-form input { flex: 1; min-width: 0; } diff --git a/internal/web/static/js/api.js b/internal/web/static/js/api.js index 396b5bc..8bf6e19 100644 --- a/internal/web/static/js/api.js +++ b/internal/web/static/js/api.js @@ -136,8 +136,11 @@ export const createIntegration = (id, name) => export const deleteIntegration = (id, integrationID) => call('DELETE', `/teams/${id}/integrations/${integrationID}`); -export const deadman = (id) => call('GET', `/teams/${id}/deadman`); -export const setDeadman = (id, body) => call('PUT', `/teams/${id}/deadman`, { body }); +export const deadmanSwitches = (id) => call('GET', `/teams/${id}/deadman/switches`); +export const createDeadmanSwitch = (id, body) => + call('POST', `/teams/${id}/deadman/switches`, { body }); +export const deleteDeadmanSwitch = (id, switchID) => + call('DELETE', `/teams/${id}/deadman/switches/${switchID}`); export const escalation = (id) => call('GET', `/teams/${id}/escalation`); export const setEscalation = (id, body) => call('PUT', `/teams/${id}/escalation`, { body }); diff --git a/internal/web/static/js/team.js b/internal/web/static/js/team.js index 1715ed8..9e51116 100644 --- a/internal/web/static/js/team.js +++ b/internal/web/static/js/team.js @@ -18,9 +18,9 @@ // than no form, but it is not the thing enforcing anything. import * as api from './api.js'; -import { h, clear, spinner, confirm, icon, openSheet, closeSheet, menuCard } from './ui.js'; +import { h, clear, spinner, confirm, icon, openSheet, closeSheet, menuCard, badge, labelChip } from './ui.js'; import { state, currentTeam, users as allUsers, myID } from './state.js'; -import { isoDate, addDays, mondayOf, initial } from './format.js'; +import { isoDate, addDays, mondayOf, initial, ago, when, duration } from './format.js'; const view = () => document.getElementById('view-team'); @@ -111,14 +111,14 @@ async function load(id) { return { members, escalation }; } if (tab === 'sources') return { integrations: await api.integrations(id) }; - if (tab === 'deadman') return { deadman: await api.deadman(id) }; + if (tab === 'deadman') return { deadman: await api.deadmanSwitches(id) }; const grid = gridDays(); const [members, integrations, escalation, deadman, schedule] = await Promise.all([ api.teamMembers(id), api.integrations(id), api.escalation(id), - api.deadman(id), + api.deadmanSwitches(id), api.schedule(id, isoDate(grid.start), isoDate(addDays(grid.start, grid.count - 1))), ]); return { members, integrations, escalation, deadman, schedule }; @@ -203,8 +203,8 @@ function overview() { const levels = (data.escalation?.levels || []).length; const keys = (data.integrations || []).length; const unused = (data.integrations || []).filter((i) => !i.last_used_at).length; - const switches = (data.deadman?.matchers || '') - .split(';').map((x) => x.trim()).filter(Boolean).length; + const switches = (data.deadman || []).length; + const dead = (data.deadman || []).filter((s) => s.status === 'dead').length; return h('div', { class: 'overview-menu' }, menuCard('/team/rota', 'Rota', null, @@ -220,7 +220,9 @@ function overview() { ? (unused ? `${unused} of them never used.` : 'All in use.') : 'No key yet, so nothing can reach this team.'), menuCard('/team/deadman', 'Dead man’s switches', switches || null, - switches ? 'Alerts whose absence opens an incident.' : 'Nothing watched.'), + switches + ? (dead ? `${dead} of them silent.` : 'All quiet, as they should be.') + : 'Nothing watched.'), ); } @@ -650,47 +652,146 @@ function newIntegrationForm() { // --- dead man's switches --------------------------------------------------- +const SWITCH_STATUS = { + healthy: { label: 'Healthy', hint: 'Heard from within its timeout.' }, + dead: { label: 'Dead', hint: 'Silent for longer than its timeout.' }, + dormant: { label: 'Dormant', hint: 'Nothing has matched yet, so there is nothing to lose.' }, +}; + +function switchBadge(status) { + const s = SWITCH_STATUS[status] || SWITCH_STATUS.dormant; + const el = badge(s.label, `st-${status}`); + el.title = s.hint; + return el; +} + +const timeCell = (iso) => iso + ? h('span', { title: when(iso), text: ago(iso) }) + : h('span', { class: 'muted', text: 'never' }); + +// When it last opened an incident. An incident that is still open is a link, +// because that is the thing somebody looking at a red row wants next. +const triggeredCell = (iso, incidentID) => { + if (!iso) return h('span', { class: 'muted', text: 'never' }); + return incidentID + ? h('a', { href: `/incidents/${incidentID}`, title: when(iso) }, `#${incidentID} · ${ago(iso)}`) + : h('span', { title: when(iso), text: ago(iso) }); +}; + +function switchRows(sw) { + const main = h('tr', {}, + h('td', {}, switchBadge(sw.status)), + h('td', {}, + h('strong', { text: sw.name }), + sw.name !== sw.matcher && h('div', { class: 'muted small' }, h('code', { text: sw.matcher }))), + h('td', { class: 'muted small' }, timeCell(sw.last_heartbeat_at)), + h('td', { class: 'muted small' }, triggeredCell(sw.last_triggered_at, sw.open_incident_id)), + h('td', { class: 'muted small', text: duration(sw.timeout_seconds * 1000) }), + h('td', {}, isOwner() && h('button', { + class: 'btn-sm danger', type: 'button', text: 'Remove', + onclick: async () => { + if (!(await confirm({ + title: `Remove ${sw.name}?`, + text: 'It stops being watched. An incident it already opened stays open until it is resolved.', + confirmLabel: 'Remove', + danger: true, + }))) return; + act(() => api.deleteDeadmanSwitch(teamID, sw.id)); + }, + })), + ); + + // One heartbeat is the switch's own times; several are worth telling apart, + // since a live cluster must not hide a dead one. + const sources = sw.sources.length > 1 + ? sw.sources.map((src) => h('tr', { class: 'source-row' }, + h('td', {}, switchBadge(src.status)), + h('td', { class: 'source-labels' }, + ...Object.entries(src.labels || {}) + .filter(([k]) => k !== 'alertname') + .map(([k, v]) => labelChip(k, v)), + !Object.keys(src.labels || {}).some((k) => k !== 'alertname') + && h('code', { class: 'small', text: src.fingerprint })), + h('td', { class: 'muted small' }, timeCell(src.last_heartbeat_at)), + h('td', { class: 'muted small' }, triggeredCell(src.last_triggered_at, src.incident_id)), + h('td'), h('td'))) + : []; + return [main, ...sources]; +} + function deadmanCard() { - const d = data.deadman || {}; - const matchers = h('input', { - type: 'text', value: d.matchers || '', placeholder: 'alertname=Watchdog', - class: 'wide', - }); - const timeout = h('input', { - type: 'number', min: '0', class: 'setting-value', - value: String(Math.round((d.timeout_seconds || 0) / 60)), - }); - const severity = h('select', {}, - ...['critical', 'error', 'warning', 'info'].map((s) => - h('option', { value: s, text: s, selected: (d.severity || 'critical') === s }))); - - const form = h('form', { class: 'stacked-form' }, - h('label', {}, 'Heartbeat alerts ', matchers), - h('label', {}, 'Declare dead after ', timeout, ' minutes of silence'), - h('label', {}, 'Open the incident at severity ', severity), - h('button', { class: 'btn', type: 'submit', text: 'Save switches' })); - - form.addEventListener('submit', (e) => { - e.preventDefault(); - act(() => api.setDeadman(teamID, { - matchers: matchers.value.trim(), - timeout_seconds: Number(timeout.value) * 60, - severity: severity.value, - })); - }); + const switches = data.deadman || []; return h('div', { class: 'card' }, - h('h2', { text: 'Dead man’s switches' }), + h('div', { class: 'card-head' }, + h('h2', { text: 'Dead man’s switches' }), + isOwner() && h('button', { + class: 'btn', type: 'button', text: 'New switch', onclick: openNewSwitch, + })), h('p', { class: 'muted small' }, 'Alerts whose ABSENCE is the signal. Receiving one opens nothing; going ', - 'quiet for longer than the timeout opens an incident. ', - h('code', { text: 'alertname=Watchdog,cluster=prod; alertname=EdgeHeartbeat' }), - ' — semicolons separate switches, commas separate conditions, and every ', - 'switch must name an alertname. Leave empty to watch nothing.'), - isOwner() ? form : h('p', { class: 'muted', text: d.matchers || 'Nothing watched.' }), + 'quiet for longer than the switch’s timeout opens an incident.'), + switches.length + ? h('div', { class: 'table-scroll' }, + h('table', { class: 'admin-table switch-table' }, + h('thead', {}, h('tr', {}, + h('th', { text: 'Status' }), h('th', { text: 'Switch' }), + h('th', { text: 'Last heartbeat' }), h('th', { text: 'Last triggered' }), + h('th', { text: 'Silent after' }), h('th'))), + h('tbody', {}, switches.flatMap(switchRows)))) + : h('p', { class: 'muted', text: 'Nothing watched.' }), ); } +// The form lives in the sheet, not on the page: most visits are to look at the +// list, and a form that is always open is the page this replaced. +function openNewSwitch() { + const name = h('input', { type: 'text', placeholder: 'Prod Watchdog', autofocus: true }); + const matcher = h('input', { + type: 'text', placeholder: 'alertname=Watchdog,cluster=prod', class: 'wide', required: true, + }); + const timeout = h('input', { + type: 'number', min: '1', value: '15', class: 'setting-value', required: true, + }); + const severity = h('select', {}, + ...['critical', 'error', 'warning', 'info'].map((s) => h('option', { value: s, text: s }))); + const problem = h('p', { class: 'load-error', hidden: true }); + + const form = h('form', { class: 'stacked-form' }, + h('label', {}, 'Name (optional) ', name), + h('label', {}, 'Heartbeat alert ', matcher), + h('p', { class: 'muted small' }, + 'Conditions are ', h('code', { text: 'label=value' }), ' separated by commas, and one ', + 'must be ', h('code', { text: 'alertname' }), '. Every distinct label set that ', + 'matches is watched on its own.'), + h('label', {}, 'Declare dead after ', timeout, ' minutes of silence'), + h('label', {}, 'Open the incident at severity ', severity), + problem, + h('div', { class: 'sheet-actions' }, + h('button', { class: 'btn', type: 'button', text: 'Cancel', onclick: () => closeSheet(false) }), + h('button', { class: 'btn btn-primary', type: 'submit', text: 'Add switch' }))); + + form.addEventListener('submit', async (e) => { + e.preventDefault(); + try { + await api.createDeadmanSwitch(teamID, { + name: name.value.trim(), + matcher: matcher.value.trim(), + timeout_seconds: Math.round(Number(timeout.value) * 60), + severity: severity.value, + }); + } catch (err) { + problem.textContent = err.message; + problem.hidden = false; + return; + } + closeSheet(true); + refresh(); + }); + + openSheet(() => [h('h2', { class: 'sheet-title', text: 'New switch' }), form]); +} + // --- members --------------------------------------------------------------- function membersCard() {