Compare commits

...

2 Commits

Author SHA1 Message Date
Niklas Ye e8d45f9d3d Set the chart's placeholder version to 0.24.0
CI / chart (push) Successful in 1s
CI / security (push) Successful in 16s
CI / test (push) Successful in 2m55s
Release / test (push) Successful in 4s
Release / chart (push) Successful in 2s
Release / binaries (push) Successful in 24s
Release / image (push) Successful in 1m0s
Release / scan-image (push) Successful in 4s
Cosmetic: make helm-package sets the published version and appVersion
from the tag, so these two fields decide nothing (see the comment
above them). Kept in step anyway, same as 3ee8583 and d2cdcc9, so a
tree heading for v0.24.0 doesn't say 0.23.0.
2026-09-25 23:40:55 +02:00
Niklas Ye f3918b863c List dead man's switches with their status on Team -> Switches
The page was a bare form: it did not say which switches existed or
whether they were alive. It now lists them, each with a Healthy, Dead or
Dormant badge, when its heartbeat was last heard and when it last opened
an incident (linked while that incident is open). A matcher that several
clusters satisfy is broken down per cluster, since a live cluster must
not hide a dead one. The form moved into a "New switch" sheet, and each
row has a Remove with a confirm.

That needed a switch to be a thing, so switches are rows now
(migration 009) with their own name, matcher, timeout and severity,
instead of one string with one team-wide timeout in deadman_configs.
Existing configuration is split into one row per matcher; a team whose
timeout was zero simply has none. The sweeper and the status endpoint
share one death rule (deadmanAlert.dead), so the page cannot disagree
with the pager. Incident group keys are unchanged, so incidents that
are open across the upgrade keep working.

The environment defaults (TERDUT_DEADMAN_*) are seeded into teams once
per install, recorded in settings, so a team that deletes its last
switch does not get it back on the next restart. Installs that already
had per-team rows are marked as seeded by the migration.

Removing a switch stops the watching but leaves an incident it already
opened open until someone resolves it.

API: GET/PUT /api/teams/{id}/deadman are replaced by
GET/POST /deadman/switches and DELETE /deadman/switches/{switchID}.
terdut-tui does not call them, so nothing to mirror there.
2026-09-25 23:40:55 +02:00
12 changed files with 870 additions and 315 deletions
+23 -14
View File
@@ -513,19 +513,27 @@ nothing unless something downstream notices it stop. That is what
`TERDUT_DEADMAN_MATCHERS` defaults to. `TERDUT_DEADMAN_MATCHERS` defaults to.
**Switches belong to a team**, which decides which of its own alerts are **Switches belong to a team**, which decides which of its own alerts are
heartbeats and how long a silence has to last. An owner sets them through heartbeats and how long a silence has to last. Each **switch** is a row of its
`PUT /api/teams/{teamID}/deadman`; a missed heartbeat opens an incident in the own — a name, one matcher, a timeout and a severity — so switches in one team
team whose integration received it. can have different deadlines. An owner adds and removes them on **Team →
Switches**, which lists each with a status (**healthy**, **dead**, or
**dormant** until its first heartbeat), when it was last heard from, and when it
last opened an incident; a matcher that several clusters satisfy is broken down
per cluster. The API is `POST`/`DELETE /api/teams/{teamID}/deadman/switches`. A
missed heartbeat opens an incident in the team whose integration received it.
Removing a switch stops the watching; an incident it already opened stays open
until somebody resolves it.
The environment variables are the starting point, not the setting: at startup The environment variables are the starting point, not the setting: the **first**
every team **without** a configuration of its own is given one from them, and an time the server starts, every team is given a switch per default matcher from
owner's later edit is never overwritten by a redeploy. A team created after them, once. After that a team's switches are its own — an owner's edit or
that starts watching nothing until its owner says otherwise — inheriting an deletion is never put back by a redeploy. A team created later starts watching
install-wide heartbeat would page a new team about a source it has never heard nothing until its owner says otherwise — inheriting an install-wide heartbeat
of. would page a new team about a source it has never heard of.
A matcher is a set of exact label conditions, one of which must be the A matcher is a set of exact label conditions, one of which must be the
`alertname`, in the same format the environment variable uses: `alertname`, in the format the environment variable uses (one matcher per switch; the
variable takes several, separated by `;`):
``` ```
alertname=Watchdog,cluster=prod; alertname=EdgeHeartbeat alertname=Watchdog,cluster=prod; alertname=EdgeHeartbeat
@@ -718,8 +726,9 @@ administrator who is not in the team gets the same `404` as anybody else.
| `DELETE` | `/api/teams/{teamID}/invites/{inviteID}` | **owner** | Revoke a link before it expires | | `DELETE` | `/api/teams/{teamID}/invites/{inviteID}` | **owner** | Revoke a link before it expires |
| `GET` | `/api/teams/{teamID}/escalation` | member | The team's [escalation ladder](#escalation) `{repeat_count, fallback_topic, levels[]}`. Empty levels means the team has none | | `GET` | `/api/teams/{teamID}/escalation` | member | The team's [escalation ladder](#escalation) `{repeat_count, fallback_topic, levels[]}`. Empty levels means the team has none |
| `PUT` | `/api/teams/{teamID}/escalation` | **owner** | Replace it wholesale. `400` for a level with no targets or no timeout — a rung that pages nobody is a silence with a number on it | | `PUT` | `/api/teams/{teamID}/escalation` | **owner** | Replace it wholesale. `400` for a level with no targets or no timeout — a rung that pages nobody is a silence with a number on it |
| `GET` | `/api/teams/{teamID}/deadman` | member | The team's [dead man's switch](#dead-mans-switch) configuration `{matchers, timeout_seconds, severity}` | | `GET` | `/api/teams/{teamID}/deadman/switches` | member | The team's [dead man's switches](#dead-mans-switch), each `{id, name, matcher, timeout_seconds, severity, status, last_heartbeat_at, last_triggered_at, open_incident_id, sources[]}`. `status` is `healthy`, `dead` or `dormant`; `sources` has one entry per heartbeat fingerprint. Empty when the team watches nothing |
| `PUT` | `/api/teams/{teamID}/deadman` | **owner** | Replace it. `400` when no matcher names an `alertname`, because a switch that silently watches nothing is the failure this feature exists to prevent | | `POST` | `/api/teams/{teamID}/deadman/switches` | **owner** | Add one: `{name?, matcher, timeout_seconds, severity?}`. `400` when the matcher names no `alertname` or holds several, or the timeout is not positive — a switch that silently watches nothing is the failure this feature exists to prevent |
| `DELETE` | `/api/teams/{teamID}/deadman/switches/{switchID}` | **owner** | Stop watching. An incident it opened stays open. `404` for a switch of another team |
### Notifications ### Notifications
@@ -964,8 +973,8 @@ What changes, and will need attention:
**Dead man's switches moved too.** `TERDUT_DEADMAN_MATCHERS`, `_TIMEOUT` and **Dead man's switches moved too.** `TERDUT_DEADMAN_MATCHERS`, `_TIMEOUT` and
`_SEVERITY` are no longer the setting; they are the default each existing team `_SEVERITY` are no longer the setting; they are the default each existing team
is seeded with at startup, after which an owner edits them per team through is seeded with at startup, after which an owner manages them per team through
`PUT /api/teams/{teamID}/deadman` and a redeploy never overwrites that. `/api/teams/{teamID}/deadman/switches` and a redeploy never overwrites that.
Nothing else about an incident changes, and incidents never move between teams: Nothing else about an incident changes, and incidents never move between teams:
an alert belongs to whichever team's key it arrived on. an alert belongs to whichever team's key it arrived on.
+2 -2
View File
@@ -15,5 +15,5 @@ type: application
# appVersion and image.tag in values.yaml no longer agree, and that is not an oversight: # appVersion and image.tag in values.yaml no longer agree, and that is not an oversight:
# image.tag stays "latest", which is what a local install actually pulls. appVersion is # image.tag stays "latest", which is what a local install actually pulls. appVersion is
# metadata and drives nothing. # metadata and drives nothing.
version: 0.23.0 version: 0.24.0
appVersion: "v0.23.0" appVersion: "v0.24.0"
+2 -2
View File
@@ -124,7 +124,7 @@ func ingest(ctx context.Context, db *sql.DB, notify NotifyConfig, teamID int64,
// Which arriving alerts are heartbeats is the team's own answer, read // Which arriving alerts are heartbeats is the team's own answer, read
// inside the transaction so an owner editing it mid-payload cannot split // inside the transaction so an owner editing it mid-payload cannot split
// one webhook across two interpretations. // one webhook across two interpretations.
deadman, err := deadmanConfigForTeam(ctx, tx, teamID) deadman, err := deadmanSetForTeam(ctx, tx, teamID)
if err != nil { if err != nil {
return err return err
} }
@@ -186,7 +186,7 @@ func ingest(ctx context.Context, db *sql.DB, notify NotifyConfig, teamID int64,
// upsertAlerts stores each alert of a payload and reports what changed. Payloads // upsertAlerts stores each alert of a payload and reports what changed. Payloads
// the ordering guard rejected are left out entirely. // the ordering guard rejected are left out entirely.
func upsertAlerts(ctx context.Context, tx *sql.Tx, deadman DeadmanConfig, teamID int64, alerts []amAlert) ([]ingested, error) { func upsertAlerts(ctx context.Context, tx *sql.Tx, deadman deadmanSet, teamID int64, alerts []amAlert) ([]ingested, error) {
now := time.Now().Unix() now := time.Now().Unix()
accepted := make([]ingested, 0, len(alerts)) accepted := make([]ingested, 0, len(alerts))
+8 -10
View File
@@ -88,27 +88,25 @@ func newDeadmanTS(t *testing.T, deadman api.DeadmanConfig, notify ...api.NotifyC
return s return s
} }
// setTeamDeadman configures the default team's switches over the API, rendering // setTeamDeadman gives the default team one switch per configured matcher, over
// the matchers back into the string form the endpoint takes. // the API, the way an owner would add them.
func setTeamDeadman(t *testing.T, s *ts, cfg api.DeadmanConfig) { func setTeamDeadman(t *testing.T, s *ts, cfg api.DeadmanConfig) {
t.Helper() t.Helper()
matchers := make([]string, 0, len(cfg.Matchers))
for _, m := range cfg.Matchers { for _, m := range cfg.Matchers {
parts := []string{"alertname=" + m.Name} parts := []string{"alertname=" + m.Name}
for k, v := range m.Labels { for k, v := range m.Labels {
parts = append(parts, k+"="+v) parts = append(parts, k+"="+v)
} }
sort.Strings(parts[1:]) sort.Strings(parts[1:])
matchers = append(matchers, strings.Join(parts, ",")) resp := s.req(t, http.MethodPost, "/api/teams/"+defaultTeam+"/deadman/switches", map[string]any{
} "matcher": strings.Join(parts, ","),
resp := s.req(t, http.MethodPut, "/api/teams/"+defaultTeam+"/deadman", map[string]any{
"matchers": strings.Join(matchers, "; "),
"timeout_seconds": int64(cfg.Timeout.Seconds()), "timeout_seconds": int64(cfg.Timeout.Seconds()),
"severity": cfg.Severity, "severity": cfg.Severity,
}) })
defer resp.Body.Close() resp.Body.Close()
if resp.StatusCode != http.StatusOK { if resp.StatusCode != http.StatusCreated {
t.Fatalf("configure the team's dead man's switches: %d", resp.StatusCode) t.Fatalf("add a dead man's switch: %d", resp.StatusCode)
}
} }
} }
+354 -174
View File
@@ -4,6 +4,8 @@ import (
"context" "context"
"database/sql" "database/sql"
"encoding/json" "encoding/json"
"errors"
"fmt"
"log" "log"
"sort" "sort"
"strings" "strings"
@@ -39,6 +41,17 @@ func (m DeadmanMatcher) String() string {
return m.Name + " (" + strings.Join(parts, ", ") + ")" return m.Name + " (" + strings.Join(parts, ", ") + ")"
} }
// config renders the matcher in the form parseDeadmanMatcher reads, which is
// what a switch row stores: `alertname=Watchdog,cluster=prod`.
func (m DeadmanMatcher) config() string {
parts := make([]string, 0, len(m.Labels))
for k, v := range m.Labels {
parts = append(parts, k+"="+v)
}
sort.Strings(parts)
return strings.Join(append([]string{"alertname=" + m.Name}, parts...), ",")
}
// matches reports whether an alert's labels satisfy every condition. // matches reports whether an alert's labels satisfy every condition.
func (m DeadmanMatcher) matches(labels map[string]string) bool { func (m DeadmanMatcher) matches(labels map[string]string) bool {
if labels["alertname"] != m.Name { if labels["alertname"] != m.Name {
@@ -52,12 +65,10 @@ func (m DeadmanMatcher) matches(labels map[string]string) bool {
return true return true
} }
// DeadmanConfig inverts the handling of the alerts it matches: receiving one // DeadmanConfig is the server-wide default a team's switches are seeded from:
// opens nothing, and the absence of one opens an incident. // the environment's matchers, timeout and severity. Switches themselves are rows
// // of a team's own — see DeadmanSwitch — and this is only how a fresh install
// The unit of monitoring is the fingerprint, not the matcher — two clusters // starts out.
// sending the same heartbeat alertname are two independent switches, so one
// healthy cluster cannot mask a dead one.
type DeadmanConfig struct { type DeadmanConfig struct {
Matchers []DeadmanMatcher Matchers []DeadmanMatcher
@@ -76,41 +87,81 @@ type DeadmanConfig struct {
// enabled reports whether there is anything to watch. // enabled reports whether there is anything to watch.
func (c DeadmanConfig) enabled() bool { return c.Timeout > 0 && len(c.Matchers) > 0 } func (c DeadmanConfig) enabled() bool { return c.Timeout > 0 && len(c.Matchers) > 0 }
// match returns the first matcher an alert satisfies. // DeadmanSwitch inverts the handling of the alerts it matches: receiving one
func (c DeadmanConfig) match(labels map[string]string) (DeadmanMatcher, bool) { // opens nothing, and the absence of one opens an incident.
if !c.enabled() { //
return DeadmanMatcher{}, false // The unit of monitoring is the fingerprint, not the switch — two clusters
} // sending the same heartbeat alertname are two independent heartbeats under one
for _, m := range c.Matchers { // switch, so one healthy cluster cannot mask a dead one.
if m.matches(labels) { type DeadmanSwitch struct {
return m, true ID int64
} Name string
} Matcher DeadmanMatcher
return DeadmanMatcher{}, false
// Timeout is how long a heartbeat may go unheard before it is declared dead.
Timeout time.Duration
// Severity is what the incident opens at.
Severity string
} }
// isDeadman is match without the matcher, for the ingest path. // deadmanSet is one team's switches.
func (c DeadmanConfig) isDeadman(labels map[string]string) bool { type deadmanSet []DeadmanSwitch
_, ok := c.match(labels)
// match returns the first switch an alert satisfies.
func (d deadmanSet) match(labels map[string]string) (DeadmanSwitch, bool) {
for _, sw := range d {
if sw.Matcher.matches(labels) {
return sw, true
}
}
return DeadmanSwitch{}, false
}
// isDeadman is match without the switch, for the ingest path.
func (d deadmanSet) isDeadman(labels map[string]string) bool {
_, ok := d.match(labels)
return ok return ok
} }
// names lists the distinct alertnames worth loading from the database. // names lists the distinct alertnames worth loading from the database.
func (c DeadmanConfig) names() []string { func (d deadmanSet) names() []string {
seen := map[string]bool{} seen := map[string]bool{}
out := make([]string, 0, len(c.Matchers)) out := make([]string, 0, len(d))
for _, m := range c.Matchers { for _, sw := range d {
if !seen[m.Name] { if !seen[sw.Matcher.Name] {
seen[m.Name] = true seen[sw.Matcher.Name] = true
out = append(out, m.Name) out = append(out, sw.Matcher.Name)
} }
} }
return out return out
} }
// parseDeadmanMatcher reads one matcher from its configured form: "," separates
// the conditions and "=" is exact label equality — `alertname=Watchdog,cluster=prod`.
// The error says what is wrong with it, in words a form can show.
func parseDeadmanMatcher(entry string) (DeadmanMatcher, error) {
m := DeadmanMatcher{Labels: map[string]string{}}
for _, cond := range strings.Split(strings.TrimSpace(entry), ",") {
k, v, ok := strings.Cut(cond, "=")
k, v = strings.TrimSpace(k), strings.TrimSpace(v)
if !ok || k == "" || v == "" {
return DeadmanMatcher{}, fmt.Errorf("%q is not label=value", strings.TrimSpace(cond))
}
if k == "alertname" {
m.Name = v
continue
}
m.Labels[k] = v
}
if m.Name == "" {
return DeadmanMatcher{}, errors.New("no alertname condition")
}
return m, nil
}
// ParseDeadmanConfig reads the matcher list from its configured form: // ParseDeadmanConfig reads the matcher list from its configured form:
// ";" separates matchers, "," separates the conditions within one, and "=" is // ";" separates matchers, and each is parsed as parseDeadmanMatcher does.
// exact label equality — `alertname=Watchdog,cluster=prod; alertname=Heartbeat`.
// //
// A malformed or alertname-less entry is dropped rather than fatal, following // A malformed or alertname-less entry is dropped rather than fatal, following
// config.duration's rule that one bad tuning knob should not take the server // config.duration's rule that one bad tuning knob should not take the server
@@ -125,28 +176,9 @@ func ParseDeadmanConfig(matchers string, timeout time.Duration, severity string)
if entry == "" { if entry == "" {
continue continue
} }
m, err := parseDeadmanMatcher(entry)
m := DeadmanMatcher{Labels: map[string]string{}} if err != nil {
malformed := false log.Printf("deadman: ignoring matcher %q: %v", entry, err)
for _, cond := range strings.Split(entry, ",") {
k, v, ok := strings.Cut(cond, "=")
k, v = strings.TrimSpace(k), strings.TrimSpace(v)
if !ok || k == "" || v == "" {
log.Printf("deadman: ignoring matcher %q: %q is not label=value", entry, strings.TrimSpace(cond))
malformed = true
break
}
if k == "alertname" {
m.Name = v
continue
}
m.Labels[k] = v
}
if malformed {
continue
}
if m.Name == "" {
log.Printf("deadman: ignoring matcher %q: no alertname condition", entry)
continue continue
} }
cfg.Matchers = append(cfg.Matchers, m) cfg.Matchers = append(cfg.Matchers, m)
@@ -162,23 +194,35 @@ func ParseDeadmanConfig(matchers string, timeout time.Duration, severity string)
for _, m := range cfg.Matchers { for _, m := range cfg.Matchers {
rendered = append(rendered, m.String()) rendered = append(rendered, m.String())
} }
log.Printf("deadman: watching %s, timeout %s, severity %s", log.Printf("deadman: default for new teams: %s, timeout %s, severity %s",
strings.Join(rendered, "; "), timeout, severity) strings.Join(rendered, "; "), timeout, severity)
} }
return cfg return cfg
} }
// deadmanAlert is one switch: the alert row carrying its last heartbeat. // deadmanAlert is one heartbeat: the alert row carrying its last sighting, and
// the switch that claimed it.
type deadmanAlert struct { type deadmanAlert struct {
id int64 id int64
teamID int64 teamID int64
fingerprint string fingerprint string
labels map[string]string labels map[string]string
matcher DeadmanMatcher sw DeadmanSwitch
resolved bool resolved bool
receivedAt int64 receivedAt int64
} }
// dead is the one rule for a silent heartbeat, shared by the sweeper that pages
// on it and the status the Switches page shows, so the page cannot disagree
// with the pager.
//
// An explicit resolved from Alertmanager is a stronger death signal than mere
// absence: the sender is telling us the heartbeat stopped, so there is nothing
// left to wait out.
func (a deadmanAlert) dead(now time.Time) bool {
return a.resolved || a.receivedAt < now.Add(-a.sw.Timeout).Unix()
}
// groupKey is the switch's identity as an incident. Per fingerprint, so each // groupKey is the switch's identity as an incident. Per fingerprint, so each
// source is tracked on its own. // source is tracked on its own.
func (a deadmanAlert) groupKey() string { return deadmanGroupPrefix + a.fingerprint } func (a deadmanAlert) groupKey() string { return deadmanGroupPrefix + a.fingerprint }
@@ -189,13 +233,13 @@ func (a deadmanAlert) groupKey() string { return deadmanGroupPrefix + a.fingerpr
// It returns the ids of the alerts it owns, because the generic staleness // It returns the ids of the alerts it owns, because the generic staleness
// expiry must leave them alone — staleAfter and ends_at would otherwise resolve // expiry must leave them alone — staleAfter and ends_at would otherwise resolve
// a heartbeat long before its own, much tighter, timeout ever fired. // a heartbeat long before its own, much tighter, timeout ever fired.
// Each team is swept against its own configuration: its own matchers, its own // Each team is swept against its own switches, each with its own matcher,
// timeout, its own severity. A team watching nothing is skipped entirely, which // timeout and severity. A team watching nothing is skipped entirely, which is
// is most of them. // most of them.
func sweepDeadman(ctx context.Context, db *sql.DB, notify NotifyConfig) map[int64]bool { func sweepDeadman(ctx context.Context, db *sql.DB, notify NotifyConfig) map[int64]bool {
owned := map[int64]bool{} owned := map[int64]bool{}
configs, err := deadmanConfigs(ctx, db) configs, err := deadmanSets(ctx, db)
if err != nil { if err != nil {
log.Printf("deadman: load configs: %v", err) log.Printf("deadman: load configs: %v", err)
return owned return owned
@@ -203,39 +247,35 @@ func sweepDeadman(ctx context.Context, db *sql.DB, notify NotifyConfig) map[int6
now := time.Now() now := time.Now()
for teamID, cfg := range configs { for teamID, cfg := range configs {
switches, err := deadmanAlerts(ctx, db, teamID, cfg) heartbeats, err := deadmanAlerts(ctx, db, teamID, cfg)
if err != nil { if err != nil {
log.Printf("deadman: load switches for team %d: %v", teamID, err) log.Printf("deadman: load heartbeats for team %d: %v", teamID, err)
continue continue
} }
cutoff := now.Add(-cfg.Timeout).Unix()
for _, sw := range switches { for _, hb := range heartbeats {
owned[sw.id] = true owned[hb.id] = true
// An explicit resolved from Alertmanager is a stronger death signal if hb.dead(now) {
// than mere absence: the sender is telling us the heartbeat if err := deadmanDied(ctx, db, notify, hb, now); err != nil {
// stopped, so there is nothing left to wait out. log.Printf("deadman: open incident for %s: %v", hb.sw.Matcher.Name, err)
if sw.resolved || sw.receivedAt < cutoff {
if err := deadmanDied(ctx, db, cfg, notify, sw, now); err != nil {
log.Printf("deadman: open incident for %s: %v", sw.matcher.Name, err)
} }
continue continue
} }
if err := deadmanRecovered(ctx, db, sw); err != nil { if err := deadmanRecovered(ctx, db, hb); err != nil {
log.Printf("deadman: resolve incident for %s: %v", sw.matcher.Name, err) log.Printf("deadman: resolve incident for %s: %v", hb.sw.Matcher.Name, err)
} }
} }
} }
return owned return owned
} }
// deadmanAlerts loads every alert row that a matcher claims. The candidate query // deadmanAlerts loads every alert row that one of a team's switches claims. The candidate query
// is narrowed by alertname so it rides alerts_name_idx; the rest of the matching // is narrowed by alertname so it rides alerts_name_idx; the rest of the matching
// happens in Go, which keeps one implementation of the rules. The rows are read // happens in Go, which keeps one implementation of the rules. The rows are read
// in full before the caller writes, so the writes do not run against an open // in full before the caller writes, so the writes do not run against an open
// cursor over the same table. // cursor over the same table.
func deadmanAlerts(ctx context.Context, db *sql.DB, teamID int64, cfg DeadmanConfig) ([]deadmanAlert, error) { func deadmanAlerts(ctx context.Context, db *sql.DB, teamID int64, cfg deadmanSet) ([]deadmanAlert, error) {
names := cfg.names() names := cfg.names()
args := &sqlArgs{} args := &sqlArgs{}
nameList := make([]any, len(names)) nameList := make([]any, len(names))
@@ -263,11 +303,11 @@ func deadmanAlerts(ctx context.Context, db *sql.DB, teamID int64, cfg DeadmanCon
} }
json.Unmarshal([]byte(labelsJSON), &a.labels) //nolint:errcheck json.Unmarshal([]byte(labelsJSON), &a.labels) //nolint:errcheck
m, ok := cfg.match(a.labels) sw, ok := cfg.match(a.labels)
if !ok { if !ok {
continue continue
} }
a.matcher = m a.sw = sw
a.resolved = status == "resolved" a.resolved = status == "resolved"
out = append(out, a) out = append(out, a)
} }
@@ -284,16 +324,16 @@ func deadmanAlerts(ctx context.Context, db *sql.DB, teamID int64, cfg DeadmanCon
// incidentForGroup), and a source that is gone for good is a one-time page // incidentForGroup), and a source that is gone for good is a one-time page
// rather than a nag. Only a heartbeat that comes back and dies again earns a new // rather than a nag. Only a heartbeat that comes back and dies again earns a new
// incident. // incident.
func deadmanDied(ctx context.Context, db *sql.DB, cfg DeadmanConfig, notify NotifyConfig, sw deadmanAlert, now time.Time) error { func deadmanDied(ctx context.Context, db *sql.DB, notify NotifyConfig, hb deadmanAlert, now time.Time) error {
var lastTriggered, open int64 var lastTriggered, open int64
if err := db.QueryRowContext(ctx, ` if err := db.QueryRowContext(ctx, `
SELECT COALESCE(MAX(triggered_at), 0), SELECT COALESCE(MAX(triggered_at), 0),
COUNT(*) FILTER (WHERE resolved_at IS NULL) COUNT(*) FILTER (WHERE resolved_at IS NULL)
FROM incidents WHERE team_id = $1 AND group_key = $2`, FROM incidents WHERE team_id = $1 AND group_key = $2`,
sw.teamID, sw.groupKey()).Scan(&lastTriggered, &open); err != nil { hb.teamID, hb.groupKey()).Scan(&lastTriggered, &open); err != nil {
return err return err
} }
if open > 0 || sw.receivedAt <= lastTriggered { if open > 0 || hb.receivedAt <= lastTriggered {
return nil return nil
} }
@@ -306,18 +346,18 @@ func deadmanDied(ctx context.Context, db *sql.DB, cfg DeadmanConfig, notify Noti
// A heartbeat nobody has heard from is not firing, and saying otherwise in // A heartbeat nobody has heard from is not firing, and saying otherwise in
// the alert list would be a lie. An Alertmanager-sourced resolution keeps its // the alert list would be a lie. An Alertmanager-sourced resolution keeps its
// own source: it told us the truth first. // own source: it told us the truth first.
if !sw.resolved { if !hb.resolved {
if _, err := tx.ExecContext(ctx, ` if _, err := tx.ExecContext(ctx, `
UPDATE alerts UPDATE alerts
SET status = 'resolved', SET status = 'resolved',
resolution_source = $1, resolution_source = $1,
ends_at = COALESCE(ends_at, `+nowEpoch+`) ends_at = COALESCE(ends_at, `+nowEpoch+`)
WHERE id = $2 AND status = 'firing'`, resolutionDeadman, sw.id); err != nil { WHERE id = $2 AND status = 'firing'`, resolutionDeadman, hb.id); err != nil {
return err return err
} }
} }
severity := cfg.Severity severity := hb.sw.Severity
var sev *string var sev *string
if severity != "" { if severity != "" {
sev = &severity sev = &severity
@@ -325,14 +365,14 @@ func deadmanDied(ctx context.Context, db *sql.DB, cfg DeadmanConfig, notify Noti
// The incident opens in the team whose integration received the heartbeat: // The incident opens in the team whose integration received the heartbeat:
// the switch belongs to whoever is watching that source, not to the install. // the switch belongs to whoever is watching that source, not to the install.
incidentID, err := openIncident(ctx, tx, notify, sw.teamID, sw.groupKey(), incidentID, err := openIncident(ctx, tx, notify, hb.teamID, hb.groupKey(),
"No heartbeat from "+sw.matcher.String(), sw.labels, sev) "No heartbeat from "+hb.sw.Matcher.String(), hb.labels, sev)
if err != nil { if err != nil {
return err return err
} }
alertID := sw.id alertID := hb.id
detail := "last heartbeat " + humanDuration(now.Sub(time.Unix(sw.receivedAt, 0))) + " ago" detail := "last heartbeat " + humanDuration(now.Sub(time.Unix(hb.receivedAt, 0))) + " ago"
if err := logEvent(ctx, tx, incidentID, evDeadmanSilent, nil, &alertID, &detail); err != nil { if err := logEvent(ctx, tx, incidentID, evDeadmanSilent, nil, &alertID, &detail); err != nil {
return err return err
} }
@@ -340,7 +380,7 @@ func deadmanDied(ctx context.Context, db *sql.DB, cfg DeadmanConfig, notify Noti
if err := tx.Commit(); err != nil { if err := tx.Commit(); err != nil {
return err return err
} }
log.Printf("deadman: %s went silent, opened incident %d", sw.matcher.String(), incidentID) log.Printf("deadman: %s went silent, opened incident %d", hb.sw.Matcher.String(), incidentID)
return nil return nil
} }
@@ -350,12 +390,12 @@ func deadmanDied(ctx context.Context, db *sql.DB, cfg DeadmanConfig, notify Noti
// member alerts (linking the heartbeat would have the settled-incident cascade // member alerts (linking the heartbeat would have the settled-incident cascade
// close it on the very same sweep that opened it), so the alert-driven cascade // close it on the very same sweep that opened it), so the alert-driven cascade
// ignores it entirely and recovery is the only automatic way out. // ignores it entirely and recovery is the only automatic way out.
func deadmanRecovered(ctx context.Context, db *sql.DB, sw deadmanAlert) error { func deadmanRecovered(ctx context.Context, db *sql.DB, hb deadmanAlert) error {
var incidentID int64 var incidentID int64
switch err := db.QueryRowContext(ctx, ` switch err := db.QueryRowContext(ctx, `
SELECT id FROM incidents SELECT id FROM incidents
WHERE team_id = $1 AND group_key = $2 AND resolved_at IS NULL`, WHERE team_id = $1 AND group_key = $2 AND resolved_at IS NULL`,
sw.teamID, sw.groupKey()).Scan(&incidentID); { hb.teamID, hb.groupKey()).Scan(&incidentID); {
case err == sql.ErrNoRows: case err == sql.ErrNoRows:
return nil return nil
case err != nil: case err != nil:
@@ -387,116 +427,256 @@ func deadmanRecovered(ctx context.Context, db *sql.DB, sw deadmanAlert) error {
if err := tx.Commit(); err != nil { if err := tx.Commit(); err != nil {
return err return err
} }
log.Printf("deadman: %s is back, resolved incident %d", sw.matcher.String(), incidentID) log.Printf("deadman: %s is back, resolved incident %d", hb.sw.Matcher.String(), incidentID)
return nil return nil
} }
// --------------------------------------------------------------------------- // ---------------------------------------------------------------------------
// Per-team configuration // A team's switches
// --------------------------------------------------------------------------- // ---------------------------------------------------------------------------
// deadmanConfigForTeam reads one team's switches. A team with no row, or with const deadmanSwitchColumns = "id, team_id, name, matcher, timeout_seconds, severity"
// nothing configured, gets a disabled config — which is the right answer rather
// than an error: most teams watch no heartbeat at all.
func deadmanConfigForTeam(ctx context.Context, q querier, teamID int64) (DeadmanConfig, error) {
var matchers, severity string
var timeout int64
err := q.QueryRowContext(ctx,
"SELECT matchers, timeout_seconds, severity FROM deadman_configs WHERE team_id = $1",
teamID).Scan(&matchers, &timeout, &severity)
if err == sql.ErrNoRows {
return DeadmanConfig{}, nil
}
if err != nil {
return DeadmanConfig{}, err
}
return parseDeadmanQuietly(matchers, time.Duration(timeout)*time.Second, severity), nil
}
// deadmanConfigs reads every team's switches in one query, for the sweeper. // scanDeadmanSwitches reads switch rows into per-team sets. A row whose matcher
func deadmanConfigs(ctx context.Context, db *sql.DB) (map[int64]DeadmanConfig, error) { // no longer parses is skipped rather than fatal: the API refuses to store one,
rows, err := db.QueryContext(ctx, // so it can only mean a hand edit, and one bad row must not stop the others
"SELECT team_id, matchers, timeout_seconds, severity FROM deadman_configs") // from being watched.
if err != nil { func scanDeadmanSwitches(rows *sql.Rows) (map[int64]deadmanSet, error) {
return nil, err
}
defer rows.Close() defer rows.Close()
out := map[int64]deadmanSet{}
out := map[int64]DeadmanConfig{}
for rows.Next() { for rows.Next() {
var sw DeadmanSwitch
var teamID, timeout int64 var teamID, timeout int64
var matchers, severity string var matcher string
if err := rows.Scan(&teamID, &matchers, &timeout, &severity); err != nil { if err := rows.Scan(&sw.ID, &teamID, &sw.Name, &matcher, &timeout, &sw.Severity); err != nil {
return nil, err return nil, err
} }
cfg := parseDeadmanQuietly(matchers, time.Duration(timeout)*time.Second, severity) m, err := parseDeadmanMatcher(matcher)
if cfg.enabled() { if err != nil {
out[teamID] = cfg log.Printf("deadman: switch %d has an unusable matcher %q: %v", sw.ID, matcher, err)
continue
} }
sw.Matcher = m
sw.Timeout = time.Duration(timeout) * time.Second
out[teamID] = append(out[teamID], sw)
} }
return out, rows.Err() return out, rows.Err()
} }
// SeedDeadmanConfigs gives every team without a row the server's environment // deadmanSetForTeam reads one team's switches. A team with none gets an empty
// configuration, so the install that upgrades into per-team switches keeps // set — which is the right answer rather than an error: most teams watch no
// watching exactly what it was watching before. // heartbeat at all.
func deadmanSetForTeam(ctx context.Context, q querier, teamID int64) (deadmanSet, error) {
rows, err := q.QueryContext(ctx,
"SELECT "+deadmanSwitchColumns+" FROM deadman_switches WHERE team_id = $1 ORDER BY id", teamID)
if err != nil {
return nil, err
}
sets, err := scanDeadmanSwitches(rows)
return sets[teamID], err
}
// deadmanSets reads every team's switches in one query, for the sweeper.
func deadmanSets(ctx context.Context, db *sql.DB) (map[int64]deadmanSet, error) {
rows, err := db.QueryContext(ctx,
"SELECT "+deadmanSwitchColumns+" FROM deadman_switches ORDER BY id")
if err != nil {
return nil, err
}
return scanDeadmanSwitches(rows)
}
// deadmanSeededKey is the settings row that records the environment defaults
// were handed out. Without it, a team that deleted its last switch would get
// the default back on the next restart.
const deadmanSeededKey = "deadman_seeded"
// SeedDeadmanConfigs gives every team the server's environment defaults as
// switches, exactly once per install, so a fresh install watches Watchdog
// without anybody setting it up.
// //
// Idempotent, and never overwrites: once a team has a row it owns its own // Once seeded it never runs again: a team's switches are its own, and a redeploy
// configuration, and a redeploy must not quietly put the environment's value // must not quietly put the environment's value back over an owner's edit or
// back over an owner's edit. // deletion. Installs that upgraded from per-team configuration were already
// seeded, which migration 009 records.
// //
// A team created after startup gets no row and therefore watches nothing until // A team created after that gets none and watches nothing until its owner says
// its owner says otherwise. That is deliberate: inheriting an install-wide // otherwise. That is deliberate: inheriting an install-wide heartbeat would page
// heartbeat would page a new team about a source it has never heard of, and a // a new team about a source it has never heard of, and a switch nobody chose is
// switch nobody chose is the kind that gets muted rather than fixed. // the kind that gets muted rather than fixed.
func SeedDeadmanConfigs(ctx context.Context, db *sql.DB, cfg DeadmanConfig) error { func SeedDeadmanConfigs(ctx context.Context, db *sql.DB, cfg DeadmanConfig) error {
matchers := make([]string, 0, len(cfg.Matchers)) if !cfg.enabled() {
for _, m := range cfg.Matchers { return nil
parts := []string{"alertname=" + m.Name}
for k, v := range m.Labels {
parts = append(parts, k+"="+v)
}
sort.Strings(parts[1:])
matchers = append(matchers, strings.Join(parts, ","))
} }
_, err := db.ExecContext(ctx, ` tx, err := db.BeginTx(ctx, nil)
INSERT INTO deadman_configs (team_id, matchers, timeout_seconds, severity) if err != nil {
SELECT id, $1, $2, $3 FROM teams
ON CONFLICT (team_id) DO NOTHING`,
strings.Join(matchers, "; "), int64(cfg.Timeout.Seconds()), cfg.Severity)
return err return err
}
defer tx.Rollback() //nolint:errcheck
res, err := tx.ExecContext(ctx,
"INSERT INTO settings (key, value) VALUES ($1, '1') ON CONFLICT (key) DO NOTHING",
deadmanSeededKey)
if err != nil {
return err
}
if n, _ := res.RowsAffected(); n == 0 {
return nil
}
for _, m := range cfg.Matchers {
if _, err := tx.ExecContext(ctx, `
INSERT INTO deadman_switches (team_id, name, matcher, timeout_seconds, severity)
SELECT id, $1, $1, $2, $3 FROM teams`,
m.config(), int64(cfg.Timeout.Seconds()), cfg.Severity); err != nil {
return err
}
}
return tx.Commit()
} }
// parseDeadmanQuietly is ParseDeadmanConfig without the startup logging: a // ---------------------------------------------------------------------------
// team's configuration is read on every sweep and every webhook, and logging it // Status
// each time would bury everything else. // ---------------------------------------------------------------------------
func parseDeadmanQuietly(matchers string, timeout time.Duration, severity string) DeadmanConfig {
cfg := DeadmanConfig{Timeout: timeout, Severity: severity} const (
for _, entry := range strings.Split(matchers, ";") { switchHealthy = "healthy"
entry = strings.TrimSpace(entry) switchDead = "dead"
if entry == "" { switchDormant = "dormant"
continue )
}
m := DeadmanMatcher{Labels: map[string]string{}} // deadmanSource is one heartbeat under a switch: a fingerprint that matched.
malformed := false type deadmanSource struct {
for _, cond := range strings.Split(entry, ",") { Fingerprint string `json:"fingerprint"`
k, v, ok := strings.Cut(cond, "=") Labels map[string]string `json:"labels"`
k, v = strings.TrimSpace(k), strings.TrimSpace(v) Status string `json:"status"`
if !ok || k == "" || v == "" { LastHeartbeatAt time.Time `json:"last_heartbeat_at"`
malformed = true LastTriggeredAt *time.Time `json:"last_triggered_at"`
break IncidentID *int64 `json:"incident_id"`
} }
if k == "alertname" {
m.Name = v // deadmanSwitchStatus is a switch as the Switches page shows it.
continue type deadmanSwitchStatus struct {
} ID int64 `json:"id"`
m.Labels[k] = v Name string `json:"name"`
} Matcher string `json:"matcher"`
if malformed || m.Name == "" { TimeoutSeconds int64 `json:"timeout_seconds"`
continue Severity string `json:"severity"`
}
cfg.Matchers = append(cfg.Matchers, m) // Status is dead when any source is, dormant when none has ever been heard
} // from, healthy otherwise — a live cluster must not hide a dead one.
return cfg Status string `json:"status"`
LastHeartbeatAt *time.Time `json:"last_heartbeat_at"`
LastTriggeredAt *time.Time `json:"last_triggered_at"`
OpenIncidentID *int64 `json:"open_incident_id"`
Sources []deadmanSource `json:"sources"`
}
// deadmanStatuses reports every switch of a team with what its heartbeats are
// doing. The liveness verdict is deadmanAlert.dead, the sweeper's own.
func deadmanStatuses(ctx context.Context, db *sql.DB, teamID int64, set deadmanSet, now time.Time) ([]deadmanSwitchStatus, error) {
out := make([]deadmanSwitchStatus, 0, len(set))
if len(set) == 0 {
return out, nil
}
heartbeats, err := deadmanAlerts(ctx, db, teamID, set)
if err != nil {
return nil, err
}
// One query for every switch's incident history, keyed the way the sweeper
// keys it.
type history struct {
triggeredAt int64
openID int64
}
incidents := map[string]history{}
rows, err := db.QueryContext(ctx, `
SELECT group_key, MAX(triggered_at), COALESCE(MAX(id) FILTER (WHERE resolved_at IS NULL), 0)
FROM incidents
WHERE team_id = $1 AND group_key LIKE $2
GROUP BY group_key`, teamID, deadmanGroupPrefix+"%")
if err != nil {
return nil, err
}
defer rows.Close()
for rows.Next() {
var key string
var h history
if err := rows.Scan(&key, &h.triggeredAt, &h.openID); err != nil {
return nil, err
}
incidents[key] = h
}
if err := rows.Err(); err != nil {
return nil, err
}
bySwitch := map[int64][]deadmanAlert{}
for _, hb := range heartbeats {
bySwitch[hb.sw.ID] = append(bySwitch[hb.sw.ID], hb)
}
later := func(cur *time.Time, unix int64) *time.Time {
t := time.Unix(unix, 0).UTC()
if cur == nil || t.After(*cur) {
return &t
}
return cur
}
for _, sw := range set {
st := deadmanSwitchStatus{
ID: sw.ID, Name: sw.Name, Matcher: sw.Matcher.config(),
TimeoutSeconds: int64(sw.Timeout.Seconds()), Severity: sw.Severity,
Status: switchDormant, Sources: []deadmanSource{},
}
for _, hb := range bySwitch[sw.ID] {
src := deadmanSource{
Fingerprint: hb.fingerprint,
Labels: hb.labels,
Status: switchHealthy,
LastHeartbeatAt: time.Unix(hb.receivedAt, 0).UTC(),
}
if hb.dead(now) {
src.Status = switchDead
}
if h, ok := incidents[hb.groupKey()]; ok {
t := time.Unix(h.triggeredAt, 0).UTC()
src.LastTriggeredAt = &t
st.LastTriggeredAt = later(st.LastTriggeredAt, h.triggeredAt)
if h.openID != 0 {
id := h.openID
src.IncidentID = &id
if st.OpenIncidentID == nil || id > *st.OpenIncidentID {
st.OpenIncidentID = &id
}
}
}
st.LastHeartbeatAt = later(st.LastHeartbeatAt, hb.receivedAt)
st.Sources = append(st.Sources, src)
switch {
case src.Status == switchDead:
st.Status = switchDead
case st.Status == switchDormant:
st.Status = switchHealthy
}
}
// Dead ones first, then by fingerprint: what needs attention leads, and
// the order does not shuffle between refreshes.
sort.Slice(st.Sources, func(i, j int) bool {
a, b := st.Sources[i], st.Sources[j]
if (a.Status == switchDead) != (b.Status == switchDead) {
return a.Status == switchDead
}
return a.Fingerprint < b.Fingerprint
})
out = append(out, st)
}
return out, nil
} }
+164 -14
View File
@@ -1,6 +1,7 @@
package api_test package api_test
import ( import (
"context"
"net/http" "net/http"
"strings" "strings"
"testing" "testing"
@@ -483,13 +484,13 @@ func TestDeadman_ConfigurationIsPerTeam(t *testing.T) {
unwatched := newTeam(t, s, "unwatched") unwatched := newTeam(t, s, "unwatched")
// Only the first team calls Watchdog a heartbeat. // Only the first team calls Watchdog a heartbeat.
resp := s.req(t, http.MethodPut, "/api/teams/"+id64(watched.id)+"/deadman", map[string]any{ resp := s.req(t, http.MethodPost, "/api/teams/"+id64(watched.id)+"/deadman/switches", map[string]any{
"matchers": "alertname=Watchdog", "matcher": "alertname=Watchdog",
"timeout_seconds": 3600, "timeout_seconds": 3600,
"severity": "critical", "severity": "critical",
}) })
resp.Body.Close() resp.Body.Close()
if resp.StatusCode != http.StatusOK { if resp.StatusCode != http.StatusCreated {
t.Fatalf("configure the watched team: %d", resp.StatusCode) t.Fatalf("configure the watched team: %d", resp.StatusCode)
} }
@@ -550,9 +551,9 @@ func TestDeadman_ConfigurationIsOwnerOnly(t *testing.T) {
decode(t, s.req(t, http.MethodPost, "/api/users/"+id64(user.ID)+"/api-keys", decode(t, s.req(t, http.MethodPost, "/api/users/"+id64(user.ID)+"/api-keys",
map[string]string{"name": "test"}), &key) map[string]string{"name": "test"}), &key)
req, _ := http.NewRequest(http.MethodPut, req, _ := http.NewRequest(http.MethodPost,
s.URL+"/api/teams/"+id64(team.id)+"/deadman", s.URL+"/api/teams/"+id64(team.id)+"/deadman/switches",
strings.NewReader(`{"matchers":"alertname=Watchdog","timeout_seconds":60}`)) strings.NewReader(`{"matcher":"alertname=Watchdog","timeout_seconds":60}`))
req.Header.Set("Authorization", "Bearer "+key.Key) req.Header.Set("Authorization", "Bearer "+key.Key)
req.Header.Set("Content-Type", "application/json") req.Header.Set("Content-Type", "application/json")
resp, err := http.DefaultClient.Do(req) resp, err := http.DefaultClient.Do(req)
@@ -564,7 +565,7 @@ func TestDeadman_ConfigurationIsOwnerOnly(t *testing.T) {
t.Errorf("a member editing the switches: expected 403, got %d", resp.StatusCode) t.Errorf("a member editing the switches: expected 403, got %d", resp.StatusCode)
} }
read, _ := http.NewRequest(http.MethodGet, s.URL+"/api/teams/"+id64(team.id)+"/deadman", nil) read, _ := http.NewRequest(http.MethodGet, s.URL+"/api/teams/"+id64(team.id)+"/deadman/switches", nil)
read.Header.Set("Authorization", "Bearer "+key.Key) read.Header.Set("Authorization", "Bearer "+key.Key)
got, err := http.DefaultClient.Do(read) got, err := http.DefaultClient.Do(read)
if err != nil { if err != nil {
@@ -577,16 +578,165 @@ func TestDeadman_ConfigurationIsOwnerOnly(t *testing.T) {
} }
// A matcher with no alertname watches nothing, silently, which is the failure // A matcher with no alertname watches nothing, silently, which is the failure
// this feature exists to prevent — so it is refused at the door. // this feature exists to prevent — so it is refused at the door, along with the
func TestDeadman_UnusableMatchersAreRejected(t *testing.T) { // other things that would make a switch unable to fire.
func TestDeadman_UnusableSwitchesAreRejected(t *testing.T) {
s, _ := deadmanTS(t, deadmanCfg()) s, _ := deadmanTS(t, deadmanCfg())
resp := s.req(t, http.MethodPut, "/api/teams/"+defaultTeam+"/deadman", map[string]any{ for name, body := range map[string]map[string]any{
"matchers": "cluster=prod", "no alertname": {"matcher": "cluster=prod", "timeout_seconds": 900},
"timeout_seconds": 900, "malformed": {"matcher": "alertname=Watchdog,garbage", "timeout_seconds": 900},
}) "several": {"matcher": "alertname=A; alertname=B", "timeout_seconds": 900},
"zero timeout": {"matcher": "alertname=Watchdog", "timeout_seconds": 0},
"bad severity": {"matcher": "alertname=Watchdog", "timeout_seconds": 900, "severity": "loud"},
"empty matcher": {"matcher": "", "timeout_seconds": 900},
} {
resp := s.req(t, http.MethodPost, "/api/teams/"+defaultTeam+"/deadman/switches", body)
resp.Body.Close() resp.Body.Close()
if resp.StatusCode != http.StatusBadRequest { if resp.StatusCode != http.StatusBadRequest {
t.Errorf("expected 400 for a matcher with no alertname, got %d", resp.StatusCode) t.Errorf("%s: expected 400, got %d", name, resp.StatusCode)
}
}
}
// ---------------------------------------------------------------------------
// The switch list
// ---------------------------------------------------------------------------
// listSwitches reads the default team's switches as the Switches page does.
func listSwitches(t *testing.T, s *ts) []map[string]any {
t.Helper()
return list(t, s.req(t, http.MethodGet, "/api/teams/"+defaultTeam+"/deadman/switches", nil))
}
// A switch is healthy while its heartbeat is fresh, dead once it is silent, and
// dormant until the first one arrives.
func TestDeadman_ListReportsStatus(t *testing.T) {
s, _ := deadmanTS(t, api.ParseDeadmanConfig("alertname=Watchdog; alertname=NeverSent", time.Hour, "critical"))
got := listSwitches(t, s)
if len(got) != 2 {
t.Fatalf("expected 2 switches, got %d", len(got))
}
for _, sw := range got {
if sw["status"] != "dormant" || sw["last_heartbeat_at"] != nil || sw["last_triggered_at"] != nil {
t.Errorf("a switch nobody has heard from should be dormant and blank, got %v", sw)
}
}
heartbeat(t, s, "fp-watchdog", nil)
got = listSwitches(t, s)
if got[0]["status"] != "healthy" || got[0]["last_heartbeat_at"] == nil {
t.Errorf("a fresh heartbeat should be healthy with a timestamp, got %v", got[0])
}
if got[1]["status"] != "dormant" {
t.Errorf("the other switch is still dormant, got %v", got[1]["status"])
}
silence(t, s, "fp-watchdog", 2*time.Hour)
sweep(t, s, noArchive)
got = listSwitches(t, s)
if got[0]["status"] != "dead" {
t.Fatalf("a silent heartbeat should be dead, got %v", got[0]["status"])
}
if got[0]["last_triggered_at"] == nil || got[0]["open_incident_id"] == nil {
t.Errorf("a dead switch should show when it triggered and its open incident, got %v", got[0])
}
}
// One matcher, several clusters: the switch is as bad as its worst heartbeat and
// each heartbeat is listed on its own.
func TestDeadman_ListBreaksDownByFingerprint(t *testing.T) {
s, _ := deadmanTS(t, deadmanCfg())
heartbeat(t, s, "fp-a", map[string]string{"cluster": "a"})
heartbeat(t, s, "fp-b", map[string]string{"cluster": "b"})
silence(t, s, "fp-b", 2*time.Hour)
sw := listSwitches(t, s)[0]
if sw["status"] != "dead" {
t.Errorf("one dead cluster makes the switch dead, got %v", sw["status"])
}
sources := sw["sources"].([]any)
if len(sources) != 2 {
t.Fatalf("expected 2 sources, got %d", len(sources))
}
first, second := sources[0].(map[string]any), sources[1].(map[string]any)
if first["fingerprint"] != "fp-b" || first["status"] != "dead" || second["status"] != "healthy" {
t.Errorf("the dead source should lead, got %v then %v", first, second)
}
}
// Every switch keeps its own deadline.
func TestDeadman_TimeoutsArePerSwitch(t *testing.T) {
s, _ := deadmanTS(t, deadmanCfg())
resp := s.req(t, http.MethodPost, "/api/teams/"+defaultTeam+"/deadman/switches", map[string]any{
"matcher": "alertname=Edge", "timeout_seconds": 300,
})
resp.Body.Close()
heartbeat(t, s, "fp-watchdog", nil)
postWebhook(t, s, []map[string]any{
amAlert("fp-edge", "Edge", "firing", "2026-05-20T10:00:00Z", zeroTime, nil),
}, `{}:{alertname="Edge"}`)
// Ten minutes of silence: past the Edge switch's five, inside Watchdog's hour.
silence(t, s, "fp-watchdog", 10*time.Minute)
silence(t, s, "fp-edge", 10*time.Minute)
got := listSwitches(t, s)
if got[0]["status"] != "healthy" || got[1]["status"] != "dead" {
t.Errorf("want Watchdog healthy and Edge dead, got %v and %v", got[0]["status"], got[1]["status"])
}
}
// Deleting is an owner's, is scoped to the team, and leaves what the switch
// already opened alone.
func TestDeadman_DeleteIsScopedToTheTeam(t *testing.T) {
s, _ := deadmanTS(t, deadmanCfg())
other := newTeam(t, s, "other")
id := int64(listSwitches(t, s)[0]["id"].(float64))
// Another team's owner cannot reach it.
resp := other.call(http.MethodDelete, "/api/teams/"+id64(other.id)+"/deadman/switches/"+id64(id), nil)
resp.Body.Close()
if resp.StatusCode != http.StatusNotFound {
t.Errorf("deleting another team's switch: expected 404, got %d", resp.StatusCode)
}
if got := len(listSwitches(t, s)); got != 1 {
t.Fatalf("the switch should have survived, %d left", got)
}
resp = s.req(t, http.MethodDelete, "/api/teams/"+defaultTeam+"/deadman/switches/"+id64(id), nil)
resp.Body.Close()
if resp.StatusCode != http.StatusNoContent {
t.Fatalf("deleting: expected 204, got %d", resp.StatusCode)
}
if got := len(listSwitches(t, s)); got != 0 {
t.Errorf("expected no switches, got %d", got)
}
}
// The environment's defaults are handed out once and then belong to the teams.
func TestDeadman_SeedRunsOnce(t *testing.T) {
s := newTS(t)
cfg := api.ParseDeadmanConfig("alertname=Watchdog", time.Hour, "critical")
if err := api.SeedDeadmanConfigs(context.Background(), s.db, cfg); err != nil {
t.Fatalf("seed: %v", err)
}
if got := len(listSwitches(t, s)); got != 1 {
t.Fatalf("the first seed should add the default, got %d switches", got)
}
// The owner deletes it; a restart must not put it back.
id := int64(listSwitches(t, s)[0]["id"].(float64))
s.req(t, http.MethodDelete, "/api/teams/"+defaultTeam+"/deadman/switches/"+id64(id), nil).Body.Close()
if err := api.SeedDeadmanConfigs(context.Background(), s.db, cfg); err != nil {
t.Fatalf("seed again: %v", err)
}
if got := len(listSwitches(t, s)); got != 0 {
t.Errorf("a second seed resurrected %d switch(es)", got)
} }
} }
+3 -2
View File
@@ -144,8 +144,9 @@ func NewRouter(db *sql.DB, notify NotifyConfig, cfg config.Config) http.Handler
// A team's own dead man's switches: which of its alerts are heartbeats, // A team's own dead man's switches: which of its alerts are heartbeats,
// and how long a silence has to last before somebody is paged. // and how long a silence has to last before somebody is paged.
r.Get("/api/teams/{teamID}/deadman", handleGetTeamDeadman(db)) r.Get("/api/teams/{teamID}/deadman/switches", handleListTeamDeadman(db))
r.Put("/api/teams/{teamID}/deadman", handleSetTeamDeadman(db)) r.Post("/api/teams/{teamID}/deadman/switches", handleCreateTeamDeadman(db))
r.Delete("/api/teams/{teamID}/deadman/switches/{switchID}", handleDeleteTeamDeadman(db))
// Integrations: where a team's alerts come in, and the key that says so. // Integrations: where a team's alerts come in, and the key that says so.
r.Get("/api/teams/{teamID}/integrations", handleListIntegrations(db)) r.Get("/api/teams/{teamID}/integrations", handleListIntegrations(db))
+89 -46
View File
@@ -539,17 +539,24 @@ func defaultTeamID(ctx context.Context, db *sql.DB) (int64, error) {
// A team's dead man's switches // A team's dead man's switches
// --------------------------------------------------------------------------- // ---------------------------------------------------------------------------
// deadmanResponse is the wire shape of a team's switch configuration. The // deadmanSwitchRequest is what creating a switch takes. The timeout is seconds,
// timeout is seconds rather than a duration string, because that is what the // because that is what the column holds and what arithmetic is done on; a client
// column holds and what arithmetic is done on; a client renders it. // renders it.
type deadmanResponse struct { type deadmanSwitchRequest struct {
TeamID int64 `json:"team_id"` Name string `json:"name"`
Matchers string `json:"matchers"` Matcher string `json:"matcher"`
TimeoutSeconds int64 `json:"timeout_seconds"` TimeoutSeconds int64 `json:"timeout_seconds"`
Severity string `json:"severity"` Severity string `json:"severity"`
} }
func handleGetTeamDeadman(db *sql.DB) http.HandlerFunc { // deadmanSeverities are the severities an incident can open at.
var deadmanSeverities = map[string]bool{"critical": true, "error": true, "warning": true, "info": true}
// handleListTeamDeadman lists a team's switches with what each one's heartbeats
// are doing. A team with none gets an empty list, which is a configuration and
// not an absence: answering 404 would make "off" indistinguishable from "this
// server does not do this".
func handleListTeamDeadman(db *sql.DB) http.HandlerFunc {
return func(w http.ResponseWriter, r *http.Request) { return func(w http.ResponseWriter, r *http.Request) {
teamID, ok := teamParam(w, r) teamID, ok := teamParam(w, r)
if !ok { if !ok {
@@ -559,27 +566,26 @@ func handleGetTeamDeadman(db *sql.DB) http.HandlerFunc {
return return
} }
out := deadmanResponse{TeamID: teamID, Severity: "critical"} set, err := deadmanSetForTeam(r.Context(), db, teamID)
err := db.QueryRowContext(r.Context(), if err != nil {
"SELECT matchers, timeout_seconds, severity FROM deadman_configs WHERE team_id = $1", respond(w, http.StatusInternalServerError, errResp("internal error"))
teamID).Scan(&out.Matchers, &out.TimeoutSeconds, &out.Severity) return
if err != nil && !errors.Is(err, sql.ErrNoRows) { }
out, err := deadmanStatuses(r.Context(), db, teamID, set, time.Now())
if err != nil {
respond(w, http.StatusInternalServerError, errResp("internal error")) respond(w, http.StatusInternalServerError, errResp("internal error"))
return return
} }
// A team with no row watches nothing, which is a configuration and not
// an absence: answering 404 would make "off" indistinguishable from
// "this server does not do this".
respond(w, http.StatusOK, out) respond(w, http.StatusOK, out)
} }
} }
// handleSetTeamDeadman replaces a team's switch configuration. // handleCreateTeamDeadman adds one switch.
// //
// Validated by parsing: a matcher string that survives ParseDeadmanConfig with // Validated by parsing: a matcher with no alertname is rejected rather than
// nothing usable in it is rejected rather than stored, because a switch that // stored, because a switch that silently watches nothing is the failure this
// silently watches nothing is the failure this feature exists to prevent. // feature exists to prevent.
func handleSetTeamDeadman(db *sql.DB) http.HandlerFunc { func handleCreateTeamDeadman(db *sql.DB) http.HandlerFunc {
return func(w http.ResponseWriter, r *http.Request) { return func(w http.ResponseWriter, r *http.Request) {
teamID, ok := teamParam(w, r) teamID, ok := teamParam(w, r)
if !ok { if !ok {
@@ -589,50 +595,87 @@ func handleSetTeamDeadman(db *sql.DB) http.HandlerFunc {
return return
} }
var req struct { var req deadmanSwitchRequest
Matchers string `json:"matchers"`
TimeoutSeconds int64 `json:"timeout_seconds"`
Severity string `json:"severity"`
}
if err := decodeJSON(r, &req); err != nil { if err := decodeJSON(r, &req); err != nil {
respond(w, http.StatusBadRequest, errResp("invalid request body")) respond(w, http.StatusBadRequest, errResp("invalid request body"))
return return
} }
req.Matchers = strings.TrimSpace(req.Matchers) req.Matcher = strings.TrimSpace(req.Matcher)
req.Name = strings.TrimSpace(req.Name)
if req.Severity == "" { if req.Severity == "" {
req.Severity = "critical" req.Severity = "critical"
} }
if req.TimeoutSeconds < 0 { if !deadmanSeverities[req.Severity] {
respond(w, http.StatusBadRequest, errResp("timeout_seconds must not be negative")) respond(w, http.StatusBadRequest, errResp("severity must be critical, error, warning or info"))
return return
} }
if req.Matchers != "" { if req.TimeoutSeconds <= 0 {
parsed := parseDeadmanQuietly(req.Matchers, time.Duration(req.TimeoutSeconds)*time.Second, req.Severity) respond(w, http.StatusBadRequest, errResp("timeout_seconds must be positive"))
if len(parsed.Matchers) == 0 { return
}
if strings.Contains(req.Matcher, ";") {
respond(w, http.StatusBadRequest, errResp("one matcher per switch: add another switch instead of separating with ;"))
return
}
m, err := parseDeadmanMatcher(req.Matcher)
if err != nil {
respond(w, http.StatusBadRequest, errResp( respond(w, http.StatusBadRequest, errResp(
"no usable matchers: each must name an alertname, as in alertname=Watchdog,cluster=prod")) "unusable matcher ("+err.Error()+"): each must name an alertname, as in alertname=Watchdog,cluster=prod"))
return return
} }
if req.Name == "" {
req.Name = m.config()
}
if len(req.Name) > 100 {
respond(w, http.StatusBadRequest, errResp("name is too long"))
return
} }
if _, err := db.ExecContext(r.Context(), ` var id int64
INSERT INTO deadman_configs (team_id, matchers, timeout_seconds, severity, updated_at) if err := db.QueryRowContext(r.Context(), `
VALUES ($1, $2, $3, $4, `+nowEpoch+`) INSERT INTO deadman_switches (team_id, name, matcher, timeout_seconds, severity)
ON CONFLICT (team_id) DO UPDATE SET VALUES ($1, $2, $3, $4, $5) RETURNING id`,
matchers = excluded.matchers, teamID, req.Name, m.config(), req.TimeoutSeconds, req.Severity).Scan(&id); err != nil {
timeout_seconds = excluded.timeout_seconds,
severity = excluded.severity,
updated_at = excluded.updated_at`,
teamID, req.Matchers, req.TimeoutSeconds, req.Severity); err != nil {
respond(w, http.StatusInternalServerError, errResp("internal error")) respond(w, http.StatusInternalServerError, errResp("internal error"))
return return
} }
respond(w, http.StatusOK, deadmanResponse{ respond(w, http.StatusCreated, deadmanSwitchStatus{
TeamID: teamID, ID: id, Name: req.Name, Matcher: m.config(),
Matchers: req.Matchers, TimeoutSeconds: req.TimeoutSeconds, Severity: req.Severity,
TimeoutSeconds: req.TimeoutSeconds, Status: switchDormant, Sources: []deadmanSource{},
Severity: req.Severity,
}) })
} }
} }
// handleDeleteTeamDeadman removes a switch. An incident it already opened stays
// open until somebody resolves it: deleting the switch says "stop watching", not
// "the problem is gone".
func handleDeleteTeamDeadman(db *sql.DB) http.HandlerFunc {
return func(w http.ResponseWriter, r *http.Request) {
teamID, ok := teamParam(w, r)
if !ok {
return
}
if !requireTeamOwner(w, r, teamID) {
return
}
switchID, err := strconv.ParseInt(chi.URLParam(r, "switchID"), 10, 64)
if err != nil {
respond(w, http.StatusBadRequest, errResp("invalid switch id"))
return
}
res, err := db.ExecContext(r.Context(),
"DELETE FROM deadman_switches WHERE id = $1 AND team_id = $2", switchID, teamID)
if err != nil {
respond(w, http.StatusInternalServerError, errResp("internal error"))
return
}
if n, _ := res.RowsAffected(); n == 0 {
respond(w, http.StatusNotFound, errResp("switch not found"))
return
}
w.WriteHeader(http.StatusNoContent)
}
}
@@ -0,0 +1,54 @@
-- Dead man's switches become rows of their own.
--
-- 004 kept a team's switches in one string with one timeout and one severity,
-- which was enough to configure them and not enough to show them: there was no
-- thing to list, nothing to hang a status on, and every switch in a team had to
-- share a deadline. A row per switch gives each its own name, matcher, timeout
-- and severity, and gives the Team → Switches page something to be a list of.
--
-- The matcher keeps the syntax the string used, one matcher per row:
-- `alertname=Watchdog,cluster=prod`. The unit of monitoring is still the
-- fingerprint, so a matcher that many clusters satisfy is still one switch row
-- watching several independent heartbeats.
CREATE TABLE deadman_switches (
id BIGSERIAL PRIMARY KEY,
team_id BIGINT NOT NULL REFERENCES teams(id) ON DELETE CASCADE,
-- What the owner calls it. Defaults to the matcher when they do not say.
name TEXT NOT NULL,
-- "," separates the label conditions, "=" is exact equality, and alertname is
-- mandatory: it is what keeps the sweeper's candidate query on an index.
matcher TEXT NOT NULL,
-- Seconds of silence before the switch is declared dead. Never zero: a switch
-- that cannot fire is deleted, not disabled.
timeout_seconds BIGINT NOT NULL CHECK (timeout_seconds > 0),
-- The severity its incidents open at. See 004 for why they carry their own.
severity TEXT NOT NULL DEFAULT 'critical',
created_at BIGINT NOT NULL DEFAULT FLOOR(EXTRACT(EPOCH FROM now()))::bigint
);
CREATE INDEX deadman_switches_team_idx ON deadman_switches (team_id);
-- Carry every team's configuration over, one row per matcher. A team whose
-- timeout was zero had switches turned off, which is now "no rows".
INSERT INTO deadman_switches (team_id, name, matcher, timeout_seconds, severity)
SELECT c.team_id, btrim(m), btrim(m), c.timeout_seconds, c.severity
FROM deadman_configs c,
LATERAL regexp_split_to_table(c.matchers, ';') AS m
WHERE c.timeout_seconds > 0
AND btrim(m) <> ''
ORDER BY c.team_id;
-- The server seeds environment defaults into teams once, and remembers that it
-- did. An install that had a row per team was already seeded; without this
-- marker the first start after upgrading would seed teams that had switched
-- theirs off.
INSERT INTO settings (key, value)
SELECT 'deadman_seeded', '1'
WHERE EXISTS (SELECT 1 FROM deadman_configs);
DROP TABLE deadman_configs;
+17 -1
View File
@@ -359,7 +359,11 @@ input:focus, textarea:focus { outline: none; border-color: var(--accent); box-sh
.badge.st-triggered, .badge.st-firing { background: var(--crit-soft); color: var(--crit); } .badge.st-triggered, .badge.st-firing { background: var(--crit-soft); color: var(--crit); }
.badge.st-acknowledged { background: var(--warn-soft); color: var(--warn); } .badge.st-acknowledged { background: var(--warn-soft); color: var(--warn); }
.badge.st-snoozed { background: var(--snooze-soft); color: var(--snooze); } .badge.st-snoozed { background: var(--snooze-soft); color: var(--snooze); }
.badge.st-resolved { background: var(--ok-soft); color: var(--ok); } .badge.st-resolved, .badge.st-healthy { background: var(--ok-soft); color: var(--ok); }
.badge.st-dead { background: var(--crit-soft); color: var(--crit); }
/* Dormant is the plain badge on purpose: nothing has gone wrong and nothing has
gone right, which is what the muted default already says. */
.badge.st-dormant { background: var(--surface-2); color: var(--muted); }
.badge.sev-critical { background: var(--crit-soft); color: var(--crit); } .badge.sev-critical { background: var(--crit-soft); color: var(--crit); }
.badge.sev-warning { background: var(--warn-soft); color: var(--warn); } .badge.sev-warning { background: var(--warn-soft); color: var(--warn); }
.badge.sev-info { background: var(--info-soft); color: var(--info); } .badge.sev-info { background: var(--info-soft); color: var(--info); }
@@ -691,6 +695,18 @@ kbd {
.disabled-row td { opacity: 0.55; } .disabled-row td { opacity: 0.55; }
.btn-sm.danger { color: var(--crit); border-color: var(--crit-soft); } .btn-sm.danger { color: var(--crit); border-color: var(--crit-soft); }
/* --- dead man's switches -------------------------------------------------
Six columns do not fit a phone, so the table scrolls inside its card rather
than the page. A heartbeat under a switch with several is indented, the way
the escalation ladder indents its levels. */
.card-head { display: flex; align-items: center; justify-content: space-between; gap: 12px; flex-wrap: wrap; }
.table-scroll { overflow-x: auto; margin-top: 12px; }
.switch-table th, .switch-table td { white-space: nowrap; }
.switch-table td:nth-child(2) { white-space: normal; min-width: 12em; }
.switch-table .source-row td { border-bottom-style: dashed; }
.switch-table .source-row td:first-child { padding-left: 16px; }
.source-labels { display: flex; flex-wrap: wrap; gap: 4px; align-items: center; }
.inline-form { display: flex; gap: 8px; margin-top: 12px; } .inline-form { display: flex; gap: 8px; margin-top: 12px; }
.inline-form input { flex: 1; min-width: 0; } .inline-form input { flex: 1; min-width: 0; }
+5 -2
View File
@@ -136,8 +136,11 @@ export const createIntegration = (id, name) =>
export const deleteIntegration = (id, integrationID) => export const deleteIntegration = (id, integrationID) =>
call('DELETE', `/teams/${id}/integrations/${integrationID}`); call('DELETE', `/teams/${id}/integrations/${integrationID}`);
export const deadman = (id) => call('GET', `/teams/${id}/deadman`); export const deadmanSwitches = (id) => call('GET', `/teams/${id}/deadman/switches`);
export const setDeadman = (id, body) => call('PUT', `/teams/${id}/deadman`, { body }); export const createDeadmanSwitch = (id, body) =>
call('POST', `/teams/${id}/deadman/switches`, { body });
export const deleteDeadmanSwitch = (id, switchID) =>
call('DELETE', `/teams/${id}/deadman/switches/${switchID}`);
export const escalation = (id) => call('GET', `/teams/${id}/escalation`); export const escalation = (id) => call('GET', `/teams/${id}/escalation`);
export const setEscalation = (id, body) => call('PUT', `/teams/${id}/escalation`, { body }); export const setEscalation = (id, body) => call('PUT', `/teams/${id}/escalation`, { body });
+140 -39
View File
@@ -18,9 +18,9 @@
// than no form, but it is not the thing enforcing anything. // than no form, but it is not the thing enforcing anything.
import * as api from './api.js'; import * as api from './api.js';
import { h, clear, spinner, confirm, icon, openSheet, closeSheet, menuCard } from './ui.js'; import { h, clear, spinner, confirm, icon, openSheet, closeSheet, menuCard, badge, labelChip } from './ui.js';
import { state, currentTeam, users as allUsers, myID } from './state.js'; import { state, currentTeam, users as allUsers, myID } from './state.js';
import { isoDate, addDays, mondayOf, initial } from './format.js'; import { isoDate, addDays, mondayOf, initial, ago, when, duration } from './format.js';
const view = () => document.getElementById('view-team'); const view = () => document.getElementById('view-team');
@@ -111,14 +111,14 @@ async function load(id) {
return { members, escalation }; return { members, escalation };
} }
if (tab === 'sources') return { integrations: await api.integrations(id) }; if (tab === 'sources') return { integrations: await api.integrations(id) };
if (tab === 'deadman') return { deadman: await api.deadman(id) }; if (tab === 'deadman') return { deadman: await api.deadmanSwitches(id) };
const grid = gridDays(); const grid = gridDays();
const [members, integrations, escalation, deadman, schedule] = await Promise.all([ const [members, integrations, escalation, deadman, schedule] = await Promise.all([
api.teamMembers(id), api.teamMembers(id),
api.integrations(id), api.integrations(id),
api.escalation(id), api.escalation(id),
api.deadman(id), api.deadmanSwitches(id),
api.schedule(id, isoDate(grid.start), isoDate(addDays(grid.start, grid.count - 1))), api.schedule(id, isoDate(grid.start), isoDate(addDays(grid.start, grid.count - 1))),
]); ]);
return { members, integrations, escalation, deadman, schedule }; return { members, integrations, escalation, deadman, schedule };
@@ -203,8 +203,8 @@ function overview() {
const levels = (data.escalation?.levels || []).length; const levels = (data.escalation?.levels || []).length;
const keys = (data.integrations || []).length; const keys = (data.integrations || []).length;
const unused = (data.integrations || []).filter((i) => !i.last_used_at).length; const unused = (data.integrations || []).filter((i) => !i.last_used_at).length;
const switches = (data.deadman?.matchers || '') const switches = (data.deadman || []).length;
.split(';').map((x) => x.trim()).filter(Boolean).length; const dead = (data.deadman || []).filter((s) => s.status === 'dead').length;
return h('div', { class: 'overview-menu' }, return h('div', { class: 'overview-menu' },
menuCard('/team/rota', 'Rota', null, menuCard('/team/rota', 'Rota', null,
@@ -220,7 +220,9 @@ function overview() {
? (unused ? `${unused} of them never used.` : 'All in use.') ? (unused ? `${unused} of them never used.` : 'All in use.')
: 'No key yet, so nothing can reach this team.'), : 'No key yet, so nothing can reach this team.'),
menuCard('/team/deadman', 'Dead man’s switches', switches || null, menuCard('/team/deadman', 'Dead man’s switches', switches || null,
switches ? 'Alerts whose absence opens an incident.' : 'Nothing watched.'), switches
? (dead ? `${dead} of them silent.` : 'All quiet, as they should be.')
: 'Nothing watched.'),
); );
} }
@@ -650,47 +652,146 @@ function newIntegrationForm() {
// --- dead man's switches --------------------------------------------------- // --- dead man's switches ---------------------------------------------------
const SWITCH_STATUS = {
healthy: { label: 'Healthy', hint: 'Heard from within its timeout.' },
dead: { label: 'Dead', hint: 'Silent for longer than its timeout.' },
dormant: { label: 'Dormant', hint: 'Nothing has matched yet, so there is nothing to lose.' },
};
function switchBadge(status) {
const s = SWITCH_STATUS[status] || SWITCH_STATUS.dormant;
const el = badge(s.label, `st-${status}`);
el.title = s.hint;
return el;
}
const timeCell = (iso) => iso
? h('span', { title: when(iso), text: ago(iso) })
: h('span', { class: 'muted', text: 'never' });
// When it last opened an incident. An incident that is still open is a link,
// because that is the thing somebody looking at a red row wants next.
const triggeredCell = (iso, incidentID) => {
if (!iso) return h('span', { class: 'muted', text: 'never' });
return incidentID
? h('a', { href: `/incidents/${incidentID}`, title: when(iso) }, `#${incidentID} · ${ago(iso)}`)
: h('span', { title: when(iso), text: ago(iso) });
};
function switchRows(sw) {
const main = h('tr', {},
h('td', {}, switchBadge(sw.status)),
h('td', {},
h('strong', { text: sw.name }),
sw.name !== sw.matcher && h('div', { class: 'muted small' }, h('code', { text: sw.matcher }))),
h('td', { class: 'muted small' }, timeCell(sw.last_heartbeat_at)),
h('td', { class: 'muted small' }, triggeredCell(sw.last_triggered_at, sw.open_incident_id)),
h('td', { class: 'muted small', text: duration(sw.timeout_seconds * 1000) }),
h('td', {}, isOwner() && h('button', {
class: 'btn-sm danger', type: 'button', text: 'Remove',
onclick: async () => {
if (!(await confirm({
title: `Remove ${sw.name}?`,
text: 'It stops being watched. An incident it already opened stays open until it is resolved.',
confirmLabel: 'Remove',
danger: true,
}))) return;
act(() => api.deleteDeadmanSwitch(teamID, sw.id));
},
})),
);
// One heartbeat is the switch's own times; several are worth telling apart,
// since a live cluster must not hide a dead one.
const sources = sw.sources.length > 1
? sw.sources.map((src) => h('tr', { class: 'source-row' },
h('td', {}, switchBadge(src.status)),
h('td', { class: 'source-labels' },
...Object.entries(src.labels || {})
.filter(([k]) => k !== 'alertname')
.map(([k, v]) => labelChip(k, v)),
!Object.keys(src.labels || {}).some((k) => k !== 'alertname')
&& h('code', { class: 'small', text: src.fingerprint })),
h('td', { class: 'muted small' }, timeCell(src.last_heartbeat_at)),
h('td', { class: 'muted small' }, triggeredCell(src.last_triggered_at, src.incident_id)),
h('td'), h('td')))
: [];
return [main, ...sources];
}
function deadmanCard() { function deadmanCard() {
const d = data.deadman || {}; const switches = data.deadman || [];
const matchers = h('input', {
type: 'text', value: d.matchers || '', placeholder: 'alertname=Watchdog',
class: 'wide',
});
const timeout = h('input', {
type: 'number', min: '0', class: 'setting-value',
value: String(Math.round((d.timeout_seconds || 0) / 60)),
});
const severity = h('select', {},
...['critical', 'error', 'warning', 'info'].map((s) =>
h('option', { value: s, text: s, selected: (d.severity || 'critical') === s })));
const form = h('form', { class: 'stacked-form' },
h('label', {}, 'Heartbeat alerts ', matchers),
h('label', {}, 'Declare dead after ', timeout, ' minutes of silence'),
h('label', {}, 'Open the incident at severity ', severity),
h('button', { class: 'btn', type: 'submit', text: 'Save switches' }));
form.addEventListener('submit', (e) => {
e.preventDefault();
act(() => api.setDeadman(teamID, {
matchers: matchers.value.trim(),
timeout_seconds: Number(timeout.value) * 60,
severity: severity.value,
}));
});
return h('div', { class: 'card' }, return h('div', { class: 'card' },
h('div', { class: 'card-head' },
h('h2', { text: 'Dead man’s switches' }), h('h2', { text: 'Dead man’s switches' }),
isOwner() && h('button', {
class: 'btn', type: 'button', text: 'New switch', onclick: openNewSwitch,
})),
h('p', { class: 'muted small' }, h('p', { class: 'muted small' },
'Alerts whose ABSENCE is the signal. Receiving one opens nothing; going ', 'Alerts whose ABSENCE is the signal. Receiving one opens nothing; going ',
'quiet for longer than the timeout opens an incident. ', 'quiet for longer than the switch’s timeout opens an incident.'),
h('code', { text: 'alertname=Watchdog,cluster=prod; alertname=EdgeHeartbeat' }), switches.length
' — semicolons separate switches, commas separate conditions, and every ', ? h('div', { class: 'table-scroll' },
'switch must name an alertname. Leave empty to watch nothing.'), h('table', { class: 'admin-table switch-table' },
isOwner() ? form : h('p', { class: 'muted', text: d.matchers || 'Nothing watched.' }), h('thead', {}, h('tr', {},
h('th', { text: 'Status' }), h('th', { text: 'Switch' }),
h('th', { text: 'Last heartbeat' }), h('th', { text: 'Last triggered' }),
h('th', { text: 'Silent after' }), h('th'))),
h('tbody', {}, switches.flatMap(switchRows))))
: h('p', { class: 'muted', text: 'Nothing watched.' }),
); );
} }
// The form lives in the sheet, not on the page: most visits are to look at the
// list, and a form that is always open is the page this replaced.
function openNewSwitch() {
const name = h('input', { type: 'text', placeholder: 'Prod Watchdog', autofocus: true });
const matcher = h('input', {
type: 'text', placeholder: 'alertname=Watchdog,cluster=prod', class: 'wide', required: true,
});
const timeout = h('input', {
type: 'number', min: '1', value: '15', class: 'setting-value', required: true,
});
const severity = h('select', {},
...['critical', 'error', 'warning', 'info'].map((s) => h('option', { value: s, text: s })));
const problem = h('p', { class: 'load-error', hidden: true });
const form = h('form', { class: 'stacked-form' },
h('label', {}, 'Name (optional) ', name),
h('label', {}, 'Heartbeat alert ', matcher),
h('p', { class: 'muted small' },
'Conditions are ', h('code', { text: 'label=value' }), ' separated by commas, and one ',
'must be ', h('code', { text: 'alertname' }), '. Every distinct label set that ',
'matches is watched on its own.'),
h('label', {}, 'Declare dead after ', timeout, ' minutes of silence'),
h('label', {}, 'Open the incident at severity ', severity),
problem,
h('div', { class: 'sheet-actions' },
h('button', { class: 'btn', type: 'button', text: 'Cancel', onclick: () => closeSheet(false) }),
h('button', { class: 'btn btn-primary', type: 'submit', text: 'Add switch' })));
form.addEventListener('submit', async (e) => {
e.preventDefault();
try {
await api.createDeadmanSwitch(teamID, {
name: name.value.trim(),
matcher: matcher.value.trim(),
timeout_seconds: Math.round(Number(timeout.value) * 60),
severity: severity.value,
});
} catch (err) {
problem.textContent = err.message;
problem.hidden = false;
return;
}
closeSheet(true);
refresh();
});
openSheet(() => [h('h2', { class: 'sheet-title', text: 'New switch' }), form]);
}
// --- members --------------------------------------------------------------- // --- members ---------------------------------------------------------------
function membersCard() { function membersCard() {