Compare commits
2 Commits
| Author | SHA1 | Date | |
|---|---|---|---|
| e8d45f9d3d | |||
| f3918b863c |
@@ -513,19 +513,27 @@ nothing unless something downstream notices it stop. That is what
|
||||
`TERDUT_DEADMAN_MATCHERS` defaults to.
|
||||
|
||||
**Switches belong to a team**, which decides which of its own alerts are
|
||||
heartbeats and how long a silence has to last. An owner sets them through
|
||||
`PUT /api/teams/{teamID}/deadman`; a missed heartbeat opens an incident in the
|
||||
team whose integration received it.
|
||||
heartbeats and how long a silence has to last. Each **switch** is a row of its
|
||||
own — a name, one matcher, a timeout and a severity — so switches in one team
|
||||
can have different deadlines. An owner adds and removes them on **Team →
|
||||
Switches**, which lists each with a status (**healthy**, **dead**, or
|
||||
**dormant** until its first heartbeat), when it was last heard from, and when it
|
||||
last opened an incident; a matcher that several clusters satisfy is broken down
|
||||
per cluster. The API is `POST`/`DELETE /api/teams/{teamID}/deadman/switches`. A
|
||||
missed heartbeat opens an incident in the team whose integration received it.
|
||||
Removing a switch stops the watching; an incident it already opened stays open
|
||||
until somebody resolves it.
|
||||
|
||||
The environment variables are the starting point, not the setting: at startup
|
||||
every team **without** a configuration of its own is given one from them, and an
|
||||
owner's later edit is never overwritten by a redeploy. A team created after
|
||||
that starts watching nothing until its owner says otherwise — inheriting an
|
||||
install-wide heartbeat would page a new team about a source it has never heard
|
||||
of.
|
||||
The environment variables are the starting point, not the setting: the **first**
|
||||
time the server starts, every team is given a switch per default matcher from
|
||||
them, once. After that a team's switches are its own — an owner's edit or
|
||||
deletion is never put back by a redeploy. A team created later starts watching
|
||||
nothing until its owner says otherwise — inheriting an install-wide heartbeat
|
||||
would page a new team about a source it has never heard of.
|
||||
|
||||
A matcher is a set of exact label conditions, one of which must be the
|
||||
`alertname`, in the same format the environment variable uses:
|
||||
`alertname`, in the format the environment variable uses (one matcher per switch; the
|
||||
variable takes several, separated by `;`):
|
||||
|
||||
```
|
||||
alertname=Watchdog,cluster=prod; alertname=EdgeHeartbeat
|
||||
@@ -718,8 +726,9 @@ administrator who is not in the team gets the same `404` as anybody else.
|
||||
| `DELETE` | `/api/teams/{teamID}/invites/{inviteID}` | **owner** | Revoke a link before it expires |
|
||||
| `GET` | `/api/teams/{teamID}/escalation` | member | The team's [escalation ladder](#escalation) `{repeat_count, fallback_topic, levels[]}`. Empty levels means the team has none |
|
||||
| `PUT` | `/api/teams/{teamID}/escalation` | **owner** | Replace it wholesale. `400` for a level with no targets or no timeout — a rung that pages nobody is a silence with a number on it |
|
||||
| `GET` | `/api/teams/{teamID}/deadman` | member | The team's [dead man's switch](#dead-mans-switch) configuration `{matchers, timeout_seconds, severity}` |
|
||||
| `PUT` | `/api/teams/{teamID}/deadman` | **owner** | Replace it. `400` when no matcher names an `alertname`, because a switch that silently watches nothing is the failure this feature exists to prevent |
|
||||
| `GET` | `/api/teams/{teamID}/deadman/switches` | member | The team's [dead man's switches](#dead-mans-switch), each `{id, name, matcher, timeout_seconds, severity, status, last_heartbeat_at, last_triggered_at, open_incident_id, sources[]}`. `status` is `healthy`, `dead` or `dormant`; `sources` has one entry per heartbeat fingerprint. Empty when the team watches nothing |
|
||||
| `POST` | `/api/teams/{teamID}/deadman/switches` | **owner** | Add one: `{name?, matcher, timeout_seconds, severity?}`. `400` when the matcher names no `alertname` or holds several, or the timeout is not positive — a switch that silently watches nothing is the failure this feature exists to prevent |
|
||||
| `DELETE` | `/api/teams/{teamID}/deadman/switches/{switchID}` | **owner** | Stop watching. An incident it opened stays open. `404` for a switch of another team |
|
||||
|
||||
### Notifications
|
||||
|
||||
@@ -964,8 +973,8 @@ What changes, and will need attention:
|
||||
|
||||
**Dead man's switches moved too.** `TERDUT_DEADMAN_MATCHERS`, `_TIMEOUT` and
|
||||
`_SEVERITY` are no longer the setting; they are the default each existing team
|
||||
is seeded with at startup, after which an owner edits them per team through
|
||||
`PUT /api/teams/{teamID}/deadman` and a redeploy never overwrites that.
|
||||
is seeded with at startup, after which an owner manages them per team through
|
||||
`/api/teams/{teamID}/deadman/switches` and a redeploy never overwrites that.
|
||||
|
||||
Nothing else about an incident changes, and incidents never move between teams:
|
||||
an alert belongs to whichever team's key it arrived on.
|
||||
|
||||
@@ -15,5 +15,5 @@ type: application
|
||||
# appVersion and image.tag in values.yaml no longer agree, and that is not an oversight:
|
||||
# image.tag stays "latest", which is what a local install actually pulls. appVersion is
|
||||
# metadata and drives nothing.
|
||||
version: 0.23.0
|
||||
appVersion: "v0.23.0"
|
||||
version: 0.24.0
|
||||
appVersion: "v0.24.0"
|
||||
|
||||
@@ -124,7 +124,7 @@ func ingest(ctx context.Context, db *sql.DB, notify NotifyConfig, teamID int64,
|
||||
// Which arriving alerts are heartbeats is the team's own answer, read
|
||||
// inside the transaction so an owner editing it mid-payload cannot split
|
||||
// one webhook across two interpretations.
|
||||
deadman, err := deadmanConfigForTeam(ctx, tx, teamID)
|
||||
deadman, err := deadmanSetForTeam(ctx, tx, teamID)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
@@ -186,7 +186,7 @@ func ingest(ctx context.Context, db *sql.DB, notify NotifyConfig, teamID int64,
|
||||
|
||||
// upsertAlerts stores each alert of a payload and reports what changed. Payloads
|
||||
// the ordering guard rejected are left out entirely.
|
||||
func upsertAlerts(ctx context.Context, tx *sql.Tx, deadman DeadmanConfig, teamID int64, alerts []amAlert) ([]ingested, error) {
|
||||
func upsertAlerts(ctx context.Context, tx *sql.Tx, deadman deadmanSet, teamID int64, alerts []amAlert) ([]ingested, error) {
|
||||
now := time.Now().Unix()
|
||||
accepted := make([]ingested, 0, len(alerts))
|
||||
|
||||
|
||||
+11
-13
@@ -88,27 +88,25 @@ func newDeadmanTS(t *testing.T, deadman api.DeadmanConfig, notify ...api.NotifyC
|
||||
return s
|
||||
}
|
||||
|
||||
// setTeamDeadman configures the default team's switches over the API, rendering
|
||||
// the matchers back into the string form the endpoint takes.
|
||||
// setTeamDeadman gives the default team one switch per configured matcher, over
|
||||
// the API, the way an owner would add them.
|
||||
func setTeamDeadman(t *testing.T, s *ts, cfg api.DeadmanConfig) {
|
||||
t.Helper()
|
||||
matchers := make([]string, 0, len(cfg.Matchers))
|
||||
for _, m := range cfg.Matchers {
|
||||
parts := []string{"alertname=" + m.Name}
|
||||
for k, v := range m.Labels {
|
||||
parts = append(parts, k+"="+v)
|
||||
}
|
||||
sort.Strings(parts[1:])
|
||||
matchers = append(matchers, strings.Join(parts, ","))
|
||||
}
|
||||
resp := s.req(t, http.MethodPut, "/api/teams/"+defaultTeam+"/deadman", map[string]any{
|
||||
"matchers": strings.Join(matchers, "; "),
|
||||
"timeout_seconds": int64(cfg.Timeout.Seconds()),
|
||||
"severity": cfg.Severity,
|
||||
})
|
||||
defer resp.Body.Close()
|
||||
if resp.StatusCode != http.StatusOK {
|
||||
t.Fatalf("configure the team's dead man's switches: %d", resp.StatusCode)
|
||||
resp := s.req(t, http.MethodPost, "/api/teams/"+defaultTeam+"/deadman/switches", map[string]any{
|
||||
"matcher": strings.Join(parts, ","),
|
||||
"timeout_seconds": int64(cfg.Timeout.Seconds()),
|
||||
"severity": cfg.Severity,
|
||||
})
|
||||
resp.Body.Close()
|
||||
if resp.StatusCode != http.StatusCreated {
|
||||
t.Fatalf("add a dead man's switch: %d", resp.StatusCode)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
+354
-174
@@ -4,6 +4,8 @@ import (
|
||||
"context"
|
||||
"database/sql"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"fmt"
|
||||
"log"
|
||||
"sort"
|
||||
"strings"
|
||||
@@ -39,6 +41,17 @@ func (m DeadmanMatcher) String() string {
|
||||
return m.Name + " (" + strings.Join(parts, ", ") + ")"
|
||||
}
|
||||
|
||||
// config renders the matcher in the form parseDeadmanMatcher reads, which is
|
||||
// what a switch row stores: `alertname=Watchdog,cluster=prod`.
|
||||
func (m DeadmanMatcher) config() string {
|
||||
parts := make([]string, 0, len(m.Labels))
|
||||
for k, v := range m.Labels {
|
||||
parts = append(parts, k+"="+v)
|
||||
}
|
||||
sort.Strings(parts)
|
||||
return strings.Join(append([]string{"alertname=" + m.Name}, parts...), ",")
|
||||
}
|
||||
|
||||
// matches reports whether an alert's labels satisfy every condition.
|
||||
func (m DeadmanMatcher) matches(labels map[string]string) bool {
|
||||
if labels["alertname"] != m.Name {
|
||||
@@ -52,12 +65,10 @@ func (m DeadmanMatcher) matches(labels map[string]string) bool {
|
||||
return true
|
||||
}
|
||||
|
||||
// DeadmanConfig inverts the handling of the alerts it matches: receiving one
|
||||
// opens nothing, and the absence of one opens an incident.
|
||||
//
|
||||
// The unit of monitoring is the fingerprint, not the matcher — two clusters
|
||||
// sending the same heartbeat alertname are two independent switches, so one
|
||||
// healthy cluster cannot mask a dead one.
|
||||
// DeadmanConfig is the server-wide default a team's switches are seeded from:
|
||||
// the environment's matchers, timeout and severity. Switches themselves are rows
|
||||
// of a team's own — see DeadmanSwitch — and this is only how a fresh install
|
||||
// starts out.
|
||||
type DeadmanConfig struct {
|
||||
Matchers []DeadmanMatcher
|
||||
|
||||
@@ -76,41 +87,81 @@ type DeadmanConfig struct {
|
||||
// enabled reports whether there is anything to watch.
|
||||
func (c DeadmanConfig) enabled() bool { return c.Timeout > 0 && len(c.Matchers) > 0 }
|
||||
|
||||
// match returns the first matcher an alert satisfies.
|
||||
func (c DeadmanConfig) match(labels map[string]string) (DeadmanMatcher, bool) {
|
||||
if !c.enabled() {
|
||||
return DeadmanMatcher{}, false
|
||||
}
|
||||
for _, m := range c.Matchers {
|
||||
if m.matches(labels) {
|
||||
return m, true
|
||||
}
|
||||
}
|
||||
return DeadmanMatcher{}, false
|
||||
// DeadmanSwitch inverts the handling of the alerts it matches: receiving one
|
||||
// opens nothing, and the absence of one opens an incident.
|
||||
//
|
||||
// The unit of monitoring is the fingerprint, not the switch — two clusters
|
||||
// sending the same heartbeat alertname are two independent heartbeats under one
|
||||
// switch, so one healthy cluster cannot mask a dead one.
|
||||
type DeadmanSwitch struct {
|
||||
ID int64
|
||||
Name string
|
||||
Matcher DeadmanMatcher
|
||||
|
||||
// Timeout is how long a heartbeat may go unheard before it is declared dead.
|
||||
Timeout time.Duration
|
||||
|
||||
// Severity is what the incident opens at.
|
||||
Severity string
|
||||
}
|
||||
|
||||
// isDeadman is match without the matcher, for the ingest path.
|
||||
func (c DeadmanConfig) isDeadman(labels map[string]string) bool {
|
||||
_, ok := c.match(labels)
|
||||
// deadmanSet is one team's switches.
|
||||
type deadmanSet []DeadmanSwitch
|
||||
|
||||
// match returns the first switch an alert satisfies.
|
||||
func (d deadmanSet) match(labels map[string]string) (DeadmanSwitch, bool) {
|
||||
for _, sw := range d {
|
||||
if sw.Matcher.matches(labels) {
|
||||
return sw, true
|
||||
}
|
||||
}
|
||||
return DeadmanSwitch{}, false
|
||||
}
|
||||
|
||||
// isDeadman is match without the switch, for the ingest path.
|
||||
func (d deadmanSet) isDeadman(labels map[string]string) bool {
|
||||
_, ok := d.match(labels)
|
||||
return ok
|
||||
}
|
||||
|
||||
// names lists the distinct alertnames worth loading from the database.
|
||||
func (c DeadmanConfig) names() []string {
|
||||
func (d deadmanSet) names() []string {
|
||||
seen := map[string]bool{}
|
||||
out := make([]string, 0, len(c.Matchers))
|
||||
for _, m := range c.Matchers {
|
||||
if !seen[m.Name] {
|
||||
seen[m.Name] = true
|
||||
out = append(out, m.Name)
|
||||
out := make([]string, 0, len(d))
|
||||
for _, sw := range d {
|
||||
if !seen[sw.Matcher.Name] {
|
||||
seen[sw.Matcher.Name] = true
|
||||
out = append(out, sw.Matcher.Name)
|
||||
}
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// parseDeadmanMatcher reads one matcher from its configured form: "," separates
|
||||
// the conditions and "=" is exact label equality — `alertname=Watchdog,cluster=prod`.
|
||||
// The error says what is wrong with it, in words a form can show.
|
||||
func parseDeadmanMatcher(entry string) (DeadmanMatcher, error) {
|
||||
m := DeadmanMatcher{Labels: map[string]string{}}
|
||||
for _, cond := range strings.Split(strings.TrimSpace(entry), ",") {
|
||||
k, v, ok := strings.Cut(cond, "=")
|
||||
k, v = strings.TrimSpace(k), strings.TrimSpace(v)
|
||||
if !ok || k == "" || v == "" {
|
||||
return DeadmanMatcher{}, fmt.Errorf("%q is not label=value", strings.TrimSpace(cond))
|
||||
}
|
||||
if k == "alertname" {
|
||||
m.Name = v
|
||||
continue
|
||||
}
|
||||
m.Labels[k] = v
|
||||
}
|
||||
if m.Name == "" {
|
||||
return DeadmanMatcher{}, errors.New("no alertname condition")
|
||||
}
|
||||
return m, nil
|
||||
}
|
||||
|
||||
// ParseDeadmanConfig reads the matcher list from its configured form:
|
||||
// ";" separates matchers, "," separates the conditions within one, and "=" is
|
||||
// exact label equality — `alertname=Watchdog,cluster=prod; alertname=Heartbeat`.
|
||||
// ";" separates matchers, and each is parsed as parseDeadmanMatcher does.
|
||||
//
|
||||
// A malformed or alertname-less entry is dropped rather than fatal, following
|
||||
// config.duration's rule that one bad tuning knob should not take the server
|
||||
@@ -125,28 +176,9 @@ func ParseDeadmanConfig(matchers string, timeout time.Duration, severity string)
|
||||
if entry == "" {
|
||||
continue
|
||||
}
|
||||
|
||||
m := DeadmanMatcher{Labels: map[string]string{}}
|
||||
malformed := false
|
||||
for _, cond := range strings.Split(entry, ",") {
|
||||
k, v, ok := strings.Cut(cond, "=")
|
||||
k, v = strings.TrimSpace(k), strings.TrimSpace(v)
|
||||
if !ok || k == "" || v == "" {
|
||||
log.Printf("deadman: ignoring matcher %q: %q is not label=value", entry, strings.TrimSpace(cond))
|
||||
malformed = true
|
||||
break
|
||||
}
|
||||
if k == "alertname" {
|
||||
m.Name = v
|
||||
continue
|
||||
}
|
||||
m.Labels[k] = v
|
||||
}
|
||||
if malformed {
|
||||
continue
|
||||
}
|
||||
if m.Name == "" {
|
||||
log.Printf("deadman: ignoring matcher %q: no alertname condition", entry)
|
||||
m, err := parseDeadmanMatcher(entry)
|
||||
if err != nil {
|
||||
log.Printf("deadman: ignoring matcher %q: %v", entry, err)
|
||||
continue
|
||||
}
|
||||
cfg.Matchers = append(cfg.Matchers, m)
|
||||
@@ -162,23 +194,35 @@ func ParseDeadmanConfig(matchers string, timeout time.Duration, severity string)
|
||||
for _, m := range cfg.Matchers {
|
||||
rendered = append(rendered, m.String())
|
||||
}
|
||||
log.Printf("deadman: watching %s, timeout %s, severity %s",
|
||||
log.Printf("deadman: default for new teams: %s, timeout %s, severity %s",
|
||||
strings.Join(rendered, "; "), timeout, severity)
|
||||
}
|
||||
return cfg
|
||||
}
|
||||
|
||||
// deadmanAlert is one switch: the alert row carrying its last heartbeat.
|
||||
// deadmanAlert is one heartbeat: the alert row carrying its last sighting, and
|
||||
// the switch that claimed it.
|
||||
type deadmanAlert struct {
|
||||
id int64
|
||||
teamID int64
|
||||
fingerprint string
|
||||
labels map[string]string
|
||||
matcher DeadmanMatcher
|
||||
sw DeadmanSwitch
|
||||
resolved bool
|
||||
receivedAt int64
|
||||
}
|
||||
|
||||
// dead is the one rule for a silent heartbeat, shared by the sweeper that pages
|
||||
// on it and the status the Switches page shows, so the page cannot disagree
|
||||
// with the pager.
|
||||
//
|
||||
// An explicit resolved from Alertmanager is a stronger death signal than mere
|
||||
// absence: the sender is telling us the heartbeat stopped, so there is nothing
|
||||
// left to wait out.
|
||||
func (a deadmanAlert) dead(now time.Time) bool {
|
||||
return a.resolved || a.receivedAt < now.Add(-a.sw.Timeout).Unix()
|
||||
}
|
||||
|
||||
// groupKey is the switch's identity as an incident. Per fingerprint, so each
|
||||
// source is tracked on its own.
|
||||
func (a deadmanAlert) groupKey() string { return deadmanGroupPrefix + a.fingerprint }
|
||||
@@ -189,13 +233,13 @@ func (a deadmanAlert) groupKey() string { return deadmanGroupPrefix + a.fingerpr
|
||||
// It returns the ids of the alerts it owns, because the generic staleness
|
||||
// expiry must leave them alone — staleAfter and ends_at would otherwise resolve
|
||||
// a heartbeat long before its own, much tighter, timeout ever fired.
|
||||
// Each team is swept against its own configuration: its own matchers, its own
|
||||
// timeout, its own severity. A team watching nothing is skipped entirely, which
|
||||
// is most of them.
|
||||
// Each team is swept against its own switches, each with its own matcher,
|
||||
// timeout and severity. A team watching nothing is skipped entirely, which is
|
||||
// most of them.
|
||||
func sweepDeadman(ctx context.Context, db *sql.DB, notify NotifyConfig) map[int64]bool {
|
||||
owned := map[int64]bool{}
|
||||
|
||||
configs, err := deadmanConfigs(ctx, db)
|
||||
configs, err := deadmanSets(ctx, db)
|
||||
if err != nil {
|
||||
log.Printf("deadman: load configs: %v", err)
|
||||
return owned
|
||||
@@ -203,39 +247,35 @@ func sweepDeadman(ctx context.Context, db *sql.DB, notify NotifyConfig) map[int6
|
||||
|
||||
now := time.Now()
|
||||
for teamID, cfg := range configs {
|
||||
switches, err := deadmanAlerts(ctx, db, teamID, cfg)
|
||||
heartbeats, err := deadmanAlerts(ctx, db, teamID, cfg)
|
||||
if err != nil {
|
||||
log.Printf("deadman: load switches for team %d: %v", teamID, err)
|
||||
log.Printf("deadman: load heartbeats for team %d: %v", teamID, err)
|
||||
continue
|
||||
}
|
||||
cutoff := now.Add(-cfg.Timeout).Unix()
|
||||
|
||||
for _, sw := range switches {
|
||||
owned[sw.id] = true
|
||||
for _, hb := range heartbeats {
|
||||
owned[hb.id] = true
|
||||
|
||||
// An explicit resolved from Alertmanager is a stronger death signal
|
||||
// than mere absence: the sender is telling us the heartbeat
|
||||
// stopped, so there is nothing left to wait out.
|
||||
if sw.resolved || sw.receivedAt < cutoff {
|
||||
if err := deadmanDied(ctx, db, cfg, notify, sw, now); err != nil {
|
||||
log.Printf("deadman: open incident for %s: %v", sw.matcher.Name, err)
|
||||
if hb.dead(now) {
|
||||
if err := deadmanDied(ctx, db, notify, hb, now); err != nil {
|
||||
log.Printf("deadman: open incident for %s: %v", hb.sw.Matcher.Name, err)
|
||||
}
|
||||
continue
|
||||
}
|
||||
if err := deadmanRecovered(ctx, db, sw); err != nil {
|
||||
log.Printf("deadman: resolve incident for %s: %v", sw.matcher.Name, err)
|
||||
if err := deadmanRecovered(ctx, db, hb); err != nil {
|
||||
log.Printf("deadman: resolve incident for %s: %v", hb.sw.Matcher.Name, err)
|
||||
}
|
||||
}
|
||||
}
|
||||
return owned
|
||||
}
|
||||
|
||||
// deadmanAlerts loads every alert row that a matcher claims. The candidate query
|
||||
// deadmanAlerts loads every alert row that one of a team's switches claims. The candidate query
|
||||
// is narrowed by alertname so it rides alerts_name_idx; the rest of the matching
|
||||
// happens in Go, which keeps one implementation of the rules. The rows are read
|
||||
// in full before the caller writes, so the writes do not run against an open
|
||||
// cursor over the same table.
|
||||
func deadmanAlerts(ctx context.Context, db *sql.DB, teamID int64, cfg DeadmanConfig) ([]deadmanAlert, error) {
|
||||
func deadmanAlerts(ctx context.Context, db *sql.DB, teamID int64, cfg deadmanSet) ([]deadmanAlert, error) {
|
||||
names := cfg.names()
|
||||
args := &sqlArgs{}
|
||||
nameList := make([]any, len(names))
|
||||
@@ -263,11 +303,11 @@ func deadmanAlerts(ctx context.Context, db *sql.DB, teamID int64, cfg DeadmanCon
|
||||
}
|
||||
json.Unmarshal([]byte(labelsJSON), &a.labels) //nolint:errcheck
|
||||
|
||||
m, ok := cfg.match(a.labels)
|
||||
sw, ok := cfg.match(a.labels)
|
||||
if !ok {
|
||||
continue
|
||||
}
|
||||
a.matcher = m
|
||||
a.sw = sw
|
||||
a.resolved = status == "resolved"
|
||||
out = append(out, a)
|
||||
}
|
||||
@@ -284,16 +324,16 @@ func deadmanAlerts(ctx context.Context, db *sql.DB, teamID int64, cfg DeadmanCon
|
||||
// incidentForGroup), and a source that is gone for good is a one-time page
|
||||
// rather than a nag. Only a heartbeat that comes back and dies again earns a new
|
||||
// incident.
|
||||
func deadmanDied(ctx context.Context, db *sql.DB, cfg DeadmanConfig, notify NotifyConfig, sw deadmanAlert, now time.Time) error {
|
||||
func deadmanDied(ctx context.Context, db *sql.DB, notify NotifyConfig, hb deadmanAlert, now time.Time) error {
|
||||
var lastTriggered, open int64
|
||||
if err := db.QueryRowContext(ctx, `
|
||||
SELECT COALESCE(MAX(triggered_at), 0),
|
||||
COUNT(*) FILTER (WHERE resolved_at IS NULL)
|
||||
FROM incidents WHERE team_id = $1 AND group_key = $2`,
|
||||
sw.teamID, sw.groupKey()).Scan(&lastTriggered, &open); err != nil {
|
||||
hb.teamID, hb.groupKey()).Scan(&lastTriggered, &open); err != nil {
|
||||
return err
|
||||
}
|
||||
if open > 0 || sw.receivedAt <= lastTriggered {
|
||||
if open > 0 || hb.receivedAt <= lastTriggered {
|
||||
return nil
|
||||
}
|
||||
|
||||
@@ -306,18 +346,18 @@ func deadmanDied(ctx context.Context, db *sql.DB, cfg DeadmanConfig, notify Noti
|
||||
// A heartbeat nobody has heard from is not firing, and saying otherwise in
|
||||
// the alert list would be a lie. An Alertmanager-sourced resolution keeps its
|
||||
// own source: it told us the truth first.
|
||||
if !sw.resolved {
|
||||
if !hb.resolved {
|
||||
if _, err := tx.ExecContext(ctx, `
|
||||
UPDATE alerts
|
||||
SET status = 'resolved',
|
||||
resolution_source = $1,
|
||||
ends_at = COALESCE(ends_at, `+nowEpoch+`)
|
||||
WHERE id = $2 AND status = 'firing'`, resolutionDeadman, sw.id); err != nil {
|
||||
WHERE id = $2 AND status = 'firing'`, resolutionDeadman, hb.id); err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
|
||||
severity := cfg.Severity
|
||||
severity := hb.sw.Severity
|
||||
var sev *string
|
||||
if severity != "" {
|
||||
sev = &severity
|
||||
@@ -325,14 +365,14 @@ func deadmanDied(ctx context.Context, db *sql.DB, cfg DeadmanConfig, notify Noti
|
||||
|
||||
// The incident opens in the team whose integration received the heartbeat:
|
||||
// the switch belongs to whoever is watching that source, not to the install.
|
||||
incidentID, err := openIncident(ctx, tx, notify, sw.teamID, sw.groupKey(),
|
||||
"No heartbeat from "+sw.matcher.String(), sw.labels, sev)
|
||||
incidentID, err := openIncident(ctx, tx, notify, hb.teamID, hb.groupKey(),
|
||||
"No heartbeat from "+hb.sw.Matcher.String(), hb.labels, sev)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
alertID := sw.id
|
||||
detail := "last heartbeat " + humanDuration(now.Sub(time.Unix(sw.receivedAt, 0))) + " ago"
|
||||
alertID := hb.id
|
||||
detail := "last heartbeat " + humanDuration(now.Sub(time.Unix(hb.receivedAt, 0))) + " ago"
|
||||
if err := logEvent(ctx, tx, incidentID, evDeadmanSilent, nil, &alertID, &detail); err != nil {
|
||||
return err
|
||||
}
|
||||
@@ -340,7 +380,7 @@ func deadmanDied(ctx context.Context, db *sql.DB, cfg DeadmanConfig, notify Noti
|
||||
if err := tx.Commit(); err != nil {
|
||||
return err
|
||||
}
|
||||
log.Printf("deadman: %s went silent, opened incident %d", sw.matcher.String(), incidentID)
|
||||
log.Printf("deadman: %s went silent, opened incident %d", hb.sw.Matcher.String(), incidentID)
|
||||
return nil
|
||||
}
|
||||
|
||||
@@ -350,12 +390,12 @@ func deadmanDied(ctx context.Context, db *sql.DB, cfg DeadmanConfig, notify Noti
|
||||
// member alerts (linking the heartbeat would have the settled-incident cascade
|
||||
// close it on the very same sweep that opened it), so the alert-driven cascade
|
||||
// ignores it entirely and recovery is the only automatic way out.
|
||||
func deadmanRecovered(ctx context.Context, db *sql.DB, sw deadmanAlert) error {
|
||||
func deadmanRecovered(ctx context.Context, db *sql.DB, hb deadmanAlert) error {
|
||||
var incidentID int64
|
||||
switch err := db.QueryRowContext(ctx, `
|
||||
SELECT id FROM incidents
|
||||
WHERE team_id = $1 AND group_key = $2 AND resolved_at IS NULL`,
|
||||
sw.teamID, sw.groupKey()).Scan(&incidentID); {
|
||||
hb.teamID, hb.groupKey()).Scan(&incidentID); {
|
||||
case err == sql.ErrNoRows:
|
||||
return nil
|
||||
case err != nil:
|
||||
@@ -387,116 +427,256 @@ func deadmanRecovered(ctx context.Context, db *sql.DB, sw deadmanAlert) error {
|
||||
if err := tx.Commit(); err != nil {
|
||||
return err
|
||||
}
|
||||
log.Printf("deadman: %s is back, resolved incident %d", sw.matcher.String(), incidentID)
|
||||
log.Printf("deadman: %s is back, resolved incident %d", hb.sw.Matcher.String(), incidentID)
|
||||
return nil
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// Per-team configuration
|
||||
// A team's switches
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
// deadmanConfigForTeam reads one team's switches. A team with no row, or with
|
||||
// nothing configured, gets a disabled config — which is the right answer rather
|
||||
// than an error: most teams watch no heartbeat at all.
|
||||
func deadmanConfigForTeam(ctx context.Context, q querier, teamID int64) (DeadmanConfig, error) {
|
||||
var matchers, severity string
|
||||
var timeout int64
|
||||
err := q.QueryRowContext(ctx,
|
||||
"SELECT matchers, timeout_seconds, severity FROM deadman_configs WHERE team_id = $1",
|
||||
teamID).Scan(&matchers, &timeout, &severity)
|
||||
if err == sql.ErrNoRows {
|
||||
return DeadmanConfig{}, nil
|
||||
}
|
||||
if err != nil {
|
||||
return DeadmanConfig{}, err
|
||||
}
|
||||
return parseDeadmanQuietly(matchers, time.Duration(timeout)*time.Second, severity), nil
|
||||
}
|
||||
const deadmanSwitchColumns = "id, team_id, name, matcher, timeout_seconds, severity"
|
||||
|
||||
// deadmanConfigs reads every team's switches in one query, for the sweeper.
|
||||
func deadmanConfigs(ctx context.Context, db *sql.DB) (map[int64]DeadmanConfig, error) {
|
||||
rows, err := db.QueryContext(ctx,
|
||||
"SELECT team_id, matchers, timeout_seconds, severity FROM deadman_configs")
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
// scanDeadmanSwitches reads switch rows into per-team sets. A row whose matcher
|
||||
// no longer parses is skipped rather than fatal: the API refuses to store one,
|
||||
// so it can only mean a hand edit, and one bad row must not stop the others
|
||||
// from being watched.
|
||||
func scanDeadmanSwitches(rows *sql.Rows) (map[int64]deadmanSet, error) {
|
||||
defer rows.Close()
|
||||
|
||||
out := map[int64]DeadmanConfig{}
|
||||
out := map[int64]deadmanSet{}
|
||||
for rows.Next() {
|
||||
var sw DeadmanSwitch
|
||||
var teamID, timeout int64
|
||||
var matchers, severity string
|
||||
if err := rows.Scan(&teamID, &matchers, &timeout, &severity); err != nil {
|
||||
var matcher string
|
||||
if err := rows.Scan(&sw.ID, &teamID, &sw.Name, &matcher, &timeout, &sw.Severity); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
cfg := parseDeadmanQuietly(matchers, time.Duration(timeout)*time.Second, severity)
|
||||
if cfg.enabled() {
|
||||
out[teamID] = cfg
|
||||
m, err := parseDeadmanMatcher(matcher)
|
||||
if err != nil {
|
||||
log.Printf("deadman: switch %d has an unusable matcher %q: %v", sw.ID, matcher, err)
|
||||
continue
|
||||
}
|
||||
sw.Matcher = m
|
||||
sw.Timeout = time.Duration(timeout) * time.Second
|
||||
out[teamID] = append(out[teamID], sw)
|
||||
}
|
||||
return out, rows.Err()
|
||||
}
|
||||
|
||||
// SeedDeadmanConfigs gives every team without a row the server's environment
|
||||
// configuration, so the install that upgrades into per-team switches keeps
|
||||
// watching exactly what it was watching before.
|
||||
// deadmanSetForTeam reads one team's switches. A team with none gets an empty
|
||||
// set — which is the right answer rather than an error: most teams watch no
|
||||
// heartbeat at all.
|
||||
func deadmanSetForTeam(ctx context.Context, q querier, teamID int64) (deadmanSet, error) {
|
||||
rows, err := q.QueryContext(ctx,
|
||||
"SELECT "+deadmanSwitchColumns+" FROM deadman_switches WHERE team_id = $1 ORDER BY id", teamID)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
sets, err := scanDeadmanSwitches(rows)
|
||||
return sets[teamID], err
|
||||
}
|
||||
|
||||
// deadmanSets reads every team's switches in one query, for the sweeper.
|
||||
func deadmanSets(ctx context.Context, db *sql.DB) (map[int64]deadmanSet, error) {
|
||||
rows, err := db.QueryContext(ctx,
|
||||
"SELECT "+deadmanSwitchColumns+" FROM deadman_switches ORDER BY id")
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return scanDeadmanSwitches(rows)
|
||||
}
|
||||
|
||||
// deadmanSeededKey is the settings row that records the environment defaults
|
||||
// were handed out. Without it, a team that deleted its last switch would get
|
||||
// the default back on the next restart.
|
||||
const deadmanSeededKey = "deadman_seeded"
|
||||
|
||||
// SeedDeadmanConfigs gives every team the server's environment defaults as
|
||||
// switches, exactly once per install, so a fresh install watches Watchdog
|
||||
// without anybody setting it up.
|
||||
//
|
||||
// Idempotent, and never overwrites: once a team has a row it owns its own
|
||||
// configuration, and a redeploy must not quietly put the environment's value
|
||||
// back over an owner's edit.
|
||||
// Once seeded it never runs again: a team's switches are its own, and a redeploy
|
||||
// must not quietly put the environment's value back over an owner's edit or
|
||||
// deletion. Installs that upgraded from per-team configuration were already
|
||||
// seeded, which migration 009 records.
|
||||
//
|
||||
// A team created after startup gets no row and therefore watches nothing until
|
||||
// its owner says otherwise. That is deliberate: inheriting an install-wide
|
||||
// heartbeat would page a new team about a source it has never heard of, and a
|
||||
// switch nobody chose is the kind that gets muted rather than fixed.
|
||||
// A team created after that gets none and watches nothing until its owner says
|
||||
// otherwise. That is deliberate: inheriting an install-wide heartbeat would page
|
||||
// a new team about a source it has never heard of, and a switch nobody chose is
|
||||
// the kind that gets muted rather than fixed.
|
||||
func SeedDeadmanConfigs(ctx context.Context, db *sql.DB, cfg DeadmanConfig) error {
|
||||
matchers := make([]string, 0, len(cfg.Matchers))
|
||||
if !cfg.enabled() {
|
||||
return nil
|
||||
}
|
||||
|
||||
tx, err := db.BeginTx(ctx, nil)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
defer tx.Rollback() //nolint:errcheck
|
||||
|
||||
res, err := tx.ExecContext(ctx,
|
||||
"INSERT INTO settings (key, value) VALUES ($1, '1') ON CONFLICT (key) DO NOTHING",
|
||||
deadmanSeededKey)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if n, _ := res.RowsAffected(); n == 0 {
|
||||
return nil
|
||||
}
|
||||
|
||||
for _, m := range cfg.Matchers {
|
||||
parts := []string{"alertname=" + m.Name}
|
||||
for k, v := range m.Labels {
|
||||
parts = append(parts, k+"="+v)
|
||||
if _, err := tx.ExecContext(ctx, `
|
||||
INSERT INTO deadman_switches (team_id, name, matcher, timeout_seconds, severity)
|
||||
SELECT id, $1, $1, $2, $3 FROM teams`,
|
||||
m.config(), int64(cfg.Timeout.Seconds()), cfg.Severity); err != nil {
|
||||
return err
|
||||
}
|
||||
sort.Strings(parts[1:])
|
||||
matchers = append(matchers, strings.Join(parts, ","))
|
||||
}
|
||||
|
||||
_, err := db.ExecContext(ctx, `
|
||||
INSERT INTO deadman_configs (team_id, matchers, timeout_seconds, severity)
|
||||
SELECT id, $1, $2, $3 FROM teams
|
||||
ON CONFLICT (team_id) DO NOTHING`,
|
||||
strings.Join(matchers, "; "), int64(cfg.Timeout.Seconds()), cfg.Severity)
|
||||
return err
|
||||
return tx.Commit()
|
||||
}
|
||||
|
||||
// parseDeadmanQuietly is ParseDeadmanConfig without the startup logging: a
|
||||
// team's configuration is read on every sweep and every webhook, and logging it
|
||||
// each time would bury everything else.
|
||||
func parseDeadmanQuietly(matchers string, timeout time.Duration, severity string) DeadmanConfig {
|
||||
cfg := DeadmanConfig{Timeout: timeout, Severity: severity}
|
||||
for _, entry := range strings.Split(matchers, ";") {
|
||||
entry = strings.TrimSpace(entry)
|
||||
if entry == "" {
|
||||
continue
|
||||
}
|
||||
m := DeadmanMatcher{Labels: map[string]string{}}
|
||||
malformed := false
|
||||
for _, cond := range strings.Split(entry, ",") {
|
||||
k, v, ok := strings.Cut(cond, "=")
|
||||
k, v = strings.TrimSpace(k), strings.TrimSpace(v)
|
||||
if !ok || k == "" || v == "" {
|
||||
malformed = true
|
||||
break
|
||||
}
|
||||
if k == "alertname" {
|
||||
m.Name = v
|
||||
continue
|
||||
}
|
||||
m.Labels[k] = v
|
||||
}
|
||||
if malformed || m.Name == "" {
|
||||
continue
|
||||
}
|
||||
cfg.Matchers = append(cfg.Matchers, m)
|
||||
}
|
||||
return cfg
|
||||
// ---------------------------------------------------------------------------
|
||||
// Status
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
const (
|
||||
switchHealthy = "healthy"
|
||||
switchDead = "dead"
|
||||
switchDormant = "dormant"
|
||||
)
|
||||
|
||||
// deadmanSource is one heartbeat under a switch: a fingerprint that matched.
|
||||
type deadmanSource struct {
|
||||
Fingerprint string `json:"fingerprint"`
|
||||
Labels map[string]string `json:"labels"`
|
||||
Status string `json:"status"`
|
||||
LastHeartbeatAt time.Time `json:"last_heartbeat_at"`
|
||||
LastTriggeredAt *time.Time `json:"last_triggered_at"`
|
||||
IncidentID *int64 `json:"incident_id"`
|
||||
}
|
||||
|
||||
// deadmanSwitchStatus is a switch as the Switches page shows it.
|
||||
type deadmanSwitchStatus struct {
|
||||
ID int64 `json:"id"`
|
||||
Name string `json:"name"`
|
||||
Matcher string `json:"matcher"`
|
||||
TimeoutSeconds int64 `json:"timeout_seconds"`
|
||||
Severity string `json:"severity"`
|
||||
|
||||
// Status is dead when any source is, dormant when none has ever been heard
|
||||
// from, healthy otherwise — a live cluster must not hide a dead one.
|
||||
Status string `json:"status"`
|
||||
LastHeartbeatAt *time.Time `json:"last_heartbeat_at"`
|
||||
LastTriggeredAt *time.Time `json:"last_triggered_at"`
|
||||
OpenIncidentID *int64 `json:"open_incident_id"`
|
||||
Sources []deadmanSource `json:"sources"`
|
||||
}
|
||||
|
||||
// deadmanStatuses reports every switch of a team with what its heartbeats are
|
||||
// doing. The liveness verdict is deadmanAlert.dead, the sweeper's own.
|
||||
func deadmanStatuses(ctx context.Context, db *sql.DB, teamID int64, set deadmanSet, now time.Time) ([]deadmanSwitchStatus, error) {
|
||||
out := make([]deadmanSwitchStatus, 0, len(set))
|
||||
if len(set) == 0 {
|
||||
return out, nil
|
||||
}
|
||||
|
||||
heartbeats, err := deadmanAlerts(ctx, db, teamID, set)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
// One query for every switch's incident history, keyed the way the sweeper
|
||||
// keys it.
|
||||
type history struct {
|
||||
triggeredAt int64
|
||||
openID int64
|
||||
}
|
||||
incidents := map[string]history{}
|
||||
rows, err := db.QueryContext(ctx, `
|
||||
SELECT group_key, MAX(triggered_at), COALESCE(MAX(id) FILTER (WHERE resolved_at IS NULL), 0)
|
||||
FROM incidents
|
||||
WHERE team_id = $1 AND group_key LIKE $2
|
||||
GROUP BY group_key`, teamID, deadmanGroupPrefix+"%")
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
defer rows.Close()
|
||||
for rows.Next() {
|
||||
var key string
|
||||
var h history
|
||||
if err := rows.Scan(&key, &h.triggeredAt, &h.openID); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
incidents[key] = h
|
||||
}
|
||||
if err := rows.Err(); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
bySwitch := map[int64][]deadmanAlert{}
|
||||
for _, hb := range heartbeats {
|
||||
bySwitch[hb.sw.ID] = append(bySwitch[hb.sw.ID], hb)
|
||||
}
|
||||
|
||||
later := func(cur *time.Time, unix int64) *time.Time {
|
||||
t := time.Unix(unix, 0).UTC()
|
||||
if cur == nil || t.After(*cur) {
|
||||
return &t
|
||||
}
|
||||
return cur
|
||||
}
|
||||
|
||||
for _, sw := range set {
|
||||
st := deadmanSwitchStatus{
|
||||
ID: sw.ID, Name: sw.Name, Matcher: sw.Matcher.config(),
|
||||
TimeoutSeconds: int64(sw.Timeout.Seconds()), Severity: sw.Severity,
|
||||
Status: switchDormant, Sources: []deadmanSource{},
|
||||
}
|
||||
|
||||
for _, hb := range bySwitch[sw.ID] {
|
||||
src := deadmanSource{
|
||||
Fingerprint: hb.fingerprint,
|
||||
Labels: hb.labels,
|
||||
Status: switchHealthy,
|
||||
LastHeartbeatAt: time.Unix(hb.receivedAt, 0).UTC(),
|
||||
}
|
||||
if hb.dead(now) {
|
||||
src.Status = switchDead
|
||||
}
|
||||
if h, ok := incidents[hb.groupKey()]; ok {
|
||||
t := time.Unix(h.triggeredAt, 0).UTC()
|
||||
src.LastTriggeredAt = &t
|
||||
st.LastTriggeredAt = later(st.LastTriggeredAt, h.triggeredAt)
|
||||
if h.openID != 0 {
|
||||
id := h.openID
|
||||
src.IncidentID = &id
|
||||
if st.OpenIncidentID == nil || id > *st.OpenIncidentID {
|
||||
st.OpenIncidentID = &id
|
||||
}
|
||||
}
|
||||
}
|
||||
st.LastHeartbeatAt = later(st.LastHeartbeatAt, hb.receivedAt)
|
||||
st.Sources = append(st.Sources, src)
|
||||
|
||||
switch {
|
||||
case src.Status == switchDead:
|
||||
st.Status = switchDead
|
||||
case st.Status == switchDormant:
|
||||
st.Status = switchHealthy
|
||||
}
|
||||
}
|
||||
|
||||
// Dead ones first, then by fingerprint: what needs attention leads, and
|
||||
// the order does not shuffle between refreshes.
|
||||
sort.Slice(st.Sources, func(i, j int) bool {
|
||||
a, b := st.Sources[i], st.Sources[j]
|
||||
if (a.Status == switchDead) != (b.Status == switchDead) {
|
||||
return a.Status == switchDead
|
||||
}
|
||||
return a.Fingerprint < b.Fingerprint
|
||||
})
|
||||
out = append(out, st)
|
||||
}
|
||||
return out, nil
|
||||
}
|
||||
|
||||
+166
-16
@@ -1,6 +1,7 @@
|
||||
package api_test
|
||||
|
||||
import (
|
||||
"context"
|
||||
"net/http"
|
||||
"strings"
|
||||
"testing"
|
||||
@@ -483,13 +484,13 @@ func TestDeadman_ConfigurationIsPerTeam(t *testing.T) {
|
||||
unwatched := newTeam(t, s, "unwatched")
|
||||
|
||||
// Only the first team calls Watchdog a heartbeat.
|
||||
resp := s.req(t, http.MethodPut, "/api/teams/"+id64(watched.id)+"/deadman", map[string]any{
|
||||
"matchers": "alertname=Watchdog",
|
||||
resp := s.req(t, http.MethodPost, "/api/teams/"+id64(watched.id)+"/deadman/switches", map[string]any{
|
||||
"matcher": "alertname=Watchdog",
|
||||
"timeout_seconds": 3600,
|
||||
"severity": "critical",
|
||||
})
|
||||
resp.Body.Close()
|
||||
if resp.StatusCode != http.StatusOK {
|
||||
if resp.StatusCode != http.StatusCreated {
|
||||
t.Fatalf("configure the watched team: %d", resp.StatusCode)
|
||||
}
|
||||
|
||||
@@ -550,9 +551,9 @@ func TestDeadman_ConfigurationIsOwnerOnly(t *testing.T) {
|
||||
decode(t, s.req(t, http.MethodPost, "/api/users/"+id64(user.ID)+"/api-keys",
|
||||
map[string]string{"name": "test"}), &key)
|
||||
|
||||
req, _ := http.NewRequest(http.MethodPut,
|
||||
s.URL+"/api/teams/"+id64(team.id)+"/deadman",
|
||||
strings.NewReader(`{"matchers":"alertname=Watchdog","timeout_seconds":60}`))
|
||||
req, _ := http.NewRequest(http.MethodPost,
|
||||
s.URL+"/api/teams/"+id64(team.id)+"/deadman/switches",
|
||||
strings.NewReader(`{"matcher":"alertname=Watchdog","timeout_seconds":60}`))
|
||||
req.Header.Set("Authorization", "Bearer "+key.Key)
|
||||
req.Header.Set("Content-Type", "application/json")
|
||||
resp, err := http.DefaultClient.Do(req)
|
||||
@@ -564,7 +565,7 @@ func TestDeadman_ConfigurationIsOwnerOnly(t *testing.T) {
|
||||
t.Errorf("a member editing the switches: expected 403, got %d", resp.StatusCode)
|
||||
}
|
||||
|
||||
read, _ := http.NewRequest(http.MethodGet, s.URL+"/api/teams/"+id64(team.id)+"/deadman", nil)
|
||||
read, _ := http.NewRequest(http.MethodGet, s.URL+"/api/teams/"+id64(team.id)+"/deadman/switches", nil)
|
||||
read.Header.Set("Authorization", "Bearer "+key.Key)
|
||||
got, err := http.DefaultClient.Do(read)
|
||||
if err != nil {
|
||||
@@ -577,16 +578,165 @@ func TestDeadman_ConfigurationIsOwnerOnly(t *testing.T) {
|
||||
}
|
||||
|
||||
// A matcher with no alertname watches nothing, silently, which is the failure
|
||||
// this feature exists to prevent — so it is refused at the door.
|
||||
func TestDeadman_UnusableMatchersAreRejected(t *testing.T) {
|
||||
// this feature exists to prevent — so it is refused at the door, along with the
|
||||
// other things that would make a switch unable to fire.
|
||||
func TestDeadman_UnusableSwitchesAreRejected(t *testing.T) {
|
||||
s, _ := deadmanTS(t, deadmanCfg())
|
||||
|
||||
resp := s.req(t, http.MethodPut, "/api/teams/"+defaultTeam+"/deadman", map[string]any{
|
||||
"matchers": "cluster=prod",
|
||||
"timeout_seconds": 900,
|
||||
})
|
||||
resp.Body.Close()
|
||||
if resp.StatusCode != http.StatusBadRequest {
|
||||
t.Errorf("expected 400 for a matcher with no alertname, got %d", resp.StatusCode)
|
||||
for name, body := range map[string]map[string]any{
|
||||
"no alertname": {"matcher": "cluster=prod", "timeout_seconds": 900},
|
||||
"malformed": {"matcher": "alertname=Watchdog,garbage", "timeout_seconds": 900},
|
||||
"several": {"matcher": "alertname=A; alertname=B", "timeout_seconds": 900},
|
||||
"zero timeout": {"matcher": "alertname=Watchdog", "timeout_seconds": 0},
|
||||
"bad severity": {"matcher": "alertname=Watchdog", "timeout_seconds": 900, "severity": "loud"},
|
||||
"empty matcher": {"matcher": "", "timeout_seconds": 900},
|
||||
} {
|
||||
resp := s.req(t, http.MethodPost, "/api/teams/"+defaultTeam+"/deadman/switches", body)
|
||||
resp.Body.Close()
|
||||
if resp.StatusCode != http.StatusBadRequest {
|
||||
t.Errorf("%s: expected 400, got %d", name, resp.StatusCode)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// The switch list
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
// listSwitches reads the default team's switches as the Switches page does.
|
||||
func listSwitches(t *testing.T, s *ts) []map[string]any {
|
||||
t.Helper()
|
||||
return list(t, s.req(t, http.MethodGet, "/api/teams/"+defaultTeam+"/deadman/switches", nil))
|
||||
}
|
||||
|
||||
// A switch is healthy while its heartbeat is fresh, dead once it is silent, and
|
||||
// dormant until the first one arrives.
|
||||
func TestDeadman_ListReportsStatus(t *testing.T) {
|
||||
s, _ := deadmanTS(t, api.ParseDeadmanConfig("alertname=Watchdog; alertname=NeverSent", time.Hour, "critical"))
|
||||
|
||||
got := listSwitches(t, s)
|
||||
if len(got) != 2 {
|
||||
t.Fatalf("expected 2 switches, got %d", len(got))
|
||||
}
|
||||
for _, sw := range got {
|
||||
if sw["status"] != "dormant" || sw["last_heartbeat_at"] != nil || sw["last_triggered_at"] != nil {
|
||||
t.Errorf("a switch nobody has heard from should be dormant and blank, got %v", sw)
|
||||
}
|
||||
}
|
||||
|
||||
heartbeat(t, s, "fp-watchdog", nil)
|
||||
got = listSwitches(t, s)
|
||||
if got[0]["status"] != "healthy" || got[0]["last_heartbeat_at"] == nil {
|
||||
t.Errorf("a fresh heartbeat should be healthy with a timestamp, got %v", got[0])
|
||||
}
|
||||
if got[1]["status"] != "dormant" {
|
||||
t.Errorf("the other switch is still dormant, got %v", got[1]["status"])
|
||||
}
|
||||
|
||||
silence(t, s, "fp-watchdog", 2*time.Hour)
|
||||
sweep(t, s, noArchive)
|
||||
got = listSwitches(t, s)
|
||||
if got[0]["status"] != "dead" {
|
||||
t.Fatalf("a silent heartbeat should be dead, got %v", got[0]["status"])
|
||||
}
|
||||
if got[0]["last_triggered_at"] == nil || got[0]["open_incident_id"] == nil {
|
||||
t.Errorf("a dead switch should show when it triggered and its open incident, got %v", got[0])
|
||||
}
|
||||
}
|
||||
|
||||
// One matcher, several clusters: the switch is as bad as its worst heartbeat and
|
||||
// each heartbeat is listed on its own.
|
||||
func TestDeadman_ListBreaksDownByFingerprint(t *testing.T) {
|
||||
s, _ := deadmanTS(t, deadmanCfg())
|
||||
|
||||
heartbeat(t, s, "fp-a", map[string]string{"cluster": "a"})
|
||||
heartbeat(t, s, "fp-b", map[string]string{"cluster": "b"})
|
||||
silence(t, s, "fp-b", 2*time.Hour)
|
||||
|
||||
sw := listSwitches(t, s)[0]
|
||||
if sw["status"] != "dead" {
|
||||
t.Errorf("one dead cluster makes the switch dead, got %v", sw["status"])
|
||||
}
|
||||
sources := sw["sources"].([]any)
|
||||
if len(sources) != 2 {
|
||||
t.Fatalf("expected 2 sources, got %d", len(sources))
|
||||
}
|
||||
first, second := sources[0].(map[string]any), sources[1].(map[string]any)
|
||||
if first["fingerprint"] != "fp-b" || first["status"] != "dead" || second["status"] != "healthy" {
|
||||
t.Errorf("the dead source should lead, got %v then %v", first, second)
|
||||
}
|
||||
}
|
||||
|
||||
// Every switch keeps its own deadline.
|
||||
func TestDeadman_TimeoutsArePerSwitch(t *testing.T) {
|
||||
s, _ := deadmanTS(t, deadmanCfg())
|
||||
resp := s.req(t, http.MethodPost, "/api/teams/"+defaultTeam+"/deadman/switches", map[string]any{
|
||||
"matcher": "alertname=Edge", "timeout_seconds": 300,
|
||||
})
|
||||
resp.Body.Close()
|
||||
|
||||
heartbeat(t, s, "fp-watchdog", nil)
|
||||
postWebhook(t, s, []map[string]any{
|
||||
amAlert("fp-edge", "Edge", "firing", "2026-05-20T10:00:00Z", zeroTime, nil),
|
||||
}, `{}:{alertname="Edge"}`)
|
||||
|
||||
// Ten minutes of silence: past the Edge switch's five, inside Watchdog's hour.
|
||||
silence(t, s, "fp-watchdog", 10*time.Minute)
|
||||
silence(t, s, "fp-edge", 10*time.Minute)
|
||||
|
||||
got := listSwitches(t, s)
|
||||
if got[0]["status"] != "healthy" || got[1]["status"] != "dead" {
|
||||
t.Errorf("want Watchdog healthy and Edge dead, got %v and %v", got[0]["status"], got[1]["status"])
|
||||
}
|
||||
}
|
||||
|
||||
// Deleting is an owner's, is scoped to the team, and leaves what the switch
|
||||
// already opened alone.
|
||||
func TestDeadman_DeleteIsScopedToTheTeam(t *testing.T) {
|
||||
s, _ := deadmanTS(t, deadmanCfg())
|
||||
other := newTeam(t, s, "other")
|
||||
|
||||
id := int64(listSwitches(t, s)[0]["id"].(float64))
|
||||
|
||||
// Another team's owner cannot reach it.
|
||||
resp := other.call(http.MethodDelete, "/api/teams/"+id64(other.id)+"/deadman/switches/"+id64(id), nil)
|
||||
resp.Body.Close()
|
||||
if resp.StatusCode != http.StatusNotFound {
|
||||
t.Errorf("deleting another team's switch: expected 404, got %d", resp.StatusCode)
|
||||
}
|
||||
if got := len(listSwitches(t, s)); got != 1 {
|
||||
t.Fatalf("the switch should have survived, %d left", got)
|
||||
}
|
||||
|
||||
resp = s.req(t, http.MethodDelete, "/api/teams/"+defaultTeam+"/deadman/switches/"+id64(id), nil)
|
||||
resp.Body.Close()
|
||||
if resp.StatusCode != http.StatusNoContent {
|
||||
t.Fatalf("deleting: expected 204, got %d", resp.StatusCode)
|
||||
}
|
||||
if got := len(listSwitches(t, s)); got != 0 {
|
||||
t.Errorf("expected no switches, got %d", got)
|
||||
}
|
||||
}
|
||||
|
||||
// The environment's defaults are handed out once and then belong to the teams.
|
||||
func TestDeadman_SeedRunsOnce(t *testing.T) {
|
||||
s := newTS(t)
|
||||
cfg := api.ParseDeadmanConfig("alertname=Watchdog", time.Hour, "critical")
|
||||
|
||||
if err := api.SeedDeadmanConfigs(context.Background(), s.db, cfg); err != nil {
|
||||
t.Fatalf("seed: %v", err)
|
||||
}
|
||||
if got := len(listSwitches(t, s)); got != 1 {
|
||||
t.Fatalf("the first seed should add the default, got %d switches", got)
|
||||
}
|
||||
|
||||
// The owner deletes it; a restart must not put it back.
|
||||
id := int64(listSwitches(t, s)[0]["id"].(float64))
|
||||
s.req(t, http.MethodDelete, "/api/teams/"+defaultTeam+"/deadman/switches/"+id64(id), nil).Body.Close()
|
||||
if err := api.SeedDeadmanConfigs(context.Background(), s.db, cfg); err != nil {
|
||||
t.Fatalf("seed again: %v", err)
|
||||
}
|
||||
if got := len(listSwitches(t, s)); got != 0 {
|
||||
t.Errorf("a second seed resurrected %d switch(es)", got)
|
||||
}
|
||||
}
|
||||
|
||||
@@ -144,8 +144,9 @@ func NewRouter(db *sql.DB, notify NotifyConfig, cfg config.Config) http.Handler
|
||||
|
||||
// A team's own dead man's switches: which of its alerts are heartbeats,
|
||||
// and how long a silence has to last before somebody is paged.
|
||||
r.Get("/api/teams/{teamID}/deadman", handleGetTeamDeadman(db))
|
||||
r.Put("/api/teams/{teamID}/deadman", handleSetTeamDeadman(db))
|
||||
r.Get("/api/teams/{teamID}/deadman/switches", handleListTeamDeadman(db))
|
||||
r.Post("/api/teams/{teamID}/deadman/switches", handleCreateTeamDeadman(db))
|
||||
r.Delete("/api/teams/{teamID}/deadman/switches/{switchID}", handleDeleteTeamDeadman(db))
|
||||
|
||||
// Integrations: where a team's alerts come in, and the key that says so.
|
||||
r.Get("/api/teams/{teamID}/integrations", handleListIntegrations(db))
|
||||
|
||||
+92
-49
@@ -539,17 +539,24 @@ func defaultTeamID(ctx context.Context, db *sql.DB) (int64, error) {
|
||||
// A team's dead man's switches
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
// deadmanResponse is the wire shape of a team's switch configuration. The
|
||||
// timeout is seconds rather than a duration string, because that is what the
|
||||
// column holds and what arithmetic is done on; a client renders it.
|
||||
type deadmanResponse struct {
|
||||
TeamID int64 `json:"team_id"`
|
||||
Matchers string `json:"matchers"`
|
||||
// deadmanSwitchRequest is what creating a switch takes. The timeout is seconds,
|
||||
// because that is what the column holds and what arithmetic is done on; a client
|
||||
// renders it.
|
||||
type deadmanSwitchRequest struct {
|
||||
Name string `json:"name"`
|
||||
Matcher string `json:"matcher"`
|
||||
TimeoutSeconds int64 `json:"timeout_seconds"`
|
||||
Severity string `json:"severity"`
|
||||
}
|
||||
|
||||
func handleGetTeamDeadman(db *sql.DB) http.HandlerFunc {
|
||||
// deadmanSeverities are the severities an incident can open at.
|
||||
var deadmanSeverities = map[string]bool{"critical": true, "error": true, "warning": true, "info": true}
|
||||
|
||||
// handleListTeamDeadman lists a team's switches with what each one's heartbeats
|
||||
// are doing. A team with none gets an empty list, which is a configuration and
|
||||
// not an absence: answering 404 would make "off" indistinguishable from "this
|
||||
// server does not do this".
|
||||
func handleListTeamDeadman(db *sql.DB) http.HandlerFunc {
|
||||
return func(w http.ResponseWriter, r *http.Request) {
|
||||
teamID, ok := teamParam(w, r)
|
||||
if !ok {
|
||||
@@ -559,27 +566,26 @@ func handleGetTeamDeadman(db *sql.DB) http.HandlerFunc {
|
||||
return
|
||||
}
|
||||
|
||||
out := deadmanResponse{TeamID: teamID, Severity: "critical"}
|
||||
err := db.QueryRowContext(r.Context(),
|
||||
"SELECT matchers, timeout_seconds, severity FROM deadman_configs WHERE team_id = $1",
|
||||
teamID).Scan(&out.Matchers, &out.TimeoutSeconds, &out.Severity)
|
||||
if err != nil && !errors.Is(err, sql.ErrNoRows) {
|
||||
set, err := deadmanSetForTeam(r.Context(), db, teamID)
|
||||
if err != nil {
|
||||
respond(w, http.StatusInternalServerError, errResp("internal error"))
|
||||
return
|
||||
}
|
||||
out, err := deadmanStatuses(r.Context(), db, teamID, set, time.Now())
|
||||
if err != nil {
|
||||
respond(w, http.StatusInternalServerError, errResp("internal error"))
|
||||
return
|
||||
}
|
||||
// A team with no row watches nothing, which is a configuration and not
|
||||
// an absence: answering 404 would make "off" indistinguishable from
|
||||
// "this server does not do this".
|
||||
respond(w, http.StatusOK, out)
|
||||
}
|
||||
}
|
||||
|
||||
// handleSetTeamDeadman replaces a team's switch configuration.
|
||||
// handleCreateTeamDeadman adds one switch.
|
||||
//
|
||||
// Validated by parsing: a matcher string that survives ParseDeadmanConfig with
|
||||
// nothing usable in it is rejected rather than stored, because a switch that
|
||||
// silently watches nothing is the failure this feature exists to prevent.
|
||||
func handleSetTeamDeadman(db *sql.DB) http.HandlerFunc {
|
||||
// Validated by parsing: a matcher with no alertname is rejected rather than
|
||||
// stored, because a switch that silently watches nothing is the failure this
|
||||
// feature exists to prevent.
|
||||
func handleCreateTeamDeadman(db *sql.DB) http.HandlerFunc {
|
||||
return func(w http.ResponseWriter, r *http.Request) {
|
||||
teamID, ok := teamParam(w, r)
|
||||
if !ok {
|
||||
@@ -589,50 +595,87 @@ func handleSetTeamDeadman(db *sql.DB) http.HandlerFunc {
|
||||
return
|
||||
}
|
||||
|
||||
var req struct {
|
||||
Matchers string `json:"matchers"`
|
||||
TimeoutSeconds int64 `json:"timeout_seconds"`
|
||||
Severity string `json:"severity"`
|
||||
}
|
||||
var req deadmanSwitchRequest
|
||||
if err := decodeJSON(r, &req); err != nil {
|
||||
respond(w, http.StatusBadRequest, errResp("invalid request body"))
|
||||
return
|
||||
}
|
||||
req.Matchers = strings.TrimSpace(req.Matchers)
|
||||
req.Matcher = strings.TrimSpace(req.Matcher)
|
||||
req.Name = strings.TrimSpace(req.Name)
|
||||
if req.Severity == "" {
|
||||
req.Severity = "critical"
|
||||
}
|
||||
if req.TimeoutSeconds < 0 {
|
||||
respond(w, http.StatusBadRequest, errResp("timeout_seconds must not be negative"))
|
||||
if !deadmanSeverities[req.Severity] {
|
||||
respond(w, http.StatusBadRequest, errResp("severity must be critical, error, warning or info"))
|
||||
return
|
||||
}
|
||||
if req.Matchers != "" {
|
||||
parsed := parseDeadmanQuietly(req.Matchers, time.Duration(req.TimeoutSeconds)*time.Second, req.Severity)
|
||||
if len(parsed.Matchers) == 0 {
|
||||
respond(w, http.StatusBadRequest, errResp(
|
||||
"no usable matchers: each must name an alertname, as in alertname=Watchdog,cluster=prod"))
|
||||
return
|
||||
}
|
||||
if req.TimeoutSeconds <= 0 {
|
||||
respond(w, http.StatusBadRequest, errResp("timeout_seconds must be positive"))
|
||||
return
|
||||
}
|
||||
if strings.Contains(req.Matcher, ";") {
|
||||
respond(w, http.StatusBadRequest, errResp("one matcher per switch: add another switch instead of separating with ;"))
|
||||
return
|
||||
}
|
||||
m, err := parseDeadmanMatcher(req.Matcher)
|
||||
if err != nil {
|
||||
respond(w, http.StatusBadRequest, errResp(
|
||||
"unusable matcher ("+err.Error()+"): each must name an alertname, as in alertname=Watchdog,cluster=prod"))
|
||||
return
|
||||
}
|
||||
if req.Name == "" {
|
||||
req.Name = m.config()
|
||||
}
|
||||
if len(req.Name) > 100 {
|
||||
respond(w, http.StatusBadRequest, errResp("name is too long"))
|
||||
return
|
||||
}
|
||||
|
||||
if _, err := db.ExecContext(r.Context(), `
|
||||
INSERT INTO deadman_configs (team_id, matchers, timeout_seconds, severity, updated_at)
|
||||
VALUES ($1, $2, $3, $4, `+nowEpoch+`)
|
||||
ON CONFLICT (team_id) DO UPDATE SET
|
||||
matchers = excluded.matchers,
|
||||
timeout_seconds = excluded.timeout_seconds,
|
||||
severity = excluded.severity,
|
||||
updated_at = excluded.updated_at`,
|
||||
teamID, req.Matchers, req.TimeoutSeconds, req.Severity); err != nil {
|
||||
var id int64
|
||||
if err := db.QueryRowContext(r.Context(), `
|
||||
INSERT INTO deadman_switches (team_id, name, matcher, timeout_seconds, severity)
|
||||
VALUES ($1, $2, $3, $4, $5) RETURNING id`,
|
||||
teamID, req.Name, m.config(), req.TimeoutSeconds, req.Severity).Scan(&id); err != nil {
|
||||
respond(w, http.StatusInternalServerError, errResp("internal error"))
|
||||
return
|
||||
}
|
||||
|
||||
respond(w, http.StatusOK, deadmanResponse{
|
||||
TeamID: teamID,
|
||||
Matchers: req.Matchers,
|
||||
TimeoutSeconds: req.TimeoutSeconds,
|
||||
Severity: req.Severity,
|
||||
respond(w, http.StatusCreated, deadmanSwitchStatus{
|
||||
ID: id, Name: req.Name, Matcher: m.config(),
|
||||
TimeoutSeconds: req.TimeoutSeconds, Severity: req.Severity,
|
||||
Status: switchDormant, Sources: []deadmanSource{},
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
// handleDeleteTeamDeadman removes a switch. An incident it already opened stays
|
||||
// open until somebody resolves it: deleting the switch says "stop watching", not
|
||||
// "the problem is gone".
|
||||
func handleDeleteTeamDeadman(db *sql.DB) http.HandlerFunc {
|
||||
return func(w http.ResponseWriter, r *http.Request) {
|
||||
teamID, ok := teamParam(w, r)
|
||||
if !ok {
|
||||
return
|
||||
}
|
||||
if !requireTeamOwner(w, r, teamID) {
|
||||
return
|
||||
}
|
||||
switchID, err := strconv.ParseInt(chi.URLParam(r, "switchID"), 10, 64)
|
||||
if err != nil {
|
||||
respond(w, http.StatusBadRequest, errResp("invalid switch id"))
|
||||
return
|
||||
}
|
||||
|
||||
res, err := db.ExecContext(r.Context(),
|
||||
"DELETE FROM deadman_switches WHERE id = $1 AND team_id = $2", switchID, teamID)
|
||||
if err != nil {
|
||||
respond(w, http.StatusInternalServerError, errResp("internal error"))
|
||||
return
|
||||
}
|
||||
if n, _ := res.RowsAffected(); n == 0 {
|
||||
respond(w, http.StatusNotFound, errResp("switch not found"))
|
||||
return
|
||||
}
|
||||
w.WriteHeader(http.StatusNoContent)
|
||||
}
|
||||
}
|
||||
|
||||
@@ -0,0 +1,54 @@
|
||||
-- Dead man's switches become rows of their own.
|
||||
--
|
||||
-- 004 kept a team's switches in one string with one timeout and one severity,
|
||||
-- which was enough to configure them and not enough to show them: there was no
|
||||
-- thing to list, nothing to hang a status on, and every switch in a team had to
|
||||
-- share a deadline. A row per switch gives each its own name, matcher, timeout
|
||||
-- and severity, and gives the Team → Switches page something to be a list of.
|
||||
--
|
||||
-- The matcher keeps the syntax the string used, one matcher per row:
|
||||
-- `alertname=Watchdog,cluster=prod`. The unit of monitoring is still the
|
||||
-- fingerprint, so a matcher that many clusters satisfy is still one switch row
|
||||
-- watching several independent heartbeats.
|
||||
CREATE TABLE deadman_switches (
|
||||
id BIGSERIAL PRIMARY KEY,
|
||||
team_id BIGINT NOT NULL REFERENCES teams(id) ON DELETE CASCADE,
|
||||
|
||||
-- What the owner calls it. Defaults to the matcher when they do not say.
|
||||
name TEXT NOT NULL,
|
||||
|
||||
-- "," separates the label conditions, "=" is exact equality, and alertname is
|
||||
-- mandatory: it is what keeps the sweeper's candidate query on an index.
|
||||
matcher TEXT NOT NULL,
|
||||
|
||||
-- Seconds of silence before the switch is declared dead. Never zero: a switch
|
||||
-- that cannot fire is deleted, not disabled.
|
||||
timeout_seconds BIGINT NOT NULL CHECK (timeout_seconds > 0),
|
||||
|
||||
-- The severity its incidents open at. See 004 for why they carry their own.
|
||||
severity TEXT NOT NULL DEFAULT 'critical',
|
||||
|
||||
created_at BIGINT NOT NULL DEFAULT FLOOR(EXTRACT(EPOCH FROM now()))::bigint
|
||||
);
|
||||
|
||||
CREATE INDEX deadman_switches_team_idx ON deadman_switches (team_id);
|
||||
|
||||
-- Carry every team's configuration over, one row per matcher. A team whose
|
||||
-- timeout was zero had switches turned off, which is now "no rows".
|
||||
INSERT INTO deadman_switches (team_id, name, matcher, timeout_seconds, severity)
|
||||
SELECT c.team_id, btrim(m), btrim(m), c.timeout_seconds, c.severity
|
||||
FROM deadman_configs c,
|
||||
LATERAL regexp_split_to_table(c.matchers, ';') AS m
|
||||
WHERE c.timeout_seconds > 0
|
||||
AND btrim(m) <> ''
|
||||
ORDER BY c.team_id;
|
||||
|
||||
-- The server seeds environment defaults into teams once, and remembers that it
|
||||
-- did. An install that had a row per team was already seeded; without this
|
||||
-- marker the first start after upgrading would seed teams that had switched
|
||||
-- theirs off.
|
||||
INSERT INTO settings (key, value)
|
||||
SELECT 'deadman_seeded', '1'
|
||||
WHERE EXISTS (SELECT 1 FROM deadman_configs);
|
||||
|
||||
DROP TABLE deadman_configs;
|
||||
@@ -359,7 +359,11 @@ input:focus, textarea:focus { outline: none; border-color: var(--accent); box-sh
|
||||
.badge.st-triggered, .badge.st-firing { background: var(--crit-soft); color: var(--crit); }
|
||||
.badge.st-acknowledged { background: var(--warn-soft); color: var(--warn); }
|
||||
.badge.st-snoozed { background: var(--snooze-soft); color: var(--snooze); }
|
||||
.badge.st-resolved { background: var(--ok-soft); color: var(--ok); }
|
||||
.badge.st-resolved, .badge.st-healthy { background: var(--ok-soft); color: var(--ok); }
|
||||
.badge.st-dead { background: var(--crit-soft); color: var(--crit); }
|
||||
/* Dormant is the plain badge on purpose: nothing has gone wrong and nothing has
|
||||
gone right, which is what the muted default already says. */
|
||||
.badge.st-dormant { background: var(--surface-2); color: var(--muted); }
|
||||
.badge.sev-critical { background: var(--crit-soft); color: var(--crit); }
|
||||
.badge.sev-warning { background: var(--warn-soft); color: var(--warn); }
|
||||
.badge.sev-info { background: var(--info-soft); color: var(--info); }
|
||||
@@ -691,6 +695,18 @@ kbd {
|
||||
.disabled-row td { opacity: 0.55; }
|
||||
.btn-sm.danger { color: var(--crit); border-color: var(--crit-soft); }
|
||||
|
||||
/* --- dead man's switches -------------------------------------------------
|
||||
Six columns do not fit a phone, so the table scrolls inside its card rather
|
||||
than the page. A heartbeat under a switch with several is indented, the way
|
||||
the escalation ladder indents its levels. */
|
||||
.card-head { display: flex; align-items: center; justify-content: space-between; gap: 12px; flex-wrap: wrap; }
|
||||
.table-scroll { overflow-x: auto; margin-top: 12px; }
|
||||
.switch-table th, .switch-table td { white-space: nowrap; }
|
||||
.switch-table td:nth-child(2) { white-space: normal; min-width: 12em; }
|
||||
.switch-table .source-row td { border-bottom-style: dashed; }
|
||||
.switch-table .source-row td:first-child { padding-left: 16px; }
|
||||
.source-labels { display: flex; flex-wrap: wrap; gap: 4px; align-items: center; }
|
||||
|
||||
.inline-form { display: flex; gap: 8px; margin-top: 12px; }
|
||||
.inline-form input { flex: 1; min-width: 0; }
|
||||
|
||||
|
||||
@@ -136,8 +136,11 @@ export const createIntegration = (id, name) =>
|
||||
export const deleteIntegration = (id, integrationID) =>
|
||||
call('DELETE', `/teams/${id}/integrations/${integrationID}`);
|
||||
|
||||
export const deadman = (id) => call('GET', `/teams/${id}/deadman`);
|
||||
export const setDeadman = (id, body) => call('PUT', `/teams/${id}/deadman`, { body });
|
||||
export const deadmanSwitches = (id) => call('GET', `/teams/${id}/deadman/switches`);
|
||||
export const createDeadmanSwitch = (id, body) =>
|
||||
call('POST', `/teams/${id}/deadman/switches`, { body });
|
||||
export const deleteDeadmanSwitch = (id, switchID) =>
|
||||
call('DELETE', `/teams/${id}/deadman/switches/${switchID}`);
|
||||
|
||||
export const escalation = (id) => call('GET', `/teams/${id}/escalation`);
|
||||
export const setEscalation = (id, body) => call('PUT', `/teams/${id}/escalation`, { body });
|
||||
|
||||
+141
-40
@@ -18,9 +18,9 @@
|
||||
// than no form, but it is not the thing enforcing anything.
|
||||
|
||||
import * as api from './api.js';
|
||||
import { h, clear, spinner, confirm, icon, openSheet, closeSheet, menuCard } from './ui.js';
|
||||
import { h, clear, spinner, confirm, icon, openSheet, closeSheet, menuCard, badge, labelChip } from './ui.js';
|
||||
import { state, currentTeam, users as allUsers, myID } from './state.js';
|
||||
import { isoDate, addDays, mondayOf, initial } from './format.js';
|
||||
import { isoDate, addDays, mondayOf, initial, ago, when, duration } from './format.js';
|
||||
|
||||
const view = () => document.getElementById('view-team');
|
||||
|
||||
@@ -111,14 +111,14 @@ async function load(id) {
|
||||
return { members, escalation };
|
||||
}
|
||||
if (tab === 'sources') return { integrations: await api.integrations(id) };
|
||||
if (tab === 'deadman') return { deadman: await api.deadman(id) };
|
||||
if (tab === 'deadman') return { deadman: await api.deadmanSwitches(id) };
|
||||
|
||||
const grid = gridDays();
|
||||
const [members, integrations, escalation, deadman, schedule] = await Promise.all([
|
||||
api.teamMembers(id),
|
||||
api.integrations(id),
|
||||
api.escalation(id),
|
||||
api.deadman(id),
|
||||
api.deadmanSwitches(id),
|
||||
api.schedule(id, isoDate(grid.start), isoDate(addDays(grid.start, grid.count - 1))),
|
||||
]);
|
||||
return { members, integrations, escalation, deadman, schedule };
|
||||
@@ -203,8 +203,8 @@ function overview() {
|
||||
const levels = (data.escalation?.levels || []).length;
|
||||
const keys = (data.integrations || []).length;
|
||||
const unused = (data.integrations || []).filter((i) => !i.last_used_at).length;
|
||||
const switches = (data.deadman?.matchers || '')
|
||||
.split(';').map((x) => x.trim()).filter(Boolean).length;
|
||||
const switches = (data.deadman || []).length;
|
||||
const dead = (data.deadman || []).filter((s) => s.status === 'dead').length;
|
||||
|
||||
return h('div', { class: 'overview-menu' },
|
||||
menuCard('/team/rota', 'Rota', null,
|
||||
@@ -220,7 +220,9 @@ function overview() {
|
||||
? (unused ? `${unused} of them never used.` : 'All in use.')
|
||||
: 'No key yet, so nothing can reach this team.'),
|
||||
menuCard('/team/deadman', 'Dead man’s switches', switches || null,
|
||||
switches ? 'Alerts whose absence opens an incident.' : 'Nothing watched.'),
|
||||
switches
|
||||
? (dead ? `${dead} of them silent.` : 'All quiet, as they should be.')
|
||||
: 'Nothing watched.'),
|
||||
);
|
||||
}
|
||||
|
||||
@@ -650,47 +652,146 @@ function newIntegrationForm() {
|
||||
|
||||
// --- dead man's switches ---------------------------------------------------
|
||||
|
||||
const SWITCH_STATUS = {
|
||||
healthy: { label: 'Healthy', hint: 'Heard from within its timeout.' },
|
||||
dead: { label: 'Dead', hint: 'Silent for longer than its timeout.' },
|
||||
dormant: { label: 'Dormant', hint: 'Nothing has matched yet, so there is nothing to lose.' },
|
||||
};
|
||||
|
||||
function switchBadge(status) {
|
||||
const s = SWITCH_STATUS[status] || SWITCH_STATUS.dormant;
|
||||
const el = badge(s.label, `st-${status}`);
|
||||
el.title = s.hint;
|
||||
return el;
|
||||
}
|
||||
|
||||
const timeCell = (iso) => iso
|
||||
? h('span', { title: when(iso), text: ago(iso) })
|
||||
: h('span', { class: 'muted', text: 'never' });
|
||||
|
||||
// When it last opened an incident. An incident that is still open is a link,
|
||||
// because that is the thing somebody looking at a red row wants next.
|
||||
const triggeredCell = (iso, incidentID) => {
|
||||
if (!iso) return h('span', { class: 'muted', text: 'never' });
|
||||
return incidentID
|
||||
? h('a', { href: `/incidents/${incidentID}`, title: when(iso) }, `#${incidentID} · ${ago(iso)}`)
|
||||
: h('span', { title: when(iso), text: ago(iso) });
|
||||
};
|
||||
|
||||
function switchRows(sw) {
|
||||
const main = h('tr', {},
|
||||
h('td', {}, switchBadge(sw.status)),
|
||||
h('td', {},
|
||||
h('strong', { text: sw.name }),
|
||||
sw.name !== sw.matcher && h('div', { class: 'muted small' }, h('code', { text: sw.matcher }))),
|
||||
h('td', { class: 'muted small' }, timeCell(sw.last_heartbeat_at)),
|
||||
h('td', { class: 'muted small' }, triggeredCell(sw.last_triggered_at, sw.open_incident_id)),
|
||||
h('td', { class: 'muted small', text: duration(sw.timeout_seconds * 1000) }),
|
||||
h('td', {}, isOwner() && h('button', {
|
||||
class: 'btn-sm danger', type: 'button', text: 'Remove',
|
||||
onclick: async () => {
|
||||
if (!(await confirm({
|
||||
title: `Remove ${sw.name}?`,
|
||||
text: 'It stops being watched. An incident it already opened stays open until it is resolved.',
|
||||
confirmLabel: 'Remove',
|
||||
danger: true,
|
||||
}))) return;
|
||||
act(() => api.deleteDeadmanSwitch(teamID, sw.id));
|
||||
},
|
||||
})),
|
||||
);
|
||||
|
||||
// One heartbeat is the switch's own times; several are worth telling apart,
|
||||
// since a live cluster must not hide a dead one.
|
||||
const sources = sw.sources.length > 1
|
||||
? sw.sources.map((src) => h('tr', { class: 'source-row' },
|
||||
h('td', {}, switchBadge(src.status)),
|
||||
h('td', { class: 'source-labels' },
|
||||
...Object.entries(src.labels || {})
|
||||
.filter(([k]) => k !== 'alertname')
|
||||
.map(([k, v]) => labelChip(k, v)),
|
||||
!Object.keys(src.labels || {}).some((k) => k !== 'alertname')
|
||||
&& h('code', { class: 'small', text: src.fingerprint })),
|
||||
h('td', { class: 'muted small' }, timeCell(src.last_heartbeat_at)),
|
||||
h('td', { class: 'muted small' }, triggeredCell(src.last_triggered_at, src.incident_id)),
|
||||
h('td'), h('td')))
|
||||
: [];
|
||||
return [main, ...sources];
|
||||
}
|
||||
|
||||
function deadmanCard() {
|
||||
const d = data.deadman || {};
|
||||
const matchers = h('input', {
|
||||
type: 'text', value: d.matchers || '', placeholder: 'alertname=Watchdog',
|
||||
class: 'wide',
|
||||
});
|
||||
const timeout = h('input', {
|
||||
type: 'number', min: '0', class: 'setting-value',
|
||||
value: String(Math.round((d.timeout_seconds || 0) / 60)),
|
||||
});
|
||||
const severity = h('select', {},
|
||||
...['critical', 'error', 'warning', 'info'].map((s) =>
|
||||
h('option', { value: s, text: s, selected: (d.severity || 'critical') === s })));
|
||||
|
||||
const form = h('form', { class: 'stacked-form' },
|
||||
h('label', {}, 'Heartbeat alerts ', matchers),
|
||||
h('label', {}, 'Declare dead after ', timeout, ' minutes of silence'),
|
||||
h('label', {}, 'Open the incident at severity ', severity),
|
||||
h('button', { class: 'btn', type: 'submit', text: 'Save switches' }));
|
||||
|
||||
form.addEventListener('submit', (e) => {
|
||||
e.preventDefault();
|
||||
act(() => api.setDeadman(teamID, {
|
||||
matchers: matchers.value.trim(),
|
||||
timeout_seconds: Number(timeout.value) * 60,
|
||||
severity: severity.value,
|
||||
}));
|
||||
});
|
||||
const switches = data.deadman || [];
|
||||
|
||||
return h('div', { class: 'card' },
|
||||
h('h2', { text: 'Dead man’s switches' }),
|
||||
h('div', { class: 'card-head' },
|
||||
h('h2', { text: 'Dead man’s switches' }),
|
||||
isOwner() && h('button', {
|
||||
class: 'btn', type: 'button', text: 'New switch', onclick: openNewSwitch,
|
||||
})),
|
||||
h('p', { class: 'muted small' },
|
||||
'Alerts whose ABSENCE is the signal. Receiving one opens nothing; going ',
|
||||
'quiet for longer than the timeout opens an incident. ',
|
||||
h('code', { text: 'alertname=Watchdog,cluster=prod; alertname=EdgeHeartbeat' }),
|
||||
' — semicolons separate switches, commas separate conditions, and every ',
|
||||
'switch must name an alertname. Leave empty to watch nothing.'),
|
||||
isOwner() ? form : h('p', { class: 'muted', text: d.matchers || 'Nothing watched.' }),
|
||||
'quiet for longer than the switch’s timeout opens an incident.'),
|
||||
switches.length
|
||||
? h('div', { class: 'table-scroll' },
|
||||
h('table', { class: 'admin-table switch-table' },
|
||||
h('thead', {}, h('tr', {},
|
||||
h('th', { text: 'Status' }), h('th', { text: 'Switch' }),
|
||||
h('th', { text: 'Last heartbeat' }), h('th', { text: 'Last triggered' }),
|
||||
h('th', { text: 'Silent after' }), h('th'))),
|
||||
h('tbody', {}, switches.flatMap(switchRows))))
|
||||
: h('p', { class: 'muted', text: 'Nothing watched.' }),
|
||||
);
|
||||
}
|
||||
|
||||
// The form lives in the sheet, not on the page: most visits are to look at the
|
||||
// list, and a form that is always open is the page this replaced.
|
||||
function openNewSwitch() {
|
||||
const name = h('input', { type: 'text', placeholder: 'Prod Watchdog', autofocus: true });
|
||||
const matcher = h('input', {
|
||||
type: 'text', placeholder: 'alertname=Watchdog,cluster=prod', class: 'wide', required: true,
|
||||
});
|
||||
const timeout = h('input', {
|
||||
type: 'number', min: '1', value: '15', class: 'setting-value', required: true,
|
||||
});
|
||||
const severity = h('select', {},
|
||||
...['critical', 'error', 'warning', 'info'].map((s) => h('option', { value: s, text: s })));
|
||||
const problem = h('p', { class: 'load-error', hidden: true });
|
||||
|
||||
const form = h('form', { class: 'stacked-form' },
|
||||
h('label', {}, 'Name (optional) ', name),
|
||||
h('label', {}, 'Heartbeat alert ', matcher),
|
||||
h('p', { class: 'muted small' },
|
||||
'Conditions are ', h('code', { text: 'label=value' }), ' separated by commas, and one ',
|
||||
'must be ', h('code', { text: 'alertname' }), '. Every distinct label set that ',
|
||||
'matches is watched on its own.'),
|
||||
h('label', {}, 'Declare dead after ', timeout, ' minutes of silence'),
|
||||
h('label', {}, 'Open the incident at severity ', severity),
|
||||
problem,
|
||||
h('div', { class: 'sheet-actions' },
|
||||
h('button', { class: 'btn', type: 'button', text: 'Cancel', onclick: () => closeSheet(false) }),
|
||||
h('button', { class: 'btn btn-primary', type: 'submit', text: 'Add switch' })));
|
||||
|
||||
form.addEventListener('submit', async (e) => {
|
||||
e.preventDefault();
|
||||
try {
|
||||
await api.createDeadmanSwitch(teamID, {
|
||||
name: name.value.trim(),
|
||||
matcher: matcher.value.trim(),
|
||||
timeout_seconds: Math.round(Number(timeout.value) * 60),
|
||||
severity: severity.value,
|
||||
});
|
||||
} catch (err) {
|
||||
problem.textContent = err.message;
|
||||
problem.hidden = false;
|
||||
return;
|
||||
}
|
||||
closeSheet(true);
|
||||
refresh();
|
||||
});
|
||||
|
||||
openSheet(() => [h('h2', { class: 'sheet-title', text: 'New switch' }), form]);
|
||||
}
|
||||
|
||||
// --- members ---------------------------------------------------------------
|
||||
|
||||
function membersCard() {
|
||||
|
||||
Reference in New Issue
Block a user