Niklas Ye
74359c72ab
The rest of #4. Two halves that belong together because they are the same sentence from opposite ends: a team decides which of its alerts are heartbeats, and the UI has to be able to say which team it is talking about. Switches were three environment variables, which made them one setting for the whole install. That was the last piece of the alerting path a team could not control: it could take its own alerts on its own key and still not say which of them were heartbeats, or how long a silence had to last. They are a row per team now, edited by an owner through PUT /api/teams/{teamID}/deadman, and the sweeper runs each team against its own matchers, timeout and severity. The environment variables become the starting point rather than the setting. Every team without a configuration is seeded from them at startup, so an upgrade keeps watching exactly what it was watching, and SeedDeadmanConfigs never overwrites -- a redeploy must not put the environment's value back over an owner's edit. A team created later watches nothing until somebody says otherwise: inheriting an install-wide heartbeat would page a new team about a source it has never heard of, and a switch nobody chose is the kind that gets muted rather than fixed. A matcher string with no alertname in it is refused at the door instead of stored. Storing it would produce a switch that watches nothing silently, which is the exact failure the feature exists to prevent. NewRouter and Sweep lose their DeadmanConfig parameter -- there is no longer one answer to hand them. The type stays, because parsing a matcher string is still parsing a matcher string. The UI side: rows in the queue carry a team badge, the filter row gains a team chip per team, and "on call now" shows one card per team. All three appear only when the viewer is in more than one team -- otherwise they are the same word repeated down a list, which is noise rather than information, and the single-team install reads exactly as it did before teams existed. Verified against a live two-team server as well as in tests: the combined queue labelled by team, the team_id filter, a heartbeat that is a heartbeat in one team and an ordinary alert in another, and a new team's switches starting empty while the upgraded team keeps the environment's. Claude-Session: https://claude.ai/code/session_01RHPj4ggeFdEjKKfm4SHbD7
40 lines
2.0 KiB
SQL
40 lines
2.0 KiB
SQL
-- Dead man's switches become a team's own configuration.
|
|
--
|
|
-- They were three environment variables — TERDUT_DEADMAN_MATCHERS, _TIMEOUT and
|
|
-- _SEVERITY — which made them one setting for the whole install. That was the
|
|
-- last piece of the alerting path a team could not control: a team could take
|
|
-- its own alerts on its own key and still not say which of them were
|
|
-- heartbeats, or how long a silence had to last before somebody was paged.
|
|
--
|
|
-- One row per team rather than one row per switch. The unit of monitoring is
|
|
-- still the fingerprint, as it always was — two clusters sending the same
|
|
-- heartbeat alertname are two independent switches — and the matcher string
|
|
-- keeps the format the environment variable used, so a value can be moved from
|
|
-- one to the other unchanged.
|
|
--
|
|
-- No rows are seeded here: a migration cannot read the environment. The server
|
|
-- inserts a row per team at startup from its own configuration, and the same
|
|
-- values therefore carry forward into the first team's row without anybody
|
|
-- retyping them. See seedDeadmanConfigs.
|
|
CREATE TABLE deadman_configs (
|
|
team_id BIGINT PRIMARY KEY REFERENCES teams(id) ON DELETE CASCADE,
|
|
|
|
-- ";" separates matchers, "," the label conditions within one, "=" is exact
|
|
-- equality: `alertname=Watchdog,cluster=prod; alertname=EdgeHeartbeat`.
|
|
-- Every matcher must name an alertname. Empty watches nothing.
|
|
matchers TEXT NOT NULL DEFAULT '',
|
|
|
|
-- Seconds rather than a Go duration string: the column is compared and
|
|
-- arithmetic is done on it, and a value that has to be parsed before it can
|
|
-- be believed is a value that can be stored unparseable. Zero disables the
|
|
-- team's switches entirely.
|
|
timeout_seconds BIGINT NOT NULL DEFAULT 0,
|
|
|
|
-- The severity these incidents open at. They have no member alerts to
|
|
-- derive one from, and a heartbeat's own severity label is meaningless —
|
|
-- Watchdog ships as "none".
|
|
severity TEXT NOT NULL DEFAULT 'critical',
|
|
|
|
updated_at BIGINT NOT NULL DEFAULT FLOOR(EXTRACT(EPOCH FROM now()))::bigint
|
|
);
|