-- Dead man's switches become a team's own configuration. -- -- They were three environment variables — TERDUT_DEADMAN_MATCHERS, _TIMEOUT and -- _SEVERITY — which made them one setting for the whole install. That was the -- last piece of the alerting path a team could not control: a team could take -- its own alerts on its own key and still not say which of them were -- heartbeats, or how long a silence had to last before somebody was paged. -- -- One row per team rather than one row per switch. The unit of monitoring is -- still the fingerprint, as it always was — two clusters sending the same -- heartbeat alertname are two independent switches — and the matcher string -- keeps the format the environment variable used, so a value can be moved from -- one to the other unchanged. -- -- No rows are seeded here: a migration cannot read the environment. The server -- inserts a row per team at startup from its own configuration, and the same -- values therefore carry forward into the first team's row without anybody -- retyping them. See seedDeadmanConfigs. CREATE TABLE deadman_configs ( team_id BIGINT PRIMARY KEY REFERENCES teams(id) ON DELETE CASCADE, -- ";" separates matchers, "," the label conditions within one, "=" is exact -- equality: `alertname=Watchdog,cluster=prod; alertname=EdgeHeartbeat`. -- Every matcher must name an alertname. Empty watches nothing. matchers TEXT NOT NULL DEFAULT '', -- Seconds rather than a Go duration string: the column is compared and -- arithmetic is done on it, and a value that has to be parsed before it can -- be believed is a value that can be stored unparseable. Zero disables the -- team's switches entirely. timeout_seconds BIGINT NOT NULL DEFAULT 0, -- The severity these incidents open at. They have no member alerts to -- derive one from, and a heartbeat's own severity label is meaningless — -- Watchdog ships as "none". severity TEXT NOT NULL DEFAULT 'critical', updated_at BIGINT NOT NULL DEFAULT FLOOR(EXTRACT(EPOCH FROM now()))::bigint );