Let an operator authenticate with a seeded key, and reset the schema
CI / chart (pull_request) Successful in 2s
CI / security (pull_request) Failing after 19s
CI / test (pull_request) Successful in 5m34s

- TERDUT_OPERATOR_KEY creates or re-keys the instance-scoped service account
  "terdut-operator" at every start, so terdut-operator needs no bootstrap
  handshake. An instance-scoped account now acts as owner of every team's
  configuration, but is not a member of any team.
- POST /api/teams takes an external_id (instance service accounts only) and
  is idempotent on it, so automation finds its own team again after a crash
  instead of adopting by display name. GET /api/teams?name= is removed.
- Integration and dead man's switch names are unique per team (409). The
  escalation PUT accepts usernames and resolves them itself.
- The 18 migrations are squashed into 001_schema.sql, with no Default team.
  TERDUT_DEADMAN_* and the env seeding of switches are removed: teams carry
  their own. Existing development databases must be recreated.

Security and robustness:
- GET /api/users no longer returns other people's email or ntfy topic to
  non-admins.
- The access log records the route pattern, so integration keys and ack
  tokens in the path are not written to the log. Server errors are logged.
- Rate limits take the client address TERDUT_TRUSTED_PROXIES hops from the
  right of X-Forwarded-For instead of trusting the first, forgeable entry.
- /api/bootstrap runs in a transaction under an advisory lock, so two
  concurrent calls cannot both create an administrator.
- API key last_used_at is written at most every five minutes.

Cleanup: remove GET /api/incidents/{id}/alerts, unused exports, SQLite
remnants in comments and config.

Claude-Session: https://claude.ai/code/session_016mBLURvJoMuUEr9cB2RpUN
This commit is contained in:
Niklas Ye
2026-10-09 14:56:13 +02:00
parent 44b2eb2cc3
commit 9029d48584
46 changed files with 1445 additions and 655 deletions
-112
View File
@@ -65,28 +65,6 @@ func (m DeadmanMatcher) matches(labels map[string]string) bool {
return true
}
// DeadmanConfig is the server-wide default a team's switches are seeded from:
// the environment's matchers, timeout and severity. Switches themselves are rows
// of a team's own — see DeadmanSwitch — and this is only how a fresh install
// starts out.
type DeadmanConfig struct {
Matchers []DeadmanMatcher
// Timeout is how long a matched alert may go without a refreshing webhook
// before it is declared dead. It must be shorter than Alertmanager's
// repeat_interval for the heartbeat's route, which is what refreshes it.
// Zero disables dead man's switch handling entirely.
Timeout time.Duration
// Severity is the severity every dead man's switch incident opens at. These
// incidents have no member alerts to derive one from, and the heartbeat's
// own severity label is meaningless — Watchdog ships as "none".
Severity string
}
// enabled reports whether there is anything to watch.
func (c DeadmanConfig) enabled() bool { return c.Timeout > 0 && len(c.Matchers) > 0 }
// DeadmanSwitch inverts the handling of the alerts it matches: receiving one
// opens nothing, and the absence of one opens an incident.
//
@@ -160,46 +138,6 @@ func parseDeadmanMatcher(entry string) (DeadmanMatcher, error) {
return m, nil
}
// ParseDeadmanConfig reads the matcher list from its configured form:
// ";" separates matchers, and each is parsed as parseDeadmanMatcher does.
//
// A malformed or alertname-less entry is dropped rather than fatal, following
// config.duration's rule that one bad tuning knob should not take the server
// down. Silence would be worse here than elsewhere, though — a typo that
// disarms the switch is exactly the failure this feature exists to catch — so
// the matchers that survived are logged.
func ParseDeadmanConfig(matchers string, timeout time.Duration, severity string) DeadmanConfig {
cfg := DeadmanConfig{Timeout: timeout, Severity: severity}
for _, entry := range strings.Split(matchers, ";") {
entry = strings.TrimSpace(entry)
if entry == "" {
continue
}
m, err := parseDeadmanMatcher(entry)
if err != nil {
log.Printf("deadman: ignoring matcher %q: %v", entry, err)
continue
}
cfg.Matchers = append(cfg.Matchers, m)
}
switch {
case timeout <= 0:
log.Print("deadman: disabled (timeout is zero)")
case len(cfg.Matchers) == 0:
log.Print("deadman: disabled (no usable matchers)")
default:
rendered := make([]string, 0, len(cfg.Matchers))
for _, m := range cfg.Matchers {
rendered = append(rendered, m.String())
}
log.Printf("deadman: default for new teams: %s, timeout %s, severity %s",
strings.Join(rendered, "; "), timeout, severity)
}
return cfg
}
// deadmanAlert is one heartbeat: the alert row carrying its last sighting, and
// the switch that claimed it.
type deadmanAlert struct {
@@ -490,56 +428,6 @@ func deadmanSets(ctx context.Context, db *sql.DB) (map[int64]deadmanSet, error)
return scanDeadmanSwitches(rows)
}
// deadmanSeededKey is the settings row that records the environment defaults
// were handed out. Without it, a team that deleted its last switch would get
// the default back on the next restart.
const deadmanSeededKey = "deadman_seeded"
// SeedDeadmanConfigs gives every team the server's environment defaults as
// switches, exactly once per install, so a fresh install watches Watchdog
// without anybody setting it up.
//
// Once seeded it never runs again: a team's switches are its own, and a redeploy
// must not quietly put the environment's value back over an owner's edit or
// deletion. Installs that upgraded from per-team configuration were already
// seeded, which migration 009 records.
//
// A team created after that gets none and watches nothing until its owner says
// otherwise. That is deliberate: inheriting an install-wide heartbeat would page
// a new team about a source it has never heard of, and a switch nobody chose is
// the kind that gets muted rather than fixed.
func SeedDeadmanConfigs(ctx context.Context, db *sql.DB, cfg DeadmanConfig) error {
if !cfg.enabled() {
return nil
}
tx, err := db.BeginTx(ctx, nil)
if err != nil {
return err
}
defer tx.Rollback() //nolint:errcheck
res, err := tx.ExecContext(ctx,
"INSERT INTO settings (key, value) VALUES ($1, '1') ON CONFLICT (key) DO NOTHING",
deadmanSeededKey)
if err != nil {
return err
}
if n, _ := res.RowsAffected(); n == 0 {
return nil
}
for _, m := range cfg.Matchers {
if _, err := tx.ExecContext(ctx, `
INSERT INTO deadman_switches (team_id, name, matcher, timeout_seconds, severity)
SELECT id, $1, $1, $2, $3 FROM teams`,
m.config(), int64(cfg.Timeout.Seconds()), cfg.Severity); err != nil {
return err
}
}
return tx.Commit()
}
// ---------------------------------------------------------------------------
// Status
// ---------------------------------------------------------------------------