Scope everything to a team, and route alerts by integration key
CI / chart (pull_request) Successful in 1s
CI / security (pull_request) Successful in 13s
CI / test (pull_request) Successful in 1m49s

The core of #4, and what #1 is for: terdut stops being one shared space.
A team owns its incidents, alerts, schedule and integrations; a user sees
exactly the teams they are in. Everything that existed moves into one
Default team and every existing user becomes an owner of it, so the
upgrade is a no-op for the people using it.

Ingestion is the load-bearing half. An alert arrives on a team's
integration key, and the key is both the credential and the routing: it
says that the sender may post, and which team the alerts belong to. That
also closes the unauthenticated webhook -- the old path stays for one
release, deprecated and routed to the oldest team, so an upgrade does not
stop delivering while somebody edits the Alertmanager config.

Scoping is enforced in as few places as possible, because the failure
mode is silent. serveAs loads the caller's memberships once; list queries
carry `team_id = ANY(...)`; and every incident route goes through
incidentIDParam, which now parses the id AND checks the team in the same
call, so a new handler cannot remember the first half and forget the
second. Anything in another team is 404, never 403: whether an incident
exists is that team's business.

Two bugs this found, both of which would have been silent:

  * upsertAlerts decided "is this a new occurrence" by looking up the
    fingerprint alone. Across teams that made team B's first alert look
    like a re-send of team A's, so it opened no incident at all. The
    lookups are keyed on (team_id, fingerprint) now, as the index is.

  * Every uniqueness rule was written for one tenant. Two teams watching
    two clusters legitimately see the same fingerprint, the same
    groupKey, and want somebody on call on the same day; all three
    constraints move to include team_id.

Roles inside a team are separate from the system administrator flag: an
owner configures the team, a member works its incidents, and an admin is
NOT implicitly in every team -- administration is about accounts, not
about reading other people's incidents. An admin can still repair a team
whose owner has left, which is why requireTeamOwner lets them through.

A shift can only be given to somebody in the team. Paging a person who
cannot open the incident is worse than paging nobody.

The UI is updated only as far as keeping it working: it loads the
viewer's teams with the session and uses the first one, since nobody has
a second yet. "On call now" shows every team the viewer is in, named only
when there is more than one, so the common case reads exactly as before.
The team switcher, badges and per-team settings pages are the next step.

Breaking for API clients: the schedule endpoints moved under the team,
and /api/schedule/current returns an array rather than an object or a
404. terdut-tui will need a version for that.

Per-team dead-man configuration is deliberately not here. A heartbeat's
incident already opens in the team whose key received it, which is the
part that matters for isolation; moving the matchers out of env into
per-team rows is a change to how deadman.go is configured rather than to
who sees what.

Claude-Session: https://claude.ai/code/session_01RHPj4ggeFdEjKKfm4SHbD7
This commit is contained in:
Niklas Ye
2026-09-20 13:36:24 +02:00
parent 1377d9005b
commit a4fbd60441
27 changed files with 1549 additions and 150 deletions
+77 -31
View File
@@ -4,9 +4,12 @@ import (
"context"
"database/sql"
"encoding/json"
"errors"
"log"
"net/http"
"time"
"github.com/go-chi/chi/v5"
)
// Values for alerts.resolution_source, recording why an alert left the firing
@@ -70,36 +73,76 @@ type ingested struct {
deadman bool
}
func handleAlertmanagerWebhook(db *sql.DB, notify NotifyConfig, deadman DeadmanConfig) http.HandlerFunc {
// handleIntegrationWebhook receives alerts on a team's own integration key.
// The key in the path is both the credential and the routing: it says who may
// post, and which team the alerts belong to.
func handleIntegrationWebhook(db *sql.DB, notify NotifyConfig, deadman DeadmanConfig) http.HandlerFunc {
return func(w http.ResponseWriter, r *http.Request) {
var payload amPayload
if err := decodeJSON(r, &payload); err != nil {
respond(w, http.StatusBadRequest, errResp("invalid payload"))
teamID, err := teamIDForKey(r.Context(), db, chi.URLParam(r, "key"))
if err != nil {
if errors.Is(err, errUnknownIntegration) {
// 401 and not 404: the path is real, the key is not, and a
// sender misconfigured this way should say so in its own logs
// rather than believe it is delivering.
respond(w, http.StatusUnauthorized, errResp("unknown integration key"))
return
}
respond(w, http.StatusInternalServerError, errResp("internal error"))
return
}
// Alertmanager retries anything that is not 2xx, and a retry of a payload
// we failed to store is more useful than an error it cannot act on — so
// failures are logged, not surfaced.
if err := ingest(r.Context(), db, notify, deadman, payload); err != nil {
log.Printf("webhook ingest (group %q): %v", payload.GroupKey, err)
}
w.WriteHeader(http.StatusOK)
receiveWebhook(w, r, db, notify, deadman, teamID)
}
}
// handleLegacyWebhook is the pre-teams unauthenticated endpoint, kept for one
// release so an upgrade does not silently stop delivering while somebody edits
// the Alertmanager config. It routes to the oldest team, which on an upgraded
// install is the Default team everything was moved into.
//
// It is deprecated and unauthenticated — anything that can reach the port can
// open an incident. Move senders to an integration key and this goes away.
func handleLegacyWebhook(db *sql.DB, notify NotifyConfig, deadman DeadmanConfig) http.HandlerFunc {
return func(w http.ResponseWriter, r *http.Request) {
teamID, err := defaultTeamID(r.Context(), db)
if err != nil {
log.Printf("legacy webhook: no team to route to: %v", err)
w.WriteHeader(http.StatusOK)
return
}
log.Printf("legacy webhook: unauthenticated payload routed to team %d; "+
"move this sender to an integration key", teamID)
receiveWebhook(w, r, db, notify, deadman, teamID)
}
}
func receiveWebhook(w http.ResponseWriter, r *http.Request, db *sql.DB, notify NotifyConfig, deadman DeadmanConfig, teamID int64) {
var payload amPayload
if err := decodeJSON(r, &payload); err != nil {
respond(w, http.StatusBadRequest, errResp("invalid payload"))
return
}
// Alertmanager retries anything that is not 2xx, and a retry of a payload
// we failed to store is more useful than an error it cannot act on — so
// failures are logged, not surfaced.
if err := ingest(r.Context(), db, notify, deadman, teamID, payload); err != nil {
log.Printf("webhook ingest (team %d, group %q): %v", teamID, payload.GroupKey, err)
}
w.WriteHeader(http.StatusOK)
}
// ingest stores a payload's alerts and reconciles the incident for its group.
// The whole payload is one transaction: an incident that opened but whose alerts
// failed to link would be a work item nobody could act on.
func ingest(ctx context.Context, db *sql.DB, notify NotifyConfig, deadman DeadmanConfig, payload amPayload) error {
func ingest(ctx context.Context, db *sql.DB, notify NotifyConfig, deadman DeadmanConfig, teamID int64, payload amPayload) error {
tx, err := db.BeginTx(ctx, nil)
if err != nil {
return err
}
defer tx.Rollback() //nolint:errcheck
accepted, err := upsertAlerts(ctx, tx, deadman, payload.Alerts)
accepted, err := upsertAlerts(ctx, tx, deadman, teamID, payload.Alerts)
if err != nil {
return err
}
@@ -108,7 +151,7 @@ func ingest(ctx context.Context, db *sql.DB, notify NotifyConfig, deadman Deadma
// resolution cascade are recomputed once per incident at the end.
touched := map[int64]bool{}
incidentID, err := incidentForGroup(ctx, tx, notify, payload, accepted)
incidentID, err := incidentForGroup(ctx, tx, notify, teamID, payload, accepted)
if err != nil {
return err
}
@@ -156,7 +199,7 @@ func ingest(ctx context.Context, db *sql.DB, notify NotifyConfig, deadman Deadma
// upsertAlerts stores each alert of a payload and reports what changed. Payloads
// the ordering guard rejected are left out entirely.
func upsertAlerts(ctx context.Context, tx *sql.Tx, deadman DeadmanConfig, alerts []amAlert) ([]ingested, error) {
func upsertAlerts(ctx context.Context, tx *sql.Tx, deadman DeadmanConfig, teamID int64, alerts []amAlert) ([]ingested, error) {
now := time.Now().Unix()
accepted := make([]ingested, 0, len(alerts))
@@ -171,7 +214,8 @@ func upsertAlerts(ctx context.Context, tx *sql.Tx, deadman DeadmanConfig, alerts
var prevStartsAt int64
existed := true
switch err := tx.QueryRowContext(ctx,
"SELECT status, starts_at FROM alerts WHERE fingerprint = $1", a.Fingerprint,
"SELECT status, starts_at FROM alerts WHERE team_id = $1 AND fingerprint = $2",
teamID, a.Fingerprint,
).Scan(&prevStatus, &prevStartsAt); {
case err == sql.ErrNoRows:
existed = false
@@ -210,10 +254,10 @@ func upsertAlerts(ctx context.Context, tx *sql.Tx, deadman DeadmanConfig, alerts
// undone by a stale retry.
if _, err := tx.ExecContext(ctx, `
INSERT INTO alerts
(fingerprint, name, status, labels, annotations, starts_at, ends_at,
(team_id, fingerprint, name, status, labels, annotations, starts_at, ends_at,
generator_url, received_at, resolution_source)
VALUES ($1, $2, $3, $4::jsonb, $5::jsonb, $6, $7, $8, $9, $10)
ON CONFLICT (fingerprint) DO UPDATE SET
VALUES ($1, $2, $3, $4, $5::jsonb, $6::jsonb, $7, $8, $9, $10, $11)
ON CONFLICT (team_id, fingerprint) DO UPDATE SET
status = excluded.status,
labels = excluded.labels,
annotations = excluded.annotations,
@@ -233,7 +277,7 @@ func upsertAlerts(ctx context.Context, tx *sql.Tx, deadman DeadmanConfig, alerts
OR (excluded.starts_at = alerts.starts_at
AND (alerts.resolution_source = '`+resolutionDeadman+`'
OR NOT (alerts.status = 'resolved' AND excluded.status = 'firing')))`,
a.Fingerprint, name, a.Status,
teamID, a.Fingerprint, name, a.Status,
string(labelsJSON), string(annotationsJSON),
a.StartsAt.Unix(), endsAtUnix,
a.GeneratorURL, now, resolutionSource,
@@ -245,7 +289,8 @@ func upsertAlerts(ctx context.Context, tx *sql.Tx, deadman DeadmanConfig, alerts
var curStatus string
var curStartsAt int64
if err := tx.QueryRowContext(ctx,
"SELECT id, status, starts_at FROM alerts WHERE fingerprint = $1", a.Fingerprint,
"SELECT id, status, starts_at FROM alerts WHERE team_id = $1 AND fingerprint = $2",
teamID, a.Fingerprint,
).Scan(&id, &curStatus, &curStartsAt); err != nil {
return nil, err
}
@@ -284,7 +329,7 @@ func upsertAlerts(ctx context.Context, tx *sql.Tx, deadman DeadmanConfig, alerts
// Heartbeats do not count as anything here. A group of nothing but dead man's
// switch alerts opens no incident at all, and a mixed group gets an incident for
// its real alerts only.
func incidentForGroup(ctx context.Context, tx *sql.Tx, notify NotifyConfig, payload amPayload, accepted []ingested) (int64, error) {
func incidentForGroup(ctx context.Context, tx *sql.Tx, notify NotifyConfig, teamID int64, payload amPayload, accepted []ingested) (int64, error) {
var firstName string
anyFiring, anyNew := false, false
for _, a := range accepted {
@@ -315,7 +360,8 @@ func incidentForGroup(ctx context.Context, tx *sql.Tx, notify NotifyConfig, payl
var id int64
switch err := tx.QueryRowContext(ctx,
"SELECT id FROM incidents WHERE group_key = $1 AND resolved_at IS NULL", groupKey,
"SELECT id FROM incidents WHERE team_id = $1 AND group_key = $2 AND resolved_at IS NULL",
teamID, groupKey,
).Scan(&id); {
case err == nil:
return id, nil
@@ -326,7 +372,7 @@ func incidentForGroup(ctx context.Context, tx *sql.Tx, notify NotifyConfig, payl
if !anyNew {
return 0, nil
}
return openIncident(ctx, tx, notify, groupKey,
return openIncident(ctx, tx, notify, teamID, groupKey,
incidentTitle(payload.GroupLabels, firstName), payload.GroupLabels, nil)
}
@@ -338,8 +384,8 @@ func incidentForGroup(ctx context.Context, tx *sql.Tx, notify NotifyConfig, payl
// its own. Hence the querier rather than a *sql.Tx. A nil severity leaves the
// column for refreshSeverity to fill from the member alerts; the sweeper passes
// one because its incidents have no members to derive it from.
func openIncident(ctx context.Context, q querier, notify NotifyConfig, groupKey, title string, groupLabels map[string]string, severity *string) (int64, error) {
onCall, err := currentOnCall(ctx, q)
func openIncident(ctx context.Context, q querier, notify NotifyConfig, teamID int64, groupKey, title string, groupLabels map[string]string, severity *string) (int64, error) {
onCall, err := currentOnCall(ctx, q, teamID)
if err != nil {
return 0, err
}
@@ -351,10 +397,10 @@ func openIncident(ctx context.Context, q querier, notify NotifyConfig, groupKey,
var id int64
err = q.QueryRowContext(ctx, `
INSERT INTO incidents (group_key, title, group_labels, status, severity, triggered_at, assigned_to)
VALUES ($1, $2, $3::jsonb, 'triggered', $4, $5, $6)
INSERT INTO incidents (team_id, group_key, title, group_labels, status, severity, triggered_at, assigned_to)
VALUES ($1, $2, $3, $4::jsonb, 'triggered', $5, $6, $7)
RETURNING id`,
groupKey, title, string(labelsJSON), severity,
teamID, groupKey, title, string(labelsJSON), severity,
time.Now().Unix(), onCall).Scan(&id)
if err != nil {
return 0, err