828cf87656e131ae68be25daf4a97c577a3a7292
3 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
|
|
d728af53b1 |
Put a team's own settings in the web UI
Closes #17. Everything a team owner configures was API-only: escalation, integrations, dead man's switches, membership, and the rota -- which the on-call view still described as the TUI's job, and the TUI has been broken against this server since teams landed. Setting up the feature this whole line of work exists for meant using curl. A Team tab now holds all of it, one team at a time, with a picker for somebody in more than one. An owner edits; a member sees the same page without the controls, because the server refuses their writes anyway -- hiding a button is a courtesy to the reader, not the thing enforcing anything. The escalation editor holds a draft and sends the whole ladder, because the API replaces it wholesale: the levels are an order, and patching one rung leaves the numbering of the others undecided. Adding a level defaults to five minutes and the rota, which is the shape almost every ladder starts as. An integration key is returned exactly once, so creating one opens a panel that says so, shows the URL large with a copy button, and renders the Alertmanager receiver snippet with the URL already in it -- the next thing anybody does with that key is paste it into a config. The panel stays until it is dismissed rather than disappearing on the next re-render. The incident view gains where an incident is on the ladder and when the next page is due, which is the question somebody looking at an unacknowledged incident actually has. The API carries it: the incident payload now includes escalation_level and escalation_due_at, the latter computed in the incident SELECT by joining the level's timeout, so a list costs no extra queries. Verified against a live server by making every call the page makes, including the writes: the six reads the Team tab issues, a two-level ladder saved and read back, an integration created and its key returned once, three days of rota assigned, switches set, and an incident showing level 1 with a due time five minutes out. Claude-Session: https://claude.ai/code/session_01RHPj4ggeFdEjKKfm4SHbD7 |
||
|
|
a4fbd60441 |
Scope everything to a team, and route alerts by integration key
The core of #4, and what #1 is for: terdut stops being one shared space. A team owns its incidents, alerts, schedule and integrations; a user sees exactly the teams they are in. Everything that existed moves into one Default team and every existing user becomes an owner of it, so the upgrade is a no-op for the people using it. Ingestion is the load-bearing half. An alert arrives on a team's integration key, and the key is both the credential and the routing: it says that the sender may post, and which team the alerts belong to. That also closes the unauthenticated webhook -- the old path stays for one release, deprecated and routed to the oldest team, so an upgrade does not stop delivering while somebody edits the Alertmanager config. Scoping is enforced in as few places as possible, because the failure mode is silent. serveAs loads the caller's memberships once; list queries carry `team_id = ANY(...)`; and every incident route goes through incidentIDParam, which now parses the id AND checks the team in the same call, so a new handler cannot remember the first half and forget the second. Anything in another team is 404, never 403: whether an incident exists is that team's business. Two bugs this found, both of which would have been silent: * upsertAlerts decided "is this a new occurrence" by looking up the fingerprint alone. Across teams that made team B's first alert look like a re-send of team A's, so it opened no incident at all. The lookups are keyed on (team_id, fingerprint) now, as the index is. * Every uniqueness rule was written for one tenant. Two teams watching two clusters legitimately see the same fingerprint, the same groupKey, and want somebody on call on the same day; all three constraints move to include team_id. Roles inside a team are separate from the system administrator flag: an owner configures the team, a member works its incidents, and an admin is NOT implicitly in every team -- administration is about accounts, not about reading other people's incidents. An admin can still repair a team whose owner has left, which is why requireTeamOwner lets them through. A shift can only be given to somebody in the team. Paging a person who cannot open the incident is worse than paging nobody. The UI is updated only as far as keeping it working: it loads the viewer's teams with the session and uses the first one, since nobody has a second yet. "On call now" shows every team the viewer is in, named only when there is more than one, so the common case reads exactly as before. The team switcher, badges and per-team settings pages are the next step. Breaking for API clients: the schedule endpoints moved under the team, and /api/schedule/current returns an array rather than an object or a 404. terdut-tui will need a version for that. Per-team dead-man configuration is deliberately not here. A heartbeat's incident already opens in the team whose key received it, which is the part that matters for isolation; moving the matchers out of env into per-team rows is a change to how deadman.go is configured rather than to who sees what. Claude-Session: https://claude.ai/code/session_01RHPj4ggeFdEjKKfm4SHbD7 |
||
|
|
279ef6cf8b |
Turn incoming alerts into incidents
Release / build (amd64, linux) (push) Failing after 11s
Release / build (amd64, darwin) (push) Failing after 12s
Release / build (arm64, darwin) (push) Failing after 11s
Release / build (arm64, linux) (push) Failing after 11s
Release / release (push) Has been skipped
Release / chart (push) Failing after 13s
Release / docker (push) Failing after 19s
The alerts row was both Alertmanager's record and the human work queue, and
the two have different owners. The webhook upsert rewrites that row on every
notification; acknowledgement, comments and archiving were columns on it that
the upsert happened not to touch. So an alert that resolved and re-fired days
later still read as acknowledged by whoever acked the first occurrence — the
ack outlived the thing it referred to. Nothing recorded transitions either:
rows are mutated in place, so there was no timeline and no way to compute how
long anything took.
Alerts are now read-only signal records with two states, and incidents are
the work item: triggered, acknowledged or resolved, with an assignee, a
snooze, notes and an append-only timeline. Many alerts map to one incident,
and a new occurrence opens a new incident, which is what makes a stale ack
impossible rather than merely unlikely.
Correlation uses Alertmanager's own groupKey. It already grouped the alerts
according to the group_by routing tree the operator configured and sends the
result on every webhook, where it was being discarded; adopting it means
changing group_by in alertmanager.yml changes correlation here, with no
second grouping scheme to configure and keep in sync.
An incident opens only when an alert transitions into firing — an unseen
fingerprint, a newer startsAt, or a resolved alert starting again. The
unchanged notifications Alertmanager re-sends every repeat_interval are none
of those. That rule is what lets manual resolution be terminal: without it,
closing an incident by hand would be undone by the next re-send of an alert
that never stopped firing, and the button would be a lie. Snooze covers the
"not now" case instead. Incidents otherwise resolve by cascade, once every
alert under them has stopped firing, whether by webhook or by expiry.
New incidents are assigned to whoever holds today's schedule entry. The
schedule table has existed since the first release with nothing reading it.
Also here, following from the split:
- Incident severity is a high-water mark over its alerts, never lowered.
An incident that hit critical was a critical incident, and downgrading a
live one would demote it in the queue while the work is still open.
- /api/stats/incidents reports MTTA and MTTR, null rather than zero until
there is something to average. Neither was computable before.
- Alert archiving becomes sweeper-only housekeeping; the archive people
interact with is the incident's.
Breaking: the alert acknowledge, archive and comment endpoints are gone, and
the alert object drops the acknowledgement fields and gains incident_id. The
README maps each removed endpoint to its replacement. Migration 008 backfills
an incident per existing alert, archived ones included so no comment is
orphaned, carrying acknowledgements across and turning comments into timeline
notes.
Both documented alert contracts are untouched: received_at still advances on
every accepted payload, re-sends included, and resolution_source still says
how much to trust ends_at. The upsert is byte-for-byte what it was, now
running inside the ingest transaction.
|