7 Commits

Author	SHA1	Message	Date
Niklas Ye	a4fbd60441	Scope everything to a team, and route alerts by integration key ... The core of #4, and what #1 is for: terdut stops being one shared space. A team owns its incidents, alerts, schedule and integrations; a user sees exactly the teams they are in. Everything that existed moves into one Default team and every existing user becomes an owner of it, so the upgrade is a no-op for the people using it. Ingestion is the load-bearing half. An alert arrives on a team's integration key, and the key is both the credential and the routing: it says that the sender may post, and which team the alerts belong to. That also closes the unauthenticated webhook -- the old path stays for one release, deprecated and routed to the oldest team, so an upgrade does not stop delivering while somebody edits the Alertmanager config. Scoping is enforced in as few places as possible, because the failure mode is silent. serveAs loads the caller's memberships once; list queries carry `team_id = ANY(...)`; and every incident route goes through incidentIDParam, which now parses the id AND checks the team in the same call, so a new handler cannot remember the first half and forget the second. Anything in another team is 404, never 403: whether an incident exists is that team's business. Two bugs this found, both of which would have been silent: * upsertAlerts decided "is this a new occurrence" by looking up the fingerprint alone. Across teams that made team B's first alert look like a re-send of team A's, so it opened no incident at all. The lookups are keyed on (team_id, fingerprint) now, as the index is. * Every uniqueness rule was written for one tenant. Two teams watching two clusters legitimately see the same fingerprint, the same groupKey, and want somebody on call on the same day; all three constraints move to include team_id. Roles inside a team are separate from the system administrator flag: an owner configures the team, a member works its incidents, and an admin is NOT implicitly in every team -- administration is about accounts, not about reading other people's incidents. An admin can still repair a team whose owner has left, which is why requireTeamOwner lets them through. A shift can only be given to somebody in the team. Paging a person who cannot open the incident is worse than paging nobody. The UI is updated only as far as keeping it working: it loads the viewer's teams with the session and uses the first one, since nobody has a second yet. "On call now" shows every team the viewer is in, named only when there is more than one, so the common case reads exactly as before. The team switcher, badges and per-team settings pages are the next step. Breaking for API clients: the schedule endpoints moved under the team, and /api/schedule/current returns an array rather than an object or a 404. terdut-tui will need a version for that. Per-team dead-man configuration is deliberately not here. A heartbeat's incident already opens in the team whose key received it, which is the part that matters for isolation; moving the matchers out of env into per-team rows is a change to how deadman.go is configured rather than to who sees what. Claude-Session: https://claude.ai/code/session_01RHPj4ggeFdEjKKfm4SHbD7	2026-09-20 13:36:24 +02:00
Niklas Ye	279ef6cf8b	Turn incoming alerts into incidents ... The alerts row was both Alertmanager's record and the human work queue, and the two have different owners. The webhook upsert rewrites that row on every notification; acknowledgement, comments and archiving were columns on it that the upsert happened not to touch. So an alert that resolved and re-fired days later still read as acknowledged by whoever acked the first occurrence — the ack outlived the thing it referred to. Nothing recorded transitions either: rows are mutated in place, so there was no timeline and no way to compute how long anything took. Alerts are now read-only signal records with two states, and incidents are the work item: triggered, acknowledged or resolved, with an assignee, a snooze, notes and an append-only timeline. Many alerts map to one incident, and a new occurrence opens a new incident, which is what makes a stale ack impossible rather than merely unlikely. Correlation uses Alertmanager's own groupKey. It already grouped the alerts according to the group_by routing tree the operator configured and sends the result on every webhook, where it was being discarded; adopting it means changing group_by in alertmanager.yml changes correlation here, with no second grouping scheme to configure and keep in sync. An incident opens only when an alert transitions into firing — an unseen fingerprint, a newer startsAt, or a resolved alert starting again. The unchanged notifications Alertmanager re-sends every repeat_interval are none of those. That rule is what lets manual resolution be terminal: without it, closing an incident by hand would be undone by the next re-send of an alert that never stopped firing, and the button would be a lie. Snooze covers the "not now" case instead. Incidents otherwise resolve by cascade, once every alert under them has stopped firing, whether by webhook or by expiry. New incidents are assigned to whoever holds today's schedule entry. The schedule table has existed since the first release with nothing reading it. Also here, following from the split: - Incident severity is a high-water mark over its alerts, never lowered. An incident that hit critical was a critical incident, and downgrading a live one would demote it in the queue while the work is still open. - /api/stats/incidents reports MTTA and MTTR, null rather than zero until there is something to average. Neither was computable before. - Alert archiving becomes sweeper-only housekeeping; the archive people interact with is the incident's. Breaking: the alert acknowledge, archive and comment endpoints are gone, and the alert object drops the acknowledgement fields and gains incident_id. The README maps each removed endpoint to its replacement. Migration 008 backfills an incident per existing alert, archived ones included so no comment is orphaned, carrying acknowledgements across and turning comments into timeline notes. Both documented alert contracts are untouched: received_at still advances on every accepted payload, re-sends included, and resolution_source still says how much to trust ends_at. The upsert is byte-for-byte what it was, now running inside the ingest transaction.	2026-07-30 17:02:13 +02:00
Niklas Ye	a602ff3efc	Document received_at and resolution_source as public contract ... The API reference listed endpoints but never the alert object's fields, so two of them were load-bearing for clients while being described nowhere. received_at appeared only in passing, as a stats filter; resolution_source only inside the stale-expiry prose. Both carry meaning a client cannot derive on its own. starts_at comes from Prometheus and never changes for an alert instance, so received_at is the only signal that a firing alert is still being refreshed — it advances on every accepted webhook, including the unchanged notifications Alertmanager re-sends every repeat_interval. resolution_source then says how much to trust ends_at: under 'alertmanager' it is an end time somebody reported, but under 'expiry' nothing ever reported one, so it is either a stale watermark or the sweep timestamp, and only an upper bound. README gains an alert object field table plus a contract section for each, including the nullability rules and the advice to tolerate unrecognised resolution_source values. The field comments in models.Alert now say these are public API rather than ingest details, and the upsert carries a note at the received_at line, which is where a regression would be introduced. Three tests lock the newly documented behaviour, none of which was covered before — the whole suite passed with the received_at bump deleted from the upsert, because the expiry tests only ever set that column via SQL: - a re-send advances received_at and leaves starts_at alone - a discarded out-of-order retry does not count as a heartbeat - an expiry resolve preserves a reported ends_at watermark and stamps sweep time only when none was known	2026-07-30 09:02:44 +02:00
Niklas Ye	42e846f876	Expire stale firing alerts ... A resolved webhook was the only path out of the firing state, so a notification that was dropped, silenced, or lost to a restart pinned an alert as firing forever — Prometheus showed it resolved while terdut-server kept listing it. The archiver only ever touched resolved alerts, and both the list and stats queries compared status with plain equality, so a stale row was indistinguishable from a live one. A sweeper pass now resolves firing alerts on either of two signals: the ends_at watermark Alertmanager sets on outgoing firing notifications has passed (plus a grace period for clock skew), or no webhook has refreshed the alert within TERDUT_STALE_AFTER (default 6h, above Alertmanager's 4h repeat_interval). Such alerts get resolution_source = 'expiry', distinguishing them from a real 'alertmanager' resolve. Two related webhook bugs fixed alongside: - The upsert had no ordering guard, so a retried firing notification arriving after the resolved one resurrected the alert. Payloads for an older alert instance are now discarded: a stale retry carries the same startsAt, a genuine re-fire a newer one. - archived_at was never cleared on re-fire, leaving a re-fired alert archived and invisible in the default list. Stats now exclude archived alerts to match the default list view; this lowers historical firing/resolved totals. The chart exposes both sweeper durations via sweeper.staleAfter and sweeper.archiveAfter.	2026-07-28 11:49:39 +02:00
Niklas Ye	debc4bf78c	Add alert archiving ... Alerts can be manually archived (POST /api/alerts/{id}/archive) or unarchived (DELETE /api/alerts/{id}/archive). A background goroutine auto-archives resolved alerts older than TERDUT_ARCHIVE_AFTER (default 7d). GET /api/alerts hides archived alerts by default; ?archived=true shows them.	2026-05-22 13:22:45 +02:00
Niklas Ye	c3348a410a	Stage 4: alert acknowledgement and comments ... - Migration 004: acknowledged_by/acknowledged_at columns on alerts, alert_comments table (FK cascade on delete) - POST /api/alerts/{id}/acknowledge — stamps authed user + timestamp, returns updated alert with acknowledged_by username - DELETE /api/alerts/{id}/acknowledge — clears ack (204) - GET /api/alerts/{id}/comments — list in chronological order - POST /api/alerts/{id}/comments — add comment (returns 201) - DELETE /api/alerts/{id}/comments/{commentID} — own comments only (204) - All alert queries now LEFT JOIN users for ack username	2026-05-20 21:57:11 +02:00
Niklas Ye	9b4ca1482f	Stage 3: Alertmanager webhook ingestion and alert query API ... - Migration 003: alerts table with fingerprint UNIQUE, JSON label/annotation columns, nullable ends_at, and indexed status/name/received_at - POST /api/alertmanager/webhook — upserts each alert by fingerprint; zero endsAt ("0001-01-01") stored as NULL (still firing) - GET /api/alerts — filtered list (?status, ?name, ?from, ?to, ?limit) - GET /api/alerts/{id} — single alert lookup	2026-05-20 21:54:17 +02:00