9 Commits

Author SHA1 Message Date
Niklas Ye a4fbd60441 Scope everything to a team, and route alerts by integration key
CI / chart (pull_request) Successful in 1s
CI / security (pull_request) Successful in 13s
CI / test (pull_request) Successful in 1m49s
The core of #4, and what #1 is for: terdut stops being one shared space.
A team owns its incidents, alerts, schedule and integrations; a user sees
exactly the teams they are in. Everything that existed moves into one
Default team and every existing user becomes an owner of it, so the
upgrade is a no-op for the people using it.

Ingestion is the load-bearing half. An alert arrives on a team's
integration key, and the key is both the credential and the routing: it
says that the sender may post, and which team the alerts belong to. That
also closes the unauthenticated webhook -- the old path stays for one
release, deprecated and routed to the oldest team, so an upgrade does not
stop delivering while somebody edits the Alertmanager config.

Scoping is enforced in as few places as possible, because the failure
mode is silent. serveAs loads the caller's memberships once; list queries
carry `team_id = ANY(...)`; and every incident route goes through
incidentIDParam, which now parses the id AND checks the team in the same
call, so a new handler cannot remember the first half and forget the
second. Anything in another team is 404, never 403: whether an incident
exists is that team's business.

Two bugs this found, both of which would have been silent:

  * upsertAlerts decided "is this a new occurrence" by looking up the
    fingerprint alone. Across teams that made team B's first alert look
    like a re-send of team A's, so it opened no incident at all. The
    lookups are keyed on (team_id, fingerprint) now, as the index is.

  * Every uniqueness rule was written for one tenant. Two teams watching
    two clusters legitimately see the same fingerprint, the same
    groupKey, and want somebody on call on the same day; all three
    constraints move to include team_id.

Roles inside a team are separate from the system administrator flag: an
owner configures the team, a member works its incidents, and an admin is
NOT implicitly in every team -- administration is about accounts, not
about reading other people's incidents. An admin can still repair a team
whose owner has left, which is why requireTeamOwner lets them through.

A shift can only be given to somebody in the team. Paging a person who
cannot open the incident is worse than paging nobody.

The UI is updated only as far as keeping it working: it loads the
viewer's teams with the session and uses the first one, since nobody has
a second yet. "On call now" shows every team the viewer is in, named only
when there is more than one, so the common case reads exactly as before.
The team switcher, badges and per-team settings pages are the next step.

Breaking for API clients: the schedule endpoints moved under the team,
and /api/schedule/current returns an array rather than an object or a
404. terdut-tui will need a version for that.

Per-team dead-man configuration is deliberately not here. A heartbeat's
incident already opens in the team whose key received it, which is the
part that matters for isolation; moving the matchers out of env into
per-team rows is a change to how deadman.go is configured rather than to
who sees what.

Claude-Session: https://claude.ai/code/session_01RHPj4ggeFdEjKKfm4SHbD7
2026-09-20 13:36:24 +02:00
Niklas Ye dc39e3a5d3 Move the database to Postgres, before teams need the schema
CI / chart (pull_request) Successful in 1s
CI / security (pull_request) Successful in 17s
CI / test (pull_request) Successful in 2m5s
First step of #1, and it goes first for one reason: #4 adds a team_id to
nearly every table, and doing that twice -- once for SQLite, once for
Postgres -- is work nobody gets paid for. The teams migrations now only
have to be written against one database.

The ten SQLite migrations are replaced by a single Postgres baseline
rather than ported one by one. They were incremental in a way that has
no value on a fresh install: 004 adds columns 008 drops again, and 008's
backfill rewrites data a Postgres database never had. The history stays
in git; the schema they add up to is now 001_baseline.sql.

Timestamps stay BIGINT unix seconds and are NOT converted to timestamptz.
Everything in Go already speaks epochs, so converting would have been a
second, larger change riding along inside this one. It is worth doing on
its own. The JSON columns did move to jsonb, because #4 will want to
filter and index on labels.

Most of the port is mechanical -- 170 placeholders from ? to $1 -- but
four things needed more than a search and replace:

  * Dynamically built WHERE clauses cannot keep their numbering straight
    by hand, so they hand out placeholders through sqlArgs instead. A
    filter can now be added or reordered without renumbering anything.

  * SUM(resolved_at IS NULL) was SQLite counting a boolean as 0 or 1.
    Postgres has no sum(boolean), and this was breaking every dead man's
    switch -- silently, since the sweeper only logs. Now COUNT(*) FILTER.

  * unixepoch() became FLOOR(EXTRACT(EPOCH FROM now()))::bigint. The
    FLOOR is load-bearing: a bare cast rounds half up, so a row written
    at .6 of a second claimed a timestamp a second in the future and
    disagreed with the time.Now().Unix() the Go side stamps.

  * The unique-violation check matched SQLite's error text. It matches
    SQLSTATE 23505 now, so a renamed constraint cannot turn a 409 back
    into a 500.

Tests need a real Postgres, because there is no in-memory Postgres the
way there was an in-memory SQLite. Each test gets its own schema on a
shared server -- cheaper than a database each, and still isolated.
TERDUT_TEST_DSN says where it is; `make test-db` starts one locally and
ci.yaml runs one as a service container. An unset DSN fails the suite
rather than skipping it: a run that quietly tests nothing is worse than
one that does not run.

TestMigration_BackfillCarriesAckAndComments is deleted along with the
migrations it replayed. What it protected -- an upgrade not losing
acknowledgements and comments -- now belongs to scripts/sqlite-to-postgres.go,
which is build-tagged so the SQLite driver stays out of the server
binary. Both are meant to be deleted once this install has migrated.

The chart loses the PVC, the data volume and the python backup sidecar,
and requires database.dsnSecret.name: it provisions no database and
cannot guess where the credentials live, so a render without it is meant
to fail. Backups move to where Postgres actually runs. The other half of
that -- the postgresql CR, the k8up pg_dump annotation and the network
policy -- is a change to the wrapper chart in Ryuvia/charts and is not in
here.

Verified rather than assumed: the gate is green with -race against
Postgres 17, govulncheck and gitleaks are clean, and the migration script
was run end to end against a SQLite database built at the old schema and
seeded in every table. Ids survive, so incidents keep their numbers and
every foreign key still points where it did; the identity sequences are
moved past the copied ids, and a webhook after the migration opened
incident 12 rather than colliding at 1.
2026-09-20 10:44:12 +02:00
Niklas Ye 03504b61be gofmt: restore import grouping after the module rename
CI / test (push) Successful in 5s
The rename to git.ryuvia.com/niklas/... was a plain string substitution, so it
left the import blocks in their old order. The new path sorts before
github.com/go-chi/..., where the old one sorted after, which gofmt considers
unformatted.

go vet does not look at import order, so CI had nothing to say about it.
2026-08-19 21:21:33 +02:00
Niklas Ye 289eca8076 Move to Gitea: git.ryuvia.com/niklas/terdut-server
CI / test (push) Successful in 2m15s
The module path, the container image, the Helm chart and the CI pipeline all
named GitHub. They now name the Gitea instance everything else already runs on.

The workflows are rewritten rather than translated. Gitea's runner image is
ubuntu:22.04, whose nodejs is Node 12, so no JS action runs there at all --
actions/checkout@v4 dies with a SyntaxError before it does anything. Every step
is shell, checkout is a plain clone (this repo is public, so it needs no
credential), and the jobs that need docker or helm run in host mode because the
dind bridge a `container:` job gets cannot reach github.com or get.helm.sh.

Two consequences worth naming:

- upload-artifact/download-artifact are also JS actions, and there is no
  artifact store here, so the job that builds the binaries is the job that
  publishes them. Nothing is passed between jobs.
- setup-qemu-action is gone with the rest, and the runner has no binfmt
  registration. The Dockerfile's builder stage now runs on $BUILDPLATFORM and
  cross-compiles from TARGETARCH instead, which is what keeps the arm64 image
  buildable -- and makes it native rather than emulated.

The chart moves from a GitHub Pages index to an OCI artifact in Gitea's
registry. Publishing stays tag-only for the reason recorded in release.yaml: a
workflow triggered by the branch push cannot know the version it is about to be
tagged with.

The GitHub repository is left in place and untouched. Nothing pushes to it any
more, but its existing release downloads and chart index keep resolving.
2026-08-19 20:39:10 +02:00
Niklas Ye 279ef6cf8b Turn incoming alerts into incidents
Release / build (amd64, linux) (push) Failing after 11s
Release / build (amd64, darwin) (push) Failing after 12s
Release / build (arm64, darwin) (push) Failing after 11s
Release / build (arm64, linux) (push) Failing after 11s
Release / release (push) Has been skipped
Release / chart (push) Failing after 13s
Release / docker (push) Failing after 19s
The alerts row was both Alertmanager's record and the human work queue, and
the two have different owners. The webhook upsert rewrites that row on every
notification; acknowledgement, comments and archiving were columns on it that
the upsert happened not to touch. So an alert that resolved and re-fired days
later still read as acknowledged by whoever acked the first occurrence — the
ack outlived the thing it referred to. Nothing recorded transitions either:
rows are mutated in place, so there was no timeline and no way to compute how
long anything took.

Alerts are now read-only signal records with two states, and incidents are
the work item: triggered, acknowledged or resolved, with an assignee, a
snooze, notes and an append-only timeline. Many alerts map to one incident,
and a new occurrence opens a new incident, which is what makes a stale ack
impossible rather than merely unlikely.

Correlation uses Alertmanager's own groupKey. It already grouped the alerts
according to the group_by routing tree the operator configured and sends the
result on every webhook, where it was being discarded; adopting it means
changing group_by in alertmanager.yml changes correlation here, with no
second grouping scheme to configure and keep in sync.

An incident opens only when an alert transitions into firing — an unseen
fingerprint, a newer startsAt, or a resolved alert starting again. The
unchanged notifications Alertmanager re-sends every repeat_interval are none
of those. That rule is what lets manual resolution be terminal: without it,
closing an incident by hand would be undone by the next re-send of an alert
that never stopped firing, and the button would be a lie. Snooze covers the
"not now" case instead. Incidents otherwise resolve by cascade, once every
alert under them has stopped firing, whether by webhook or by expiry.

New incidents are assigned to whoever holds today's schedule entry. The
schedule table has existed since the first release with nothing reading it.

Also here, following from the split:

  - Incident severity is a high-water mark over its alerts, never lowered.
    An incident that hit critical was a critical incident, and downgrading a
    live one would demote it in the queue while the work is still open.
  - /api/stats/incidents reports MTTA and MTTR, null rather than zero until
    there is something to average. Neither was computable before.
  - Alert archiving becomes sweeper-only housekeeping; the archive people
    interact with is the incident's.

Breaking: the alert acknowledge, archive and comment endpoints are gone, and
the alert object drops the acknowledgement fields and gains incident_id. The
README maps each removed endpoint to its replacement. Migration 008 backfills
an incident per existing alert, archived ones included so no comment is
orphaned, carrying acknowledgements across and turning comments into timeline
notes.

Both documented alert contracts are untouched: received_at still advances on
every accepted payload, re-sends included, and resolution_source still says
how much to trust ends_at. The upsert is byte-for-byte what it was, now
running inside the ingest transaction.
2026-07-30 17:02:13 +02:00
Niklas Ye 42e846f876 Expire stale firing alerts
Release / build (amd64, darwin) (push) Failing after 12s
Release / build (arm64, darwin) (push) Failing after 11s
Release / build (arm64, linux) (push) Failing after 11s
Release / release (push) Has been skipped
Release / docker (push) Failing after 19s
Release / build (amd64, linux) (push) Failing after 12s
Release / chart (push) Failing after 9s
A resolved webhook was the only path out of the firing state, so a
notification that was dropped, silenced, or lost to a restart pinned an
alert as firing forever — Prometheus showed it resolved while
terdut-server kept listing it. The archiver only ever touched resolved
alerts, and both the list and stats queries compared status with plain
equality, so a stale row was indistinguishable from a live one.

A sweeper pass now resolves firing alerts on either of two signals: the
ends_at watermark Alertmanager sets on outgoing firing notifications has
passed (plus a grace period for clock skew), or no webhook has refreshed
the alert within TERDUT_STALE_AFTER (default 6h, above Alertmanager's 4h
repeat_interval). Such alerts get resolution_source = 'expiry',
distinguishing them from a real 'alertmanager' resolve.

Two related webhook bugs fixed alongside:

  - The upsert had no ordering guard, so a retried firing notification
    arriving after the resolved one resurrected the alert. Payloads for
    an older alert instance are now discarded: a stale retry carries the
    same startsAt, a genuine re-fire a newer one.
  - archived_at was never cleared on re-fire, leaving a re-fired alert
    archived and invisible in the default list.

Stats now exclude archived alerts to match the default list view; this
lowers historical firing/resolved totals.

The chart exposes both sweeper durations via sweeper.staleAfter and
sweeper.archiveAfter.
2026-07-28 11:49:39 +02:00
Niklas Ye debc4bf78c Add alert archiving
Release / build (amd64, darwin) (push) Failing after 2m46s
Release / build (amd64, linux) (push) Failing after 2m26s
Release / build (arm64, darwin) (push) Failing after 1m40s
Release / build (arm64, linux) (push) Failing after 10s
Release / release (push) Has been skipped
Release / chart (push) Failing after 11s
Release / docker (push) Failing after 19s
Alerts can be manually archived (POST /api/alerts/{id}/archive) or
unarchived (DELETE /api/alerts/{id}/archive). A background goroutine
auto-archives resolved alerts older than TERDUT_ARCHIVE_AFTER (default 7d).
GET /api/alerts hides archived alerts by default; ?archived=true shows them.
2026-05-22 13:22:45 +02:00
Niklas Ye c3348a410a Stage 4: alert acknowledgement and comments
- Migration 004: acknowledged_by/acknowledged_at columns on alerts,
  alert_comments table (FK cascade on delete)
- POST /api/alerts/{id}/acknowledge — stamps authed user + timestamp,
  returns updated alert with acknowledged_by username
- DELETE /api/alerts/{id}/acknowledge — clears ack (204)
- GET /api/alerts/{id}/comments — list in chronological order
- POST /api/alerts/{id}/comments — add comment (returns 201)
- DELETE /api/alerts/{id}/comments/{commentID} — own comments only (204)
- All alert queries now LEFT JOIN users for ack username
2026-05-20 21:57:11 +02:00
Niklas Ye 9b4ca1482f Stage 3: Alertmanager webhook ingestion and alert query API
- Migration 003: alerts table with fingerprint UNIQUE, JSON label/annotation
  columns, nullable ends_at, and indexed status/name/received_at
- POST /api/alertmanager/webhook — upserts each alert by fingerprint;
  zero endsAt ("0001-01-01") stored as NULL (still firing)
- GET /api/alerts — filtered list (?status, ?name, ?from, ?to, ?limit)
- GET /api/alerts/{id} — single alert lookup
2026-05-20 21:54:17 +02:00