8 Commits

Author	SHA1	Message	Date
Niklas Ye	60ebb75cd2	Show notes from similar earlier incidents ... Each incident gets a signature: the alert name plus the group labels that say what is broken, minus the ones that only say where it ran (instance, pod, container, ...). GET /api/incidents/{id}/similar returns resolved incidents in the same team with the same signature that have notes. Notes can be marked as the resolution note, "what fixed it", either with a resolution field on resolve or pinned on a note. Those lead the similar list, show on the incident page as "Seen before", and the triggered notification carries the latest one. Claude-Session: https://claude.ai/code/session_01MMados3BD1oSjevHxbmVqU	2026-09-25 15:42:25 +02:00
Niklas Ye	3183e7e5c5	Page the next person when nobody answers ... Closes #6, and closes the thing this whole line of work was opened for. Until now an unacknowledged incident re-paged the same topic every notify_repeat forever, which is a louder version of the same silence: if the person on call is asleep, out of signal or has left the company, nothing else happened. A team can now configure an ordered ladder. Each level has a timeout and a set of targets; a target is a named person or whoever the team's rota says is on call today. That second kind is the one that keeps working when the rota changes and nobody remembers to edit the policy. When a level's timeout passes with the incident still triggered, the next level is paged; off the end the chain repeats repeat_count times and then the team's fallback topic is paged once. The incident stays open throughout, because running out of people to wake is not somebody answering. Escalation rides the notifier's existing 30-second tick and its outbox rather than adding a second scheduler, and runs before delivery so a level that comes due on a tick is paged on that tick. Each target gets its own outbox row and therefore its own Acknowledge token: the button in a notification must acknowledge as the person holding the phone, not as whoever was paged first. Acknowledging or resolving takes the incident off the ladder. Snoozing pauses it -- a deliberate "not now" holds the ladder where it is and it resumes when the snooze runs out, rather than carrying on without the person who asked for quiet. Reminders and escalation never both run. A team with a ladder gets escalation; a team without keeps today's behaviour exactly. Both would mean two pages for one silence, which is how a tool gets muted. A level whose targets cannot be reached -- no topic, a disabled account, an empty rota -- is entered anyway, recorded as "nobody reachable", and the ladder moves on. Stalling on a rung that cannot ring would be the failure this feature exists to prevent, wearing the feature's clothes. A policy with such a level cannot be created, but an older row could hold one. The API replaces the ladder wholesale rather than patching a rung, because the levels are an order: editing one has to answer what happens to the numbering of the others, and a whole-ladder PUT makes that the client's decision and the edit atomic. Verified against a live server as well as in tests: alice paged, nobody answers, bob paged, nobody answers, the fallback topic paged once and the timeline reading "level 2: bob" then "escalation exhausted: paged terdut-oncall-all" -- and a second incident acknowledged before its timeout, which woke nobody else. No UI yet. The team-settings screens for escalation, integrations and dead man's switches are all still missing, and they are one piece of work rather than three. Claude-Session: https://claude.ai/code/session_01RHPj4ggeFdEjKKfm4SHbD7	2026-09-20 18:37:54 +02:00
Niklas Ye	b0a02c010b	Add an admin page, and move the behaviour settings into the database ... Closes #5. Three of the server's tunables were environment variables, which meant changing how long an incident waits before being paged again required editing a chart, merging it and waiting for a reconcile. They are behaviour rather than infrastructure, and the difference is who needs to change them and how often. The split is by who owns the value. What stays in the environment is where the server is plugged in: the listen address, the DSN, the ntfy URL and token, the public URL. Those are needed before the database is open and two of them are credentials -- the settings endpoint reports that ntfy is configured and that a token is set, and never what either is. What moves is how it behaves: the notify repeat interval, the stale window and the archive window. The environment variable becomes the seed rather than the setting, written once on first start and never overwritten, so a redeploy cannot put a chart's default back over an administrator's edit -- the rule the per-team dead man's switches already follow. The loops read the current value per tick, so a change at 02:00 is obeyed at 02:00. Key/value rather than a column per knob: #6 and #7 will both add settings, and a table shaped one-column-per-setting needs a migration for each. The cost is that values are text and the accessor has to say what type it wanted, which settings.go does in one place. Unknown keys are refused rather than stored -- a typo that wrote notify_repeat_second would otherwise sit in the table looking like configuration and doing nothing -- and each value has bounds loose enough to catch a slipped decimal point without having an opinion about anybody's rota. Disabling an account is new, and is not deleting one. Deleting a user nulls acknowledged_by and assigned_to, which quietly rewrites who did what during an incident months after the fact. A disabled user cannot authenticate by either credential, loses their sessions immediately, and stays the name on every acknowledgement they made. The check is part of the lookup in serveAs rather than a test afterwards, so there is no path where the row is loaded and the flag is then forgotten. The page itself is a fourth tab, shown only to an administrator and only as a courtesy: every endpoint under it is refused with 403 regardless, so somebody who types /admin gets an explanation rather than a blank screen. It lists teams with their size and open-incident count, users with their flags, and the settings with their bounds -- plus the environment half, read-only, so somebody hunting for the ntfy URL learns where it lives instead of concluding the server has none. Delete is disabled rather than offered-and-refused for a team with open incidents, and neither admin action is offered on your own account, since the server refuses both. Claude-Session: https://claude.ai/code/session_01RHPj4ggeFdEjKKfm4SHbD7	2026-09-20 18:23:46 +02:00
Niklas Ye	dc39e3a5d3	Move the database to Postgres, before teams need the schema ... First step of #1, and it goes first for one reason: #4 adds a team_id to nearly every table, and doing that twice -- once for SQLite, once for Postgres -- is work nobody gets paid for. The teams migrations now only have to be written against one database. The ten SQLite migrations are replaced by a single Postgres baseline rather than ported one by one. They were incremental in a way that has no value on a fresh install: 004 adds columns 008 drops again, and 008's backfill rewrites data a Postgres database never had. The history stays in git; the schema they add up to is now 001_baseline.sql. Timestamps stay BIGINT unix seconds and are NOT converted to timestamptz. Everything in Go already speaks epochs, so converting would have been a second, larger change riding along inside this one. It is worth doing on its own. The JSON columns did move to jsonb, because #4 will want to filter and index on labels. Most of the port is mechanical -- 170 placeholders from ? to $1 -- but four things needed more than a search and replace: * Dynamically built WHERE clauses cannot keep their numbering straight by hand, so they hand out placeholders through sqlArgs instead. A filter can now be added or reordered without renumbering anything. * SUM(resolved_at IS NULL) was SQLite counting a boolean as 0 or 1. Postgres has no sum(boolean), and this was breaking every dead man's switch -- silently, since the sweeper only logs. Now COUNT(*) FILTER. * unixepoch() became FLOOR(EXTRACT(EPOCH FROM now()))::bigint. The FLOOR is load-bearing: a bare cast rounds half up, so a row written at .6 of a second claimed a timestamp a second in the future and disagreed with the time.Now().Unix() the Go side stamps. * The unique-violation check matched SQLite's error text. It matches SQLSTATE 23505 now, so a renamed constraint cannot turn a 409 back into a 500. Tests need a real Postgres, because there is no in-memory Postgres the way there was an in-memory SQLite. Each test gets its own schema on a shared server -- cheaper than a database each, and still isolated. TERDUT_TEST_DSN says where it is; `make test-db` starts one locally and ci.yaml runs one as a service container. An unset DSN fails the suite rather than skipping it: a run that quietly tests nothing is worse than one that does not run. TestMigration_BackfillCarriesAckAndComments is deleted along with the migrations it replayed. What it protected -- an upgrade not losing acknowledgements and comments -- now belongs to scripts/sqlite-to-postgres.go, which is build-tagged so the SQLite driver stays out of the server binary. Both are meant to be deleted once this install has migrated. The chart loses the PVC, the data volume and the python backup sidecar, and requires database.dsnSecret.name: it provisions no database and cannot guess where the credentials live, so a render without it is meant to fail. Backups move to where Postgres actually runs. The other half of that -- the postgresql CR, the k8up pg_dump annotation and the network policy -- is a change to the wrapper chart in Ryuvia/charts and is not in here. Verified rather than assumed: the gate is green with -race against Postgres 17, govulncheck and gitleaks are clean, and the migration script was run end to end against a SQLite database built at the old schema and seeded in every table. Ids survive, so incidents keep their numbers and every foreign key still points where it did; the identity sequences are moved past the copied ids, and a webhook after the migration opened incident 12 rather than colliding at 1.	2026-09-20 10:44:12 +02:00
Niklas Ye	dc3879eca6	Serve a web UI for the incident queue, built for phones ... Whoever is on call gets paged on a phone, and until now the only ways to act on a page were the notification's Acknowledge button or a terminal. Tapping the notification itself opened /api/incidents/{id}, which a browser can only answer with a 401 in JSON. The server now serves a web UI at / covering the incident queue, each incident's alerts and timeline with every action on it, who is on call, the alert feed, and changing your own password. The notification link now points at /incidents/{id} in that UI. It is embedded in the binary and has no build step: plain HTML, CSS and ES modules under internal/web/static, served with an ETag per file and a CSP that allows nothing from any other origin. That is how rd-web is built. It avoids adding a node toolchain to the Dockerfile and the pipeline for a page this size, and it keeps the page on the same origin as the API, so no CORS is needed and nothing else has to be deployed. Paths without a file extension fall back to index.html, so a deep link survives a reload. An unknown path under /api/ still gets a JSON 404 rather than the page. Signing in uses a username and password, because pasting a 64-character API key into a phone at 3am is not a sign-in flow. Users have no password until one is set through PUT /api/users/{id}/password, or optionally at bootstrap. A user without a password is exactly where they were before this commit and can only use API keys. A login sets an HttpOnly, SameSite=Lax session cookie. It lasts 30 days and slides forward while in use, so an on-call phone does not sign itself out. Only the token's hash is stored, as for API keys. The cookie needs a CSRF guard where a bearer header does not, because browsers attach cookies to requests other sites make. So cookie- authenticated requests go through Go 1.25's http.CrossOriginProtection, and bearer requests do not. A request carrying an Authorization header is judged on that header alone and never falls back to the cookie. Changing a password ends every other session of that user. Changing your own requires the current password, so a phone left signed in cannot be used to take the account over. Failed logins are counted per username and per client address. Ten failures for one username in 15 minutes refuse that username for the rest of the window, even with the right password. That makes locking somebody out possible for anyone who knows their username. It was accepted because the alternative is unlimited guessing, and during a lockout the notification's Acknowledge button and API keys keep working. The address limit reads the first X-Forwarded-For hop, since behind the gateway RemoteAddr is Envoy. It is looser, because a whole office behind one NAT shares it. The Secure flag follows TERDUT_PUBLIC_URL, since TLS terminates at the gateway and the server itself only ever sees plain HTTP. The chart already defaults that variable to https://<hostname>. Schedule editing, statistics and user management stay in terdut-tui for now. The API they use is unchanged, and bearer authentication behaves exactly as before.	2026-09-19 17:48:21 +02:00
Niklas Ye	289eca8076	Move to Gitea: git.ryuvia.com/niklas/terdut-server ... The module path, the container image, the Helm chart and the CI pipeline all named GitHub. They now name the Gitea instance everything else already runs on. The workflows are rewritten rather than translated. Gitea's runner image is ubuntu:22.04, whose nodejs is Node 12, so no JS action runs there at all -- actions/checkout@v4 dies with a SyntaxError before it does anything. Every step is shell, checkout is a plain clone (this repo is public, so it needs no credential), and the jobs that need docker or helm run in host mode because the dind bridge a `container:` job gets cannot reach github.com or get.helm.sh. Two consequences worth naming: - upload-artifact/download-artifact are also JS actions, and there is no artifact store here, so the job that builds the binaries is the job that publishes them. Nothing is passed between jobs. - setup-qemu-action is gone with the rest, and the runner has no binfmt registration. The Dockerfile's builder stage now runs on $BUILDPLATFORM and cross-compiles from TARGETARCH instead, which is what keeps the arm64 image buildable -- and makes it native rather than emulated. The chart moves from a GitHub Pages index to an OCI artifact in Gitea's registry. Publishing stays tag-only for the reason recorded in release.yaml: a workflow triggered by the branch push cannot know the version it is about to be tagged with. The GitHub repository is left in place and untouched. Nothing pushes to it any more, but its existing release downloads and chart index keep resolving.	2026-08-19 20:39:10 +02:00
Niklas Ye	4224dbe96c	Record notification delivery on the incident timeline ... An incident's history went quiet after "Incident opened": nothing said that anybody had been paged, reminded, or told it resolved. Delivery lived only in the notifications outbox, which no API exposes, so when a page failed to arrive there was nothing in the product that said whether it had been sent. The notifier now writes two event types. A notified event once ntfy accepts the publish, carrying the kind in detail and the paged user in user_id — absent when the page went to the shared fallback topic, which belongs to nobody. And a notify_failed event when a notification exhausts its retries, which is the one worth having: without it a page that never landed leaves the timeline identical to one that did. Both are written from the delivery result rather than at enqueue. A queued notification is an intention, and the timeline is append-only, so claiming somebody was told before ntfy accepted it would be a lie that stays there. A failed timeline write is logged rather than returned, so it cannot make a delivered row look unsent and send the page twice. The topic is deliberately in neither: it is a shared secret with the ntfy server, and every API key can read the timeline. No migration — incident_events.type is free text, unlike notifications.kind.	2026-08-07 13:31:03 +02:00
Niklas Ye	bc285799d1	Page the on-call person when an incident opens ... An incident opened, got assigned to whoever held today's schedule entry, and then sat there silently until somebody thought to look. The schedule and the incident model were both built; nothing reached the person holding the pager. Notifications go out through ntfy, over plain HTTP with no new dependencies. Delivery is an outbox rather than an inline call: the pool is limited to a single connection, so a POST made while holding the webhook's transaction would stall every other request behind it. The webhook inserts a row and a notifier goroutine sends it within a tick, retrying with exponential backoff. Only opening an incident has to resolve a topic from scratch. Reminders and all-clears reuse whatever that first notification chose, which keeps configuration out of resolveIfSettled and gives the right rule for free: you only hear that something resolved if you were told it started. Each push carries an Acknowledge button, because the useful thing to do at 3am is stop the pager without unlocking anything. It POSTs to an unauthenticated /api/notify/ack/{token} — a notification body lives on the ntfy server and in the device cache, so a real API key must never appear in one. The token is minted per delivery, scoped to one incident and one action, and expires in a day. Reminders repeat until the incident stops being untouched. The stop conditions are the states that already mean somebody has it: acknowledged, snoozed, resolved, archived. Snooze is the mute button, so there is no separate reminder cap. Notifications sent to the fallback topic carry no Acknowledge button. The topic is shared, and a button on it would let any subscriber acknowledge as somebody else.	2026-08-07 08:51:38 +02:00