23 Commits

Author	SHA1	Message	Date
Niklas Ye	60ebb75cd2	Show notes from similar earlier incidents ... Each incident gets a signature: the alert name plus the group labels that say what is broken, minus the ones that only say where it ran (instance, pod, container, ...). GET /api/incidents/{id}/similar returns resolved incidents in the same team with the same signature that have notes. Notes can be marked as the resolution note, "what fixed it", either with a resolution field on resolve or pinned on a note. Those lead the similar list, show on the incident page as "Seen before", and the triggered notification carries the latest one. Claude-Session: https://claude.ai/code/session_01MMados3BD1oSjevHxbmVqU	2026-09-25 15:42:25 +02:00
Niklas Ye	a6fa673e08	Give every team a page of its own ... The Admin tab's team list was growing controls the way the user list did before ac9af8e: a Rename button behind window.prompt, a Delete beside it, and -- on the Users page, of all places -- an invite form with a team picker in front of it. The picker was the admission that an invite is a fact about a team rather than about the server, and a prompt() is the wrong place to read a 409 about a name already taken. So a team is now a subject with a page, at /admin/teams/{id}, the mirror of /admin/users/{id}: when it was created, how many are in it and how much is open, a field to rename it, the members with their roles, the invites into it, and deletion. The list goes back to being a list, and the name in it is the way in. The member list is the one thing there that needed a new endpoint. GET /api/teams/{id}/members is requireTeamMember and answers 404 to an administrator who is not in the team, and that stays exactly as it is: member means membership and nothing else. Reading a team's shape is a different question from reading its work, so it gets an endpoint of its own under AdminOnly -- GET /api/admin/teams/{id}, returning {"team", "members"} -- rather than an exception carved into that rule. It is a wrapper and not a team with the members hung off it, because "members" already means a count on the list endpoint and one name must not be a number in one answer and an array in the next. The query and its ordering are copied from handleListTeamMembers so the two answers to "who is in this team" cannot disagree. An administrator still sees none of that team's incidents, alerts or rota. Nothing about what the flag may do changed; it could already rename and delete any team, and staff one it is not in. Rename now trims what it is given, as creation has always trimmed. Before this, " " was a legal name to rename a team to but not to create one with, which is one rule stated twice and applied once. Nobody has looked at this in a browser, the caveat ac9af8e and 07914d5 both carried. What is checked is the wiring: admin_test.go covers the new endpoint for an administrator outside the team, the 404 the member-only endpoint still gives that same administrator, the 403 for a member who is not one, a 404 for a team that does not exist, a 400 for an id that is not a number, and the trim; the module graph evaluates at /admin/teams/{id}, and the server serves index.html there, so a reload survives. Claude-Session: https://claude.ai/code/session_01RHPj4ggeFdEjKKfm4SHbD7	2026-09-22 09:13:13 +02:00
Niklas Ye	ac9af8e4f5	Manage a person's account and teams from one page ... The Admin tab could make somebody an administrator and disable them, and nothing else. Setting a first password, deleting an account and seeing which teams a person is in all meant curl, and the last one meant opening each team in turn — the Team tab answers "who is in this team", which is the wrong way round when the question is about a person. A name in the user list now opens /admin/users/{id}: their email and when they joined, where their notifications go, the administrator and disabled flags, the teams they are in with their role in each, a password field for a first or forgotten one, and deletion. A section of its own rather than an expanding row, because memberships and the account actions together are more than a table row can hold and still be read on a phone. Adding somebody mints an invite link into a chosen team rather than creating a bare account. POST /api/users makes a user with no password and no team, who can sign in nowhere and would see nothing if they did; the invite machinery from #7 already solves both, and the password is chosen by the person it belongs to instead of passing through an administrator. One new endpoint, GET /api/users/{id}/teams, self or admin. /api/teams is always about the caller and cannot be asked about anybody else. It 404s for a user who does not exist, so the page can tell "in no teams" from "no such person" — an empty list is a real answer and needed to stay one. No authorisation changed, and the interesting part is why it did not. requireTeamOwner has accepted the administrator flag since a4fbd60, with the reason in its own comment: somebody has to be able to repair a team whose owner has left. It guards nine call sites, so an administrator has always been able to configure any team on this server — while #1's decision table and this README both said an admin "is not implicitly in every team", full stop. The code was right and the prose was wrong in the safe-sounding direction, which is the worse way round to have it. So the documentation moved to meet the code. The Teams table marks owner as owner-or-admin, and the Authentication section states the two directions separately: an administrator configures any team, and reads none, because callerTeamIDs is built from real memberships only. Joining a team to see its queue is a membership change and shows as one. TestAdmin_ConfiguresATeamTheyAreNotIn pins both halves — the admin renames, invites, adds and removes on a team they are not in, then sees zero of its incidents. Nothing tested this from v0.12.0 to here, which is why four releases of prose could contradict it quietly. The UI has not been opened in a browser. Its wiring is checked — every cross-module import resolves, every api.* call exists, every CSS class has a rule, and the deep link serves index.html — but nobody has clicked through it, least of all at phone width. Claude-Session: https://claude.ai/code/session_01RHPj4ggeFdEjKKfm4SHbD7	2026-09-21 12:55:42 +02:00
Niklas Ye	b39aac36b7	Add the sign-up page and the first-run checklist ... Second half of #7. The API could create accounts from invite links since the last change; this is the part somebody can actually use. /signup is the one route that works without a session. It asks the server what it may offer before showing anything: an invite link that is good names the team it leads to, a link that is not says so before somebody picks a password rather than after, and an invite-only server with no link says that instead of presenting a form it will refuse. The login card only offers "create one" when sign-up is open, so the door nobody can walk through is not advertised. Signing up signs you in and lands on the queue, because the alternative is a form saying "now go and log in" about the credential just chosen. The checklist is the other half. Four things have to be true before an alert reaches a phone -- a notification topic, somebody on the rota, an alert source, and an alert that has actually arrived -- and on a fresh install none of them are. It sits above the queue until they are. It is computed from the data rather than from stored progress: a topic is set or it is not, an integration exists or it does not. That means it cannot claim a step is done when it is not, and it comes back by itself if somebody deletes their integration a month later. The only stored state is the dismissal, which is per user and not per browser -- finishing on a laptop should not leave the phone nagging. The topic step is the only one the checklist can finish itself, and the only proof that counts is a phone buzzing, so there is a test push. POST /api/me/notify/test publishes directly rather than through the outbox, which requires an incident this deliberately does not have. Its failure is the useful part: a wrong topic, a rejected token and an ntfy that is down all look identical from the phone, which is silence, so the error comes back to the browser instead. Verified against a live server with a real ntfy stand-in, the whole path: an owner mints an invite, the sign-up page reports it valid and names the team, the invitee signs up and is signed in as a member of that team, the checklist's four questions answer correctly on a fresh install, a test push is refused with no topic and delivered with one -- "PAGED terdut-owner | terdut test" -- and the dismissal survives a reload. Claude-Session: https://claude.ai/code/session_01RHPj4ggeFdEjKKfm4SHbD7	2026-09-21 10:47:15 +02:00
Niklas Ye	d827ceedff	Add self-service sign-up and invite links ... First half of #7. Until now the only way to get an account was for somebody who already had one to create it, and the login page told people to "ask an admin" -- workable for one operator, impossible for a team. Two modes, chosen by an administrator in the settings table: invite_only, which is the default, and open. A third domain-restricted mode was considered and dropped, because with no email in this server there is nothing to verify an address against and it would only check the domain of a string somebody typed. The default is the closed door. An install that gets a public hostname before anybody has thought about sign-up should not be collecting accounts from the internet, and the failure mode of a typo in the setting is invite_only rather than open. An invite is a link, not an email. Adding SMTP to send one message would be a subsystem to run, secure and monitor; the person inviting sends the link however they already talk to the person they are inviting. A link carries the team and the role, because an account in no team sees an empty queue and can be paged by nobody -- that is not a state to invite somebody into. Links are single-use by default, expire after seven days, and can be revoked before that: a link that works forever is a credential nobody remembers issuing, sitting in a chat log. The uses counter is incremented inside the sign-up transaction and guarded by `uses < max_uses`, so two people redeeming the last use at once cannot both get in. GET /api/signup reports the mode and whether a link is usable, so the form can say "this link has expired" before somebody picks a password rather than after. It gives one answer for expired, revoked, used up and never existed: telling a stranger which it was tells them something about links they do not hold. Sign-up signs you in. The alternative is a form that says "now go and log in", which is the same credential typed twice. login and signup now share startSession rather than each minting a cookie. Rate-limited per address on its own limiter, not login's: a burst of sign-ups must not lock somebody out of logging in. The settings table grew a second shape for this. It held only durations; signup_mode is a word from a fixed list, so the admin endpoint now validates everything before writing anything -- a request that sets two settings and gets one wrong changes neither. Still to come in #7: the sign-up and invite-redemption pages, the first-run checklist, and the in-app integration instructions. The schema carries onboarding_dismissed_at for the checklist already. Claude-Session: https://claude.ai/code/session_01RHPj4ggeFdEjKKfm4SHbD7	2026-09-21 09:06:48 +02:00
Niklas Ye	3183e7e5c5	Page the next person when nobody answers ... Closes #6, and closes the thing this whole line of work was opened for. Until now an unacknowledged incident re-paged the same topic every notify_repeat forever, which is a louder version of the same silence: if the person on call is asleep, out of signal or has left the company, nothing else happened. A team can now configure an ordered ladder. Each level has a timeout and a set of targets; a target is a named person or whoever the team's rota says is on call today. That second kind is the one that keeps working when the rota changes and nobody remembers to edit the policy. When a level's timeout passes with the incident still triggered, the next level is paged; off the end the chain repeats repeat_count times and then the team's fallback topic is paged once. The incident stays open throughout, because running out of people to wake is not somebody answering. Escalation rides the notifier's existing 30-second tick and its outbox rather than adding a second scheduler, and runs before delivery so a level that comes due on a tick is paged on that tick. Each target gets its own outbox row and therefore its own Acknowledge token: the button in a notification must acknowledge as the person holding the phone, not as whoever was paged first. Acknowledging or resolving takes the incident off the ladder. Snoozing pauses it -- a deliberate "not now" holds the ladder where it is and it resumes when the snooze runs out, rather than carrying on without the person who asked for quiet. Reminders and escalation never both run. A team with a ladder gets escalation; a team without keeps today's behaviour exactly. Both would mean two pages for one silence, which is how a tool gets muted. A level whose targets cannot be reached -- no topic, a disabled account, an empty rota -- is entered anyway, recorded as "nobody reachable", and the ladder moves on. Stalling on a rung that cannot ring would be the failure this feature exists to prevent, wearing the feature's clothes. A policy with such a level cannot be created, but an older row could hold one. The API replaces the ladder wholesale rather than patching a rung, because the levels are an order: editing one has to answer what happens to the numbering of the others, and a whole-ladder PUT makes that the client's decision and the edit atomic. Verified against a live server as well as in tests: alice paged, nobody answers, bob paged, nobody answers, the fallback topic paged once and the timeline reading "level 2: bob" then "escalation exhausted: paged terdut-oncall-all" -- and a second incident acknowledged before its timeout, which woke nobody else. No UI yet. The team-settings screens for escalation, integrations and dead man's switches are all still missing, and they are one piece of work rather than three. Claude-Session: https://claude.ai/code/session_01RHPj4ggeFdEjKKfm4SHbD7	2026-09-20 18:37:54 +02:00
Niklas Ye	b0a02c010b	Add an admin page, and move the behaviour settings into the database ... Closes #5. Three of the server's tunables were environment variables, which meant changing how long an incident waits before being paged again required editing a chart, merging it and waiting for a reconcile. They are behaviour rather than infrastructure, and the difference is who needs to change them and how often. The split is by who owns the value. What stays in the environment is where the server is plugged in: the listen address, the DSN, the ntfy URL and token, the public URL. Those are needed before the database is open and two of them are credentials -- the settings endpoint reports that ntfy is configured and that a token is set, and never what either is. What moves is how it behaves: the notify repeat interval, the stale window and the archive window. The environment variable becomes the seed rather than the setting, written once on first start and never overwritten, so a redeploy cannot put a chart's default back over an administrator's edit -- the rule the per-team dead man's switches already follow. The loops read the current value per tick, so a change at 02:00 is obeyed at 02:00. Key/value rather than a column per knob: #6 and #7 will both add settings, and a table shaped one-column-per-setting needs a migration for each. The cost is that values are text and the accessor has to say what type it wanted, which settings.go does in one place. Unknown keys are refused rather than stored -- a typo that wrote notify_repeat_second would otherwise sit in the table looking like configuration and doing nothing -- and each value has bounds loose enough to catch a slipped decimal point without having an opinion about anybody's rota. Disabling an account is new, and is not deleting one. Deleting a user nulls acknowledged_by and assigned_to, which quietly rewrites who did what during an incident months after the fact. A disabled user cannot authenticate by either credential, loses their sessions immediately, and stays the name on every acknowledgement they made. The check is part of the lookup in serveAs rather than a test afterwards, so there is no path where the row is loaded and the flag is then forgotten. The page itself is a fourth tab, shown only to an administrator and only as a courtesy: every endpoint under it is refused with 403 regardless, so somebody who types /admin gets an explanation rather than a blank screen. It lists teams with their size and open-incident count, users with their flags, and the settings with their bounds -- plus the environment half, read-only, so somebody hunting for the ntfy URL learns where it lives instead of concluding the server has none. Delete is disabled rather than offered-and-refused for a team with open incidents, and neither admin action is offered on your own account, since the server refuses both. Claude-Session: https://claude.ai/code/session_01RHPj4ggeFdEjKKfm4SHbD7	2026-09-20 18:23:46 +02:00
Niklas Ye	7c87ae2af8	Remove the unauthenticated webhook and the SQLite migration script ... Both existed to carry an upgrade across, and both upgrades are done. /api/alertmanager/webhook took no credential at all: anything able to reach the port could open an incident for anybody. v0.12.0 kept it, deprecated, so the teams release did not stop delivery while the Alertmanager config was edited, and logged a line per payload asking to be moved. The cluster's Alertmanager now posts on an integration key -- verified in the log, every two minutes, with no deprecation line since the rollout -- so the door can be shut rather than left ajar until somebody remembers. A sender still posting there gets the JSON 404 every unknown /api path gets. The tests move with it, which they should have done anyway: the harness mints an integration key for the default team and posts on that, so they exercise the path production uses rather than one only they still used. scripts/sqlite-to-postgres.go goes the same way. It was written to be temporary, it was the last thing needing modernc.org/sqlite, and this install migrated on 2026-09-20. `go mod tidy` drops the driver and its six transitive dependencies with it; the module graph is now chi, pgx, pgerrcode and x/crypto. Anyone still on v0.10.x can take the script out of the v0.12.0 tag, which the README now says. Claude-Session: https://claude.ai/code/session_01RHPj4ggeFdEjKKfm4SHbD7	2026-09-20 18:11:30 +02:00
Niklas Ye	74359c72ab	Give each team its own dead man's switches, and the UI a team to show ... The rest of #4. Two halves that belong together because they are the same sentence from opposite ends: a team decides which of its alerts are heartbeats, and the UI has to be able to say which team it is talking about. Switches were three environment variables, which made them one setting for the whole install. That was the last piece of the alerting path a team could not control: it could take its own alerts on its own key and still not say which of them were heartbeats, or how long a silence had to last. They are a row per team now, edited by an owner through PUT /api/teams/{teamID}/deadman, and the sweeper runs each team against its own matchers, timeout and severity. The environment variables become the starting point rather than the setting. Every team without a configuration is seeded from them at startup, so an upgrade keeps watching exactly what it was watching, and SeedDeadmanConfigs never overwrites -- a redeploy must not put the environment's value back over an owner's edit. A team created later watches nothing until somebody says otherwise: inheriting an install-wide heartbeat would page a new team about a source it has never heard of, and a switch nobody chose is the kind that gets muted rather than fixed. A matcher string with no alertname in it is refused at the door instead of stored. Storing it would produce a switch that watches nothing silently, which is the exact failure the feature exists to prevent. NewRouter and Sweep lose their DeadmanConfig parameter -- there is no longer one answer to hand them. The type stays, because parsing a matcher string is still parsing a matcher string. The UI side: rows in the queue carry a team badge, the filter row gains a team chip per team, and "on call now" shows one card per team. All three appear only when the viewer is in more than one team -- otherwise they are the same word repeated down a list, which is noise rather than information, and the single-team install reads exactly as it did before teams existed. Verified against a live two-team server as well as in tests: the combined queue labelled by team, the team_id filter, a heartbeat that is a heartbeat in one team and an ordinary alert in another, and a new team's switches starting empty while the upgraded team keeps the environment's. Claude-Session: https://claude.ai/code/session_01RHPj4ggeFdEjKKfm4SHbD7	2026-09-20 15:18:22 +02:00
Niklas Ye	a4fbd60441	Scope everything to a team, and route alerts by integration key ... The core of #4, and what #1 is for: terdut stops being one shared space. A team owns its incidents, alerts, schedule and integrations; a user sees exactly the teams they are in. Everything that existed moves into one Default team and every existing user becomes an owner of it, so the upgrade is a no-op for the people using it. Ingestion is the load-bearing half. An alert arrives on a team's integration key, and the key is both the credential and the routing: it says that the sender may post, and which team the alerts belong to. That also closes the unauthenticated webhook -- the old path stays for one release, deprecated and routed to the oldest team, so an upgrade does not stop delivering while somebody edits the Alertmanager config. Scoping is enforced in as few places as possible, because the failure mode is silent. serveAs loads the caller's memberships once; list queries carry `team_id = ANY(...)`; and every incident route goes through incidentIDParam, which now parses the id AND checks the team in the same call, so a new handler cannot remember the first half and forget the second. Anything in another team is 404, never 403: whether an incident exists is that team's business. Two bugs this found, both of which would have been silent: * upsertAlerts decided "is this a new occurrence" by looking up the fingerprint alone. Across teams that made team B's first alert look like a re-send of team A's, so it opened no incident at all. The lookups are keyed on (team_id, fingerprint) now, as the index is. * Every uniqueness rule was written for one tenant. Two teams watching two clusters legitimately see the same fingerprint, the same groupKey, and want somebody on call on the same day; all three constraints move to include team_id. Roles inside a team are separate from the system administrator flag: an owner configures the team, a member works its incidents, and an admin is NOT implicitly in every team -- administration is about accounts, not about reading other people's incidents. An admin can still repair a team whose owner has left, which is why requireTeamOwner lets them through. A shift can only be given to somebody in the team. Paging a person who cannot open the incident is worse than paging nobody. The UI is updated only as far as keeping it working: it loads the viewer's teams with the session and uses the first one, since nobody has a second yet. "On call now" shows every team the viewer is in, named only when there is more than one, so the common case reads exactly as before. The team switcher, badges and per-team settings pages are the next step. Breaking for API clients: the schedule endpoints moved under the team, and /api/schedule/current returns an array rather than an object or a 404. terdut-tui will need a version for that. Per-team dead-man configuration is deliberately not here. A heartbeat's incident already opens in the team whose key received it, which is the part that matters for isolation; moving the matchers out of env into per-team rows is a change to how deadman.go is configured rather than to who sees what. Claude-Session: https://claude.ai/code/session_01RHPj4ggeFdEjKKfm4SHbD7	2026-09-20 13:36:24 +02:00
Niklas Ye	1377d9005b	Add a system administrator role, and gate account management behind it ... Until now every authenticated caller could create and delete users, set anybody's password and mint anybody's API keys -- auth.go said so in a comment. Defensible with one operator and a hand-made account; not once people sign themselves up (#7), and not in a multi-tenant install (#4), where the user list is no longer everybody who works here. users.is_admin is the flag. AdminOnly gates creating and deleting users and granting the flag itself. The endpoints that are self-service for your own account and administration for somebody else's -- password, ntfy topic, API keys -- go through requireSelfOrAdmin instead, because which rule applies depends on the {id} in the path rather than on the route. Minting your own API key stays self-service. A key carries exactly the rights of the user it belongs to, so issuing one is no more than signing in again; requiring an admin for it would mean a responder cannot set up the TUI without somebody else in the room. /api/users stays readable by everybody. The queue's assignment control and the on-call schedule both have to name people, and hiding the roster from the people on it buys nothing. THE MIGRATION MAKES EVERY EXISTING USER AN ADMINISTRATOR. They already hold these powers, so nobody's access changes on upgrade: it names what is already true and leaves demotion as a deliberate act. Promoting only user 1 would silently strip the others, and could leave an install whose only administrator is an account nobody has a password for. Two guards keep an install administrable: the last administrator can be neither deleted nor demoted, and nobody can delete or demote themselves -- the likelier accident, where the only admin clears their own flag while tidying up and locks the door behind them. No UI changes: there are no account-management screens yet. models.User carries is_admin (not omitempty, so a client can tell false from an old server), which is what #5's admin page will render from.	2026-09-20 13:20:04 +02:00
Niklas Ye	dc3879eca6	Serve a web UI for the incident queue, built for phones ... Whoever is on call gets paged on a phone, and until now the only ways to act on a page were the notification's Acknowledge button or a terminal. Tapping the notification itself opened /api/incidents/{id}, which a browser can only answer with a 401 in JSON. The server now serves a web UI at / covering the incident queue, each incident's alerts and timeline with every action on it, who is on call, the alert feed, and changing your own password. The notification link now points at /incidents/{id} in that UI. It is embedded in the binary and has no build step: plain HTML, CSS and ES modules under internal/web/static, served with an ETag per file and a CSP that allows nothing from any other origin. That is how rd-web is built. It avoids adding a node toolchain to the Dockerfile and the pipeline for a page this size, and it keeps the page on the same origin as the API, so no CORS is needed and nothing else has to be deployed. Paths without a file extension fall back to index.html, so a deep link survives a reload. An unknown path under /api/ still gets a JSON 404 rather than the page. Signing in uses a username and password, because pasting a 64-character API key into a phone at 3am is not a sign-in flow. Users have no password until one is set through PUT /api/users/{id}/password, or optionally at bootstrap. A user without a password is exactly where they were before this commit and can only use API keys. A login sets an HttpOnly, SameSite=Lax session cookie. It lasts 30 days and slides forward while in use, so an on-call phone does not sign itself out. Only the token's hash is stored, as for API keys. The cookie needs a CSRF guard where a bearer header does not, because browsers attach cookies to requests other sites make. So cookie- authenticated requests go through Go 1.25's http.CrossOriginProtection, and bearer requests do not. A request carrying an Authorization header is judged on that header alone and never falls back to the cookie. Changing a password ends every other session of that user. Changing your own requires the current password, so a phone left signed in cannot be used to take the account over. Failed logins are counted per username and per client address. Ten failures for one username in 15 minutes refuse that username for the rest of the window, even with the right password. That makes locking somebody out possible for anyone who knows their username. It was accepted because the alternative is unlimited guessing, and during a lockout the notification's Acknowledge button and API keys keep working. The address limit reads the first X-Forwarded-For hop, since behind the gateway RemoteAddr is Envoy. It is looser, because a whole office behind one NAT shares it. The Secure flag follows TERDUT_PUBLIC_URL, since TLS terminates at the gateway and the server itself only ever sees plain HTTP. The chart already defaults that variable to https://<hostname>. Schedule editing, statistics and user management stay in terdut-tui for now. The API they use is unchanged, and bearer authentication behaves exactly as before.	2026-09-19 17:48:21 +02:00
Niklas Ye	14c24f8fda	Notice when the Watchdog alert stops arriving ... Everything this server does assumes alerts arrive. If Prometheus stops evaluating, or Alertmanager cannot reach us, nothing arrives — and silence is indistinguishable from everything being fine. The cluster has shipped the alert for exactly this case all along: Watchdog is expr: vector(1), so it fires permanently and is re-sent forever, and it is worth nothing unless something downstream notices it stop. Nothing did. It arrived, opened no incident because a repeat_interval re-send is not a new occurrence, and when the monitoring stack died the sweeper quietly expired it and paged nobody. So the handling is inverted for a configurable set of alerts: receiving one opens no incident, and the absence of one does. TERDUT_DEADMAN_MATCHERS selects them as label matchers, defaulting to alertname=Watchdog. The unit of monitoring is the fingerprint rather than the alert name. Two clusters sending the same Watchdog are two independent switches, so a healthy one can never mask a dead one. Every matcher must name an alertname, which keeps the sweeper's candidate query on alerts_name_idx instead of JSON-extracting labels from every row, and leaves matching with a single implementation. A switch is dormant until its first heartbeat: a matcher nothing has ever sent opens nothing, so a fresh deploy or a restored database does not page. Resolving the incident by hand sticks, exactly as it does for an alert-backed one, so a decommissioned source is a one-time page rather than a nag; the switch re-arms only when the heartbeat comes back, and dying again is a new incident. The incident has no member alerts on purpose. Linking the heartbeat would have the settled-incident cascade close it on the very sweep that opened it, and there is no alert describing the problem anyway — the problem is that no alert arrived. What happened is on the timeline instead, and recovery is the only automatic way out. One narrow exemption in the ingest guard makes recovery possible at all. A heartbeat we declared dead is marked resolved, and the one that proves us wrong carries the unchanged startsAt of an alert that never stopped firing — so "resolution is terminal within an instance" would discard it forever and a switch could die exactly once. The exemption is scoped to resolution_source = 'deadman', which is the only resolution this server infers from silence on a timeout of its own, so nothing another writer set can be undone by a stale retry. Matched alerts are also held back from the generic staleness expiry, which would otherwise resolve a heartbeat as 'expiry' long before its own tighter deadline. The timeout points the opposite way to TERDUT_STALE_AFTER: staleness is a generous grace period around a repeat_interval you do not control, while this is a deadline you set deliberately and configure the heartbeat's route to beat. Inheriting a 4h or 12h repeat_interval gives a dead man's switch with a twelve hour fuse, so the README spells out the route the heartbeat needs.	2026-08-08 21:28:55 +02:00
Niklas Ye	17ee290d90	docs: the ack token is scoped, not single-use ... The handler never deletes the token: it stays valid until expires_at and is purged by the sweeper, so a second tap is an idempotent no-op rather than a rejection. Caught by pressing Acknowledge twice against the live server. What bounds the token is scope -- one incident, one action, one day -- not a use count.	2026-08-07 11:41:15 +02:00
Niklas Ye	bc285799d1	Page the on-call person when an incident opens ... An incident opened, got assigned to whoever held today's schedule entry, and then sat there silently until somebody thought to look. The schedule and the incident model were both built; nothing reached the person holding the pager. Notifications go out through ntfy, over plain HTTP with no new dependencies. Delivery is an outbox rather than an inline call: the pool is limited to a single connection, so a POST made while holding the webhook's transaction would stall every other request behind it. The webhook inserts a row and a notifier goroutine sends it within a tick, retrying with exponential backoff. Only opening an incident has to resolve a topic from scratch. Reminders and all-clears reuse whatever that first notification chose, which keeps configuration out of resolveIfSettled and gives the right rule for free: you only hear that something resolved if you were told it started. Each push carries an Acknowledge button, because the useful thing to do at 3am is stop the pager without unlocking anything. It POSTs to an unauthenticated /api/notify/ack/{token} — a notification body lives on the ntfy server and in the device cache, so a real API key must never appear in one. The token is minted per delivery, scoped to one incident and one action, and expires in a day. Reminders repeat until the incident stops being untouched. The stop conditions are the states that already mean somebody has it: acknowledged, snoozed, resolved, archived. Snooze is the mute button, so there is no separate reminder cap. Notifications sent to the fallback topic carry no Acknowledge button. The topic is shared, and a button on it would let any subscriber acknowledge as somebody else.	2026-08-07 08:51:38 +02:00
Niklas Ye	279ef6cf8b	Turn incoming alerts into incidents ... The alerts row was both Alertmanager's record and the human work queue, and the two have different owners. The webhook upsert rewrites that row on every notification; acknowledgement, comments and archiving were columns on it that the upsert happened not to touch. So an alert that resolved and re-fired days later still read as acknowledged by whoever acked the first occurrence — the ack outlived the thing it referred to. Nothing recorded transitions either: rows are mutated in place, so there was no timeline and no way to compute how long anything took. Alerts are now read-only signal records with two states, and incidents are the work item: triggered, acknowledged or resolved, with an assignee, a snooze, notes and an append-only timeline. Many alerts map to one incident, and a new occurrence opens a new incident, which is what makes a stale ack impossible rather than merely unlikely. Correlation uses Alertmanager's own groupKey. It already grouped the alerts according to the group_by routing tree the operator configured and sends the result on every webhook, where it was being discarded; adopting it means changing group_by in alertmanager.yml changes correlation here, with no second grouping scheme to configure and keep in sync. An incident opens only when an alert transitions into firing — an unseen fingerprint, a newer startsAt, or a resolved alert starting again. The unchanged notifications Alertmanager re-sends every repeat_interval are none of those. That rule is what lets manual resolution be terminal: without it, closing an incident by hand would be undone by the next re-send of an alert that never stopped firing, and the button would be a lie. Snooze covers the "not now" case instead. Incidents otherwise resolve by cascade, once every alert under them has stopped firing, whether by webhook or by expiry. New incidents are assigned to whoever holds today's schedule entry. The schedule table has existed since the first release with nothing reading it. Also here, following from the split: - Incident severity is a high-water mark over its alerts, never lowered. An incident that hit critical was a critical incident, and downgrading a live one would demote it in the queue while the work is still open. - /api/stats/incidents reports MTTA and MTTR, null rather than zero until there is something to average. Neither was computable before. - Alert archiving becomes sweeper-only housekeeping; the archive people interact with is the incident's. Breaking: the alert acknowledge, archive and comment endpoints are gone, and the alert object drops the acknowledgement fields and gains incident_id. The README maps each removed endpoint to its replacement. Migration 008 backfills an incident per existing alert, archived ones included so no comment is orphaned, carrying acknowledgements across and turning comments into timeline notes. Both documented alert contracts are untouched: received_at still advances on every accepted payload, re-sends included, and resolution_source still says how much to trust ends_at. The upsert is byte-for-byte what it was, now running inside the ingest transaction.	2026-07-30 17:02:13 +02:00
Niklas Ye	debc4bf78c	Add alert archiving ... Alerts can be manually archived (POST /api/alerts/{id}/archive) or unarchived (DELETE /api/alerts/{id}/archive). A background goroutine auto-archives resolved alerts older than TERDUT_ARCHIVE_AFTER (default 7d). GET /api/alerts hides archived alerts by default; ?archived=true shows them.	2026-05-22 13:22:45 +02:00
Niklas Ye	923fc8bf9c	Stage 6: alert statistics endpoints ... - GET /api/stats/alerts — total/firing/resolved counts - GET /api/stats/alerts/top — most frequent alert names (?limit, default 10) - GET /api/stats/alerts/by-hour — counts for all 24 hours (zeros filled in) - GET /api/stats/alerts/by-day — counts for all 7 days with names (zeros filled in) - All endpoints accept optional ?from/?to (YYYY-MM-DD) to filter by received_at	2026-05-20 22:31:18 +02:00
Niklas Ye	f8f209dcba	Stage 5: on-call schedule ... - Migration 005: schedule_entries table (date TEXT UNIQUE, one person per day) - POST /api/schedule — assign user to one or more dates in a single transaction; any date conflict rejects the whole request (409) - GET /api/schedule — list all entries ordered by date, optional ?from/?to - GET /api/schedule/current — today's on-call user (UTC date), 404 if none - DELETE /api/schedule/{id} — remove an entry (204)	2026-05-20 22:28:23 +02:00
Niklas Ye	c3348a410a	Stage 4: alert acknowledgement and comments ... - Migration 004: acknowledged_by/acknowledged_at columns on alerts, alert_comments table (FK cascade on delete) - POST /api/alerts/{id}/acknowledge — stamps authed user + timestamp, returns updated alert with acknowledged_by username - DELETE /api/alerts/{id}/acknowledge — clears ack (204) - GET /api/alerts/{id}/comments — list in chronological order - POST /api/alerts/{id}/comments — add comment (returns 201) - DELETE /api/alerts/{id}/comments/{commentID} — own comments only (204) - All alert queries now LEFT JOIN users for ack username	2026-05-20 21:57:11 +02:00
Niklas Ye	9b4ca1482f	Stage 3: Alertmanager webhook ingestion and alert query API ... - Migration 003: alerts table with fingerprint UNIQUE, JSON label/annotation columns, nullable ends_at, and indexed status/name/received_at - POST /api/alertmanager/webhook — upserts each alert by fingerprint; zero endsAt ("0001-01-01") stored as NULL (still firing) - GET /api/alerts — filtered list (?status, ?name, ?from, ?to, ?limit) - GET /api/alerts/{id} — single alert lookup	2026-05-20 21:54:17 +02:00
Niklas Ye	7c3c28b23c	Stage 2: users, API key auth, bootstrap endpoint ... - Migration 002: users and api_keys tables (Unix timestamps, FK cascade) - POST /api/bootstrap — creates first user + key when DB is empty - POST/GET/DELETE /api/users — user CRUD - POST/DELETE /api/users/{id}/api-keys — key issuance and revocation - AuthMiddleware: SHA-256 bearer token lookup, last_used_at tracking - Raw key returned once on creation; only SHA-256 hash stored	2026-05-20 21:51:09 +02:00
Niklas Ye	0387e1e017	Stage 1: project skeleton, SQLite, migrations, HTTP server ... - go mod init with chi, modernc.org/sqlite, golang.org/x/crypto - Custom embedded migration runner (no CGO dependency) - Config from TERDUT_ADDR / TERDUT_DB_PATH env vars - chi router with /healthz endpoint - Graceful shutdown on SIGINT/SIGTERM	2026-05-20 21:41:40 +02:00