8b2789b9b2ad252ca71a67a90a41d15bf4a42568
104 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
|
|
cc31c993dd |
Take the database password from PGPASSWORD, not the DSN
The chart asked for a whole DSN in a Secret. Nothing writes one: the Zalando postgres operator generates a Secret with `username` and `password` keys and no connection string, so wiring the wrapper chart up would have meant hand-maintaining a second copy of a password the operator owns and rotates on a from-scratch rebuild -- which is charts#176 again, the issue miniflux closed by doing the opposite. So the DSN becomes a plain value with no password in it, and the password arrives as PGPASSWORD from a Secret. pgx fills in from libpq's PG* environment variables whatever the DSN omits, exactly as miniflux's lib/pq does. Verified rather than assumed, against a real server: a password-less DSN connects with PGPASSWORD set, and fails with `password authentication failed` when it is wrong, so the variable is doing the work rather than being quietly ignored. It also keeps the credential out of the rendered manifest and out of `kubectl describe pod`, which a DSN-with-password does not. |
||
|
|
21f0eec807 |
Merge pull request 'Move the database to Postgres' (#8) from postgres into main
Reviewed-on: #8 |
||
|
|
dc39e3a5d3 |
Move the database to Postgres, before teams need the schema
First step of #1, and it goes first for one reason: #4 adds a team_id to nearly every table, and doing that twice -- once for SQLite, once for Postgres -- is work nobody gets paid for. The teams migrations now only have to be written against one database. The ten SQLite migrations are replaced by a single Postgres baseline rather than ported one by one. They were incremental in a way that has no value on a fresh install: 004 adds columns 008 drops again, and 008's backfill rewrites data a Postgres database never had. The history stays in git; the schema they add up to is now 001_baseline.sql. Timestamps stay BIGINT unix seconds and are NOT converted to timestamptz. Everything in Go already speaks epochs, so converting would have been a second, larger change riding along inside this one. It is worth doing on its own. The JSON columns did move to jsonb, because #4 will want to filter and index on labels. Most of the port is mechanical -- 170 placeholders from ? to $1 -- but four things needed more than a search and replace: * Dynamically built WHERE clauses cannot keep their numbering straight by hand, so they hand out placeholders through sqlArgs instead. A filter can now be added or reordered without renumbering anything. * SUM(resolved_at IS NULL) was SQLite counting a boolean as 0 or 1. Postgres has no sum(boolean), and this was breaking every dead man's switch -- silently, since the sweeper only logs. Now COUNT(*) FILTER. * unixepoch() became FLOOR(EXTRACT(EPOCH FROM now()))::bigint. The FLOOR is load-bearing: a bare cast rounds half up, so a row written at .6 of a second claimed a timestamp a second in the future and disagreed with the time.Now().Unix() the Go side stamps. * The unique-violation check matched SQLite's error text. It matches SQLSTATE 23505 now, so a renamed constraint cannot turn a 409 back into a 500. Tests need a real Postgres, because there is no in-memory Postgres the way there was an in-memory SQLite. Each test gets its own schema on a shared server -- cheaper than a database each, and still isolated. TERDUT_TEST_DSN says where it is; `make test-db` starts one locally and ci.yaml runs one as a service container. An unset DSN fails the suite rather than skipping it: a run that quietly tests nothing is worse than one that does not run. TestMigration_BackfillCarriesAckAndComments is deleted along with the migrations it replayed. What it protected -- an upgrade not losing acknowledgements and comments -- now belongs to scripts/sqlite-to-postgres.go, which is build-tagged so the SQLite driver stays out of the server binary. Both are meant to be deleted once this install has migrated. The chart loses the PVC, the data volume and the python backup sidecar, and requires database.dsnSecret.name: it provisions no database and cannot guess where the credentials live, so a render without it is meant to fail. Backups move to where Postgres actually runs. The other half of that -- the postgresql CR, the k8up pg_dump annotation and the network policy -- is a change to the wrapper chart in Ryuvia/charts and is not in here. Verified rather than assumed: the gate is green with -race against Postgres 17, govulncheck and gitleaks are clean, and the migration script was run end to end against a SQLite database built at the old schema and seeded in every table. Ids survive, so incidents keep their numbers and every foreign key still points where it did; the identity sequences are moved past the copied ids, and a webhook after the migration opened incident 12 rather than colliding at 1. |
||
|
|
989425e550 |
Set the chart's placeholder version to 0.10.2
CI / chart (push) Successful in 1s
CI / security (push) Successful in 28s
CI / test (push) Successful in 2m13s
Release / test (push) Successful in 1m9s
Release / chart (push) Successful in 2s
Release / binaries (push) Successful in 1m41s
Release / image (push) Successful in 1m42s
Release / scan-image (push) Successful in 2s
Cosmetic, as inv0.10.2 |
||
|
|
5b1ab2c568 |
Move x/crypto to v0.55.0, clear of the ssh CVEs
v0.10.1's image published, but scan-image refused it. trivy reports ten
HIGH advisories against golang.org/x/crypto v0.49.0, CVE-2026-39828
through CVE-2026-56854, all of them in x/crypto/ssh and its agent and
knownhosts packages. The last is fixed in 0.55.0 and the rest in 0.52.0.
None of them is reachable here. The server imports x/crypto/bcrypt and
nothing else from the module, and the image is FROM scratch, holding a
single binary. But trivy scans at module granularity and cannot tell
that. Shipping a known-vulnerable module version on the strength of a
reachability argument is not the call to make inside a dependency pin,
so the version moves instead.
|
||
|
|
e78f49461a |
Set the chart's placeholder version to 0.10.1
CI / chart (push) Successful in 1s
CI / security (push) Successful in 27s
CI / test (push) Successful in 2m11s
Release / test (push) Successful in 1m11s
Release / chart (push) Successful in 3s
Release / binaries (push) Successful in 1m32s
Release / image (push) Successful in 1m41s
Release / scan-image (push) Failing after 29s
Cosmetic, as inv0.10.1 |
||
|
|
9abf07f2cb |
Build on Go 1.25 again, as the Dockerfile does
v0.10.0 never produced an image. Adding bcrypt in
|
||
|
|
4cec26edde |
Set the chart's placeholder version to 0.10.0
CI / chart (push) Successful in 2s
CI / security (push) Successful in 40s
CI / test (push) Successful in 2m47s
Release / test (push) Successful in 1m9s
Release / chart (push) Successful in 2s
Release / image (push) Failing after 42s
Release / scan-image (push) Has been skipped
Release / binaries (push) Successful in 1m57s
Cosmetic, as inv0.10.0 |
||
|
|
dc3879eca6 |
Serve a web UI for the incident queue, built for phones
Whoever is on call gets paged on a phone, and until now the only ways to
act on a page were the notification's Acknowledge button or a terminal.
Tapping the notification itself opened /api/incidents/{id}, which a
browser can only answer with a 401 in JSON. The server now serves a web
UI at / covering the incident queue, each incident's alerts and timeline
with every action on it, who is on call, the alert feed, and changing
your own password. The notification link now points at /incidents/{id}
in that UI.
It is embedded in the binary and has no build step: plain HTML, CSS and
ES modules under internal/web/static, served with an ETag per file and a
CSP that allows nothing from any other origin. That is how rd-web is
built. It avoids adding a node toolchain to the Dockerfile and the
pipeline for a page this size, and it keeps the page on the same origin
as the API, so no CORS is needed and nothing else has to be deployed.
Paths without a file extension fall back to index.html, so a deep link
survives a reload. An unknown path under /api/ still gets a JSON 404
rather than the page.
Signing in uses a username and password, because pasting a 64-character
API key into a phone at 3am is not a sign-in flow. Users have no
password until one is set through PUT /api/users/{id}/password, or
optionally at bootstrap. A user without a password is exactly where they
were before this commit and can only use API keys. A login sets an
HttpOnly, SameSite=Lax session cookie. It lasts 30 days and slides
forward while in use, so an on-call phone does not sign itself out.
Only the token's hash is stored, as for API keys.
The cookie needs a CSRF guard where a bearer header does not, because
browsers attach cookies to requests other sites make. So cookie-
authenticated requests go through Go 1.25's http.CrossOriginProtection,
and bearer requests do not. A request carrying an Authorization header
is judged on that header alone and never falls back to the cookie.
Changing a password ends every other session of that user. Changing
your own requires the current password, so a phone left signed in
cannot be used to take the account over.
Failed logins are counted per username and per client address. Ten
failures for one username in 15 minutes refuse that username for the
rest of the window, even with the right password. That makes locking
somebody out possible for anyone who knows their username. It was
accepted because the alternative is unlimited guessing, and during a
lockout the notification's Acknowledge button and API keys keep
working. The address limit reads the first X-Forwarded-For hop, since
behind the gateway RemoteAddr is Envoy. It is looser, because a whole
office behind one NAT shares it.
The Secure flag follows TERDUT_PUBLIC_URL, since TLS terminates at the
gateway and the server itself only ever sees plain HTTP. The chart
already defaults that variable to https://<hostname>.
Schedule editing, statistics and user management stay in terdut-tui for
now. The API they use is unchanged, and bearer authentication behaves
exactly as before.
|
||
|
|
9669b8f477 |
Set the chart's placeholder version to 0.9.4
CI / chart (push) Successful in 0s
CI / security (push) Successful in 24s
CI / test (push) Successful in 28s
Release / test (push) Successful in 28s
Release / chart (push) Successful in 1s
Release / binaries (push) Successful in 28s
Release / image (push) Successful in 1m11s
Release / scan-image (push) Successful in 23s
Cosmetic, as in
v0.9.4
|
||
|
|
a7871ed7c6 |
Stop the bootstrap hook installing curl at run time
The hook's container was alpine:3 and its first line was `apk add --no-cache curl`. That writes the binary into the container's writable upper layer, and every exec of it afterwards is, correctly, a dropped binary: Falco's `Drop and execute new binary in container` (PCI_DSS_11.5.1, MITRE TA0003) fired twice at Critical on the upgrade to chart 0.9.3, 65ms after the container started, with evt.arg.flags=EXE_WRITABLE|EXE_UPPER_LAYER. Ryuvia/charts#100 has the event lines. A true positive of the rule and a false positive of intent, and it is not a one-off: the hook is post-install,post-upgrade, so it recurred on every release. The cluster is still in the Falco burn-in with detections routed to a null receiver, which is the only reason nobody was paged for it. Fixed here rather than with a Falco exception on purpose. An exception would have to name this container and would then stay in the rule set forever, blinding it for the one workload that already runs as root with create-secret RBAC, and it would leave the second problem untouched: this runs as a post-upgrade hook, a failed hook fails the release, so every `helm upgrade` of terdut-server depended on dl-cdn.alpinelinux.org answering. That dependency is now gone. alpine/curl is still a full Alpine, so sh, cat, sleep, grep, cut, head and tail are all present -- verified in-cluster before the swap rather than assumed, since a missing utility would surface as a failed post-upgrade hook and not as anything visible here. Digest-pinned, as the wrapper chart's own sidecar images are. The image declares an ENTRYPOINT, which the Job's `command:` overrides; a comment says so, because rewriting that to `args:` would silently run curl's entrypoint instead of the script. No change to the script's logic, to the RBAC, or to when the hook runs. Nothing on the terdut-tui side of the API moves, and no terdut-tui version is required or excluded by this. Worth recording while it is in view, and deliberately not acted on here: there is no terdut-server-admin-key secret in the namespace, so the POST returns 403, the hook logs "Server already bootstrapped, nothing to do" and exits before the secret-creating branch. On an upgrade this hook currently achieves nothing at all. Narrowing it to post-install would remove the detection outright, but that changes what the hook is for and belongs in its own change. Claude-Session: https://claude.ai/code/session_014m2pJdpCTv3mvvUUuBM54Y |
||
|
|
79f5db2636 |
Scan the source and the working tree too, not just the image
The image scan added yesterday reads the built artifact. It cannot see a vulnerable dependency the binary never calls into, and it cannot see a credential in a file that never reaches the image — this one is FROM scratch and contains a single binary, so almost nothing in the repo is in it. Those are two different questions and they need two different tools, which is why riksdata and rd-web have run govulncheck and gitleaks all along. Both run on every push and pull request rather than only on a tag, since neither needs anything published. Checked by hand before wiring in, as with the image scan. govulncheck reports no vulnerabilities the code can reach, and gitleaks finds nothing in the tree. What govulncheck does report is worth writing down, because it is the argument for having it. It found three advisories in chi and reports none of them, all three being IP spoofing in middleware.RealIP, which router.go does not use — it uses Logger and Recoverer. The analysis is symbol-level rather than dependency-level, so adding middleware.RealIP would turn this red on the next push. That is precisely when someone should be made to look, and it is a plausible thing to reach for here, since the API sits behind a gateway and real client addresses are exactly what RealIP is for. The fourth finding is an integer overflow in golang.org/x/sys/windows, which a linux/scratch image will not be calling. Both gates were checked for the failure direction as well. gitleaks exits 1 on a private key block. Worth knowing when testing it: it allowlists well-known example credentials, so the AWS key from Amazon's own documentation does not trip it and proves nothing. Neither reads git history. gitleaks runs with --no-git, which scans the working tree, so it stops a secret on the way in and says nothing about what is already committed. Claude-Session: https://claude.ai/code/session_01S7R4gWTz5wh5xCY4nCSJjN |
||
|
|
84146fc903 |
Scan the published image for known vulnerabilities
terdut-server was the only one of the three release-managed repos with no image scanning at all. riksdata and rd-web have had a scan-image job since they were set up; everything published here up to and including v0.9.3 went out without a CVE check. It scans the pushed image rather than a locally built one, for the same reason the siblings do: trivy cannot read a local image on this runner, since Talos has no docker socket and the dind sidecar shares no filesystem with the job. So it runs after image rather than gating it, and a red scan unpublishes nothing. What it means is narrower and worth stating plainly: do not bump the wrapper chart in Ryuvia/charts to that version. Checked before wiring it in rather than after. v0.9.3 scans clean at HIGH,CRITICAL with unfixed findings ignored, so this does not turn the pipeline red on arrival, and the same command exits 1 on an image that does have findings — a gate that cannot fail is not a gate. One platform is scanned, not both. The image is FROM scratch, so there are no OS packages and trivy sees a single target: the Go binary and its module graph. linux/amd64 and linux/arm64 are that same module set built for a different GOARCH, so a finding in one is a finding in both. On an image with a base layer that reasoning would not hold. Still no govulncheck and no gitleaks here, which riksdata and rd-web run in a separate CI job. This is the only security scanning terdut-server has. Claude-Session: https://claude.ai/code/session_01S7R4gWTz5wh5xCY4nCSJjN |
||
|
|
c6f1fe317e |
Correct the docs that said this repo has no publishing targets
Both CLAUDE.md and README.md claimed there were deliberately no build or
push targets because the workflow owned publishing. That stopped being true
in
|
||
|
|
f46e5f5729 |
Set the chart's placeholder version to 0.9.3
Cosmetic, and done anyway, for the same reason asv0.9.3 |
||
|
|
69fcc24a4d |
Drive the pipeline through make, the way riksdata and rd-web do
Both workflows restated the build in YAML: gofmt, go vet and go test inline
in two places, buildx inline in a third, and the chart's version sed'd into
Chart.yaml before packaging. The Makefile added in
|
||
|
|
6a4f902e38 |
Declare that this repo's release prose is English
CI / test (push) Successful in 5s
The release skill's house style opened with "everything here is written in Swedish", stated as a house default. It is not one: Swedish belongs to riksdata and rd-web because their interface copy and the riksdag data they present are inseparable from it. Nothing about an on-call tool is — the labels, the API and the data here are English, and so are the code comments and the docs. Stated as a rule that was remembered rather than read, it produced the whole of v0.9.2 in Swedish: four commits, the tag body and both charts PRs. The PRs were rewritten; the commits on main and the published tag could not be, so v0.9.2 stays as the record of the mistake. PROSE_LANG makes it a declaration each repo carries, which release-preflight prints on the prose line before any of the four texts get written. See reference/house-style.md in the skill. Claude-Session: https://claude.ai/code/session_01S7R4gWTz5wh5xCY4nCSJjN |
||
|
|
5f9c202d65 |
Beskriv hur ett släpp går till, och vad som är särskilt här
CI / test (push) Successful in 6s
Repot kom in under den gemensamma släppprocessen i |
||
|
|
477454ec3c |
Sätt chartets platshållarversion till 0.9.2
Kosmetiskt, och görs ändå. .gitea/workflows/release.yaml stämplar både
version och appVersion från git-taggen när det publicerar (
v0.9.2
|
||
|
|
10812606bf |
Lägg terdut-server under den gemensamma släppprocessen
Släppprocessen (~/.claude/skills/release) körde hittills bara riksdata och
rd-web, och vägrade den här katalogen med "not one of the release-managed
repos". Den kräver två saker: en .release.conf och ett release-vars-mål som
skriver ut IMAGE, HELM_CHART och HELM_REPO. Poängen med att fråga make i
stället för att upprepa värdena i processen är att de bara kan ha en
definition, så det som taggas, det som pushas och det som wrappern pinnar
inte kan glida isär.
Makefilen är avsiktligt inte en kopia av riksdatas. Två skillnader:
fmt, lint och test speglar .gitea/workflows/ci.yaml steg för steg, så ett
grönt "make fmt lint test" här betyder samma sak som en grön CI. gofmt-målet
är kopierat ordagrant och inte förenklat, eftersom gofmts två felsätt inte
är lika: en felformaterad fil listas på stdout med exit 0, medan en fil som
inte går att parsa ger tom stdout och exit 2 — och den naiva varianten läser
det andra som framgång (
|
||
|
|
94dec19976 |
Räkna en tom incidentlista som noll i stället för som ett fel
SUM över noll rader är NULL i SQLite, inte 0. handleStatsIncidents läste de tre statusräknarna rakt in i int64, så i samma stund som filtret inte matchade någon rad föll skanningen på "converting NULL to int64 is unsupported" och hela /api/stats/incidents svarade 500. COUNT(*) ger däremot 0 utan knot, vilket är precis varför felet inte syns förrän tabellen töms — det är det enda uttrycket i satsen som klarar noll rader. Filtret är alltid på: statsFilter lägger på archived_at IS NULL ( |
||
|
|
9046f6e026 |
ci: fail on code that is not gofmt'd
CI / test (push) Successful in 4s
go vet says nothing about import order, so when the move to git.ryuvia.com rewrote every import path without re-sorting them -- the new path sorts before github.com/..., where the old one sorted after -- both repos went through a green CI run and a release unformatted. Added to the release workflow as well as CI, so the two keep running the same checks; ci.yaml's header claims exactly that, and a check in one but not the other would quietly make it false. The step handles gofmt's two failure modes separately because they do not look alike: a misformatted file is listed on stdout with exit 0, so the failure has to be raised by hand, while a file that does not parse prints nothing to stdout and exits 2 -- which a plain emptiness test reads as success. Verified against all three cases (clean, misformatted, unparseable) before committing. |
||
|
|
03504b61be |
gofmt: restore import grouping after the module rename
CI / test (push) Successful in 5s
The rename to git.ryuvia.com/niklas/... was a plain string substitution, so it left the import blocks in their old order. The new path sorts before github.com/go-chi/..., where the old one sorted after, which gofmt considers unformatted. go vet does not look at import order, so CI had nothing to say about it. |
||
|
|
6047d1a9f7 |
ci: do not override HELM_REPOSITORY_CACHE alongside the config
Helm writes a refreshed repository index to the default cache directory and then looks for it in the overridden one, so setting both is how the chart tooling breaks on this machine already.v0.9.1 |
||
|
|
289eca8076 |
Move to Gitea: git.ryuvia.com/niklas/terdut-server
CI / test (push) Successful in 2m15s
The module path, the container image, the Helm chart and the CI pipeline all named GitHub. They now name the Gitea instance everything else already runs on. The workflows are rewritten rather than translated. Gitea's runner image is ubuntu:22.04, whose nodejs is Node 12, so no JS action runs there at all -- actions/checkout@v4 dies with a SyntaxError before it does anything. Every step is shell, checkout is a plain clone (this repo is public, so it needs no credential), and the jobs that need docker or helm run in host mode because the dind bridge a `container:` job gets cannot reach github.com or get.helm.sh. Two consequences worth naming: - upload-artifact/download-artifact are also JS actions, and there is no artifact store here, so the job that builds the binaries is the job that publishes them. Nothing is passed between jobs. - setup-qemu-action is gone with the rest, and the runner has no binfmt registration. The Dockerfile's builder stage now runs on $BUILDPLATFORM and cross-compiles from TARGETARCH instead, which is what keeps the arm64 image buildable -- and makes it native rather than emulated. The chart moves from a GitHub Pages index to an OCI artifact in Gitea's registry. Publishing stays tag-only for the reason recorded in release.yaml: a workflow triggered by the branch push cannot know the version it is about to be tagged with. The GitHub repository is left in place and untouched. Nothing pushes to it any more, but its existing release downloads and chart index keep resolving. |
||
|
|
766f43931c |
chart: publish from the tag only, not from both workflows
Two workflows published the chart and disagreed about its metadata. release.yml stamps version and appVersion from the git tag; chart-release.yml, triggered by any charts/** push to main, took Chart.yaml verbatim, where appVersion is the hardcoded "latest". Both fired for the same commit, both tried to publish the same chart version, and skip_existing turned whichever lost into a no-op — so what a release said about itself came down to which runner was quicker. Chart 0.9.0 went out that way, reading appVersion "latest". Every earlier release got the right answer by accident: Chart.yaml's version lagged the published set, so chart-release.yml always collided with an existing version and skipped, leaving release.yml to win uncontested. Bumping Chart.yaml to match the tag before cutting 0.9.0 removed that accident and the race showed itself. Making the two agree is not possible. The tag is pushed after the branch, so a workflow triggered by the main push cannot know the version it is about to be tagged with — no amount of deriving from git describe fixes that ordering. The fix is one publisher, triggered by the tag, so chart-release.yml is deleted. The chart now only ships with an app release. Nothing is lost: the sed in release.yml ties the chart version to the app version, so a chart-only change never had a version of its own to be released under. Chart fixes ride the next tag. Chart.yaml's version and appVersion are documented as the placeholders they now are, so the next person does not helpfully bump them and reintroduce this. skip_existing stays, for idempotent re-runs of a failed release rather than for the race, and a non-version tag now fails the job instead of silently publishing unstamped metadata. |
||
|
|
14c24f8fda |
Notice when the Watchdog alert stops arriving
Release / release (push) Has been skipped
Release / build (amd64, linux) (push) Has been skipped
Release / build (arm64, darwin) (push) Has been skipped
Release / test (push) Failing after 5s
Release / build (arm64, linux) (push) Has been skipped
Release / docker (push) Has been skipped
Release / chart (push) Has been skipped
Release / build (amd64, darwin) (push) Has been skipped
Everything this server does assumes alerts arrive. If Prometheus stops evaluating, or Alertmanager cannot reach us, nothing arrives — and silence is indistinguishable from everything being fine. The cluster has shipped the alert for exactly this case all along: Watchdog is expr: vector(1), so it fires permanently and is re-sent forever, and it is worth nothing unless something downstream notices it stop. Nothing did. It arrived, opened no incident because a repeat_interval re-send is not a new occurrence, and when the monitoring stack died the sweeper quietly expired it and paged nobody. So the handling is inverted for a configurable set of alerts: receiving one opens no incident, and the absence of one does. TERDUT_DEADMAN_MATCHERS selects them as label matchers, defaulting to alertname=Watchdog. The unit of monitoring is the fingerprint rather than the alert name. Two clusters sending the same Watchdog are two independent switches, so a healthy one can never mask a dead one. Every matcher must name an alertname, which keeps the sweeper's candidate query on alerts_name_idx instead of JSON-extracting labels from every row, and leaves matching with a single implementation. A switch is dormant until its first heartbeat: a matcher nothing has ever sent opens nothing, so a fresh deploy or a restored database does not page. Resolving the incident by hand sticks, exactly as it does for an alert-backed one, so a decommissioned source is a one-time page rather than a nag; the switch re-arms only when the heartbeat comes back, and dying again is a new incident. The incident has no member alerts on purpose. Linking the heartbeat would have the settled-incident cascade close it on the very sweep that opened it, and there is no alert describing the problem anyway — the problem is that no alert arrived. What happened is on the timeline instead, and recovery is the only automatic way out. One narrow exemption in the ingest guard makes recovery possible at all. A heartbeat we declared dead is marked resolved, and the one that proves us wrong carries the unchanged startsAt of an alert that never stopped firing — so "resolution is terminal within an instance" would discard it forever and a switch could die exactly once. The exemption is scoped to resolution_source = 'deadman', which is the only resolution this server infers from silence on a timeout of its own, so nothing another writer set can be undone by a stale retry. Matched alerts are also held back from the generic staleness expiry, which would otherwise resolve a heartbeat as 'expiry' long before its own tighter deadline. The timeout points the opposite way to TERDUT_STALE_AFTER: staleness is a generous grace period around a repeat_interval you do not control, while this is a deadline you set deliberately and configure the heartbeat's route to beat. Inheriting a 4h or 12h repeat_interval gives a dead man's switch with a twelve hour fuse, so the README spells out the route the heartbeat needs.v0.9.0 |
||
|
|
e5916d522a |
Let an on-call day be handed to somebody else
Release / build (amd64, darwin) (push) Has been skipped
Release / build (arm64, darwin) (push) Has been skipped
Release / docker (push) Has been skipped
Release / release (push) Has been skipped
Release / test (push) Failing after 6s
Release / build (amd64, linux) (push) Has been skipped
Release / build (arm64, linux) (push) Has been skipped
Release / chart (push) Has been skipped
A date is held by exactly one person and POST /api/schedule plain-inserts, so any date that was already taken came back 409. That made reassignment impossible through the API: the only route was to delete the entry first, and for a week that meant seven separate deletions. Worse, the reject is all-or-nothing across the request, so assigning a week where a single day happened to be taken failed entirely and placed none of the other six. The refusal itself is worth keeping. Moving a shift off the person expecting to be paged for it should not be something a plain call does by accident, so the fix is to make it possible to ask for rather than to remove the guard: "replace": true takes the dates anyway, and the flag defaults to off so every existing caller behaves exactly as before. The delete and the insert share the transaction that was already there. That matters more than the flag does — a week of free and taken days now lands as a unit, and a failure part way through leaves the rota as it was instead of with a shift deleted and nothing put back. A rota with a hole in it is worse than a rota that refused to change. One consequence worth naming: under replace a date repeated inside one request is idempotent rather than a conflict, because the second pass clears what the first wrote.v0.8.0 |
||
|
|
4224dbe96c |
Record notification delivery on the incident timeline
Release / test (push) Failing after 8s
Release / build (amd64, darwin) (push) Has been skipped
Release / build (amd64, linux) (push) Has been skipped
Release / build (arm64, darwin) (push) Has been skipped
Release / build (arm64, linux) (push) Has been skipped
Release / docker (push) Has been skipped
Release / chart (push) Has been skipped
Release / release (push) Has been skipped
An incident's history went quiet after "Incident opened": nothing said that anybody had been paged, reminded, or told it resolved. Delivery lived only in the notifications outbox, which no API exposes, so when a page failed to arrive there was nothing in the product that said whether it had been sent. The notifier now writes two event types. A notified event once ntfy accepts the publish, carrying the kind in detail and the paged user in user_id — absent when the page went to the shared fallback topic, which belongs to nobody. And a notify_failed event when a notification exhausts its retries, which is the one worth having: without it a page that never landed leaves the timeline identical to one that did. Both are written from the delivery result rather than at enqueue. A queued notification is an intention, and the timeline is append-only, so claiming somebody was told before ntfy accepted it would be a lie that stays there. A failed timeline write is logged rather than returned, so it cannot make a delivered row look unsent and send the page twice. The topic is deliberately in neither: it is a shared secret with the ntfy server, and every API key can read the timeline. No migration — incident_events.type is free text, unlike notifications.kind.v0.7.0 |
||
|
|
17ee290d90 |
docs: the ack token is scoped, not single-use
The handler never deletes the token: it stays valid until expires_at and is purged by the sweeper, so a second tap is an idempotent no-op rather than a rejection. Caught by pressing Acknowledge twice against the live server. What bounds the token is scope -- one incident, one action, one day -- not a use count. |
||
|
|
7caafbaf80 |
chart: back up the database through a python sidecar
Release / test (push) Failing after 7s
Release / build (amd64, darwin) (push) Has been skipped
Release / build (amd64, linux) (push) Has been skipped
Release / build (arm64, darwin) (push) Has been skipped
Release / build (arm64, linux) (push) Has been skipped
Release / docker (push) Has been skipped
Release / chart (push) Has been skipped
Release / release (push) Has been skipped
The image is FROM scratch, so there is no interpreter to run a k8up backupcommand in, and the database runs in WAL mode, where a file-level copy of the volume is not crash-consistent. Also switches to strategy: Recreate. The data PVC is ReadWriteOnce, so a RollingUpdate deadlocks the new pod against the old one holding it.terdut-server-0.6.0 v0.6.0 |
||
|
|
bc285799d1 |
Page the on-call person when an incident opens
An incident opened, got assigned to whoever held today's schedule entry,
and then sat there silently until somebody thought to look. The schedule
and the incident model were both built; nothing reached the person
holding the pager.
Notifications go out through ntfy, over plain HTTP with no new
dependencies. Delivery is an outbox rather than an inline call: the pool
is limited to a single connection, so a POST made while holding the
webhook's transaction would stall every other request behind it. The
webhook inserts a row and a notifier goroutine sends it within a tick,
retrying with exponential backoff.
Only opening an incident has to resolve a topic from scratch. Reminders
and all-clears reuse whatever that first notification chose, which keeps
configuration out of resolveIfSettled and gives the right rule for free:
you only hear that something resolved if you were told it started.
Each push carries an Acknowledge button, because the useful thing to do
at 3am is stop the pager without unlocking anything. It POSTs to an
unauthenticated /api/notify/ack/{token} — a notification body lives on
the ntfy server and in the device cache, so a real API key must never
appear in one. The token is minted per delivery, scoped to one incident
and one action, and expires in a day.
Reminders repeat until the incident stops being untouched. The stop
conditions are the states that already mean somebody has it: acknowledged,
snoozed, resolved, archived. Snooze is the mute button, so there is no
separate reminder cap.
Notifications sent to the fallback topic carry no Acknowledge button. The
topic is shared, and a button on it would let any subscriber acknowledge
as somebody else.
|
||
|
|
dcb2a86f9a |
chart: bind the HTTPRoute to a named gateway listener
Release / test (push) Failing after 7s
Release / build (amd64, darwin) (push) Has been skipped
Release / build (amd64, linux) (push) Has been skipped
Release / build (arm64, darwin) (push) Has been skipped
Release / build (arm64, linux) (push) Has been skipped
Release / docker (push) Has been skipped
Release / chart (push) Has been skipped
Release / release (push) Has been skipped
The route carried no sectionName, so it attached to every listener whose hostname matched — including the hostname-less plaintext HTTP listener. On a publicly reachable hostname that means the API accepts bearer tokens over cleartext. networking.listener names the listener to bind to. It defaults to empty, which keeps the previous attach-to-all behaviour. Also document the Kubernetes install path, which the README omitted.terdut-server-0.5.0 v0.5.0 |
||
|
|
be739c319f |
Check every push and pull request
The release workflow gates a tag, which is the last possible moment: a commit that breaks the suite stayed green on main until somebody decided to publish, and then failed the release instead of the change that caused it. A CI workflow now runs go vet and go test on pushes to main and on pull requests. push is scoped to main rather than left open. A branch pushed as part of a pull request would otherwise be checked twice, and gh-pages carries the published chart index with no Go code in it, so go vet there would fail on a missing go.mod. Runs for the same ref cancel each other, since a rapid series of pushes only needs the last one checked. |
||
|
|
28cf9faf77 |
Gate the release on vet and tests
CI only ever built and published. The 44 tests in internal/api ran on a laptop or not at all, so a tag could publish binaries, a container image and a Helm chart from a commit whose tests had never been run — and the tests are the only thing holding several documented contracts in place, including the received_at heartbeat and the alert ordering guard. A test job now runs go vet and go test, and build, docker and chart all depend on it. The release job is downstream of build, so a tag that fails publishes nothing at all rather than publishing three artifacts out of four. chart-release.yml is deliberately left alone. It fires on charts/** pushes and publishes the chart, which contains no Go code and only references an image tag rather than building one, so gating it on the Go suite would add a minute to every chart edit for no signal. This still only runs at release time; nothing checks a push or a pull request, so a broken commit stays green until somebody tags it. |
||
|
|
279ef6cf8b |
Turn incoming alerts into incidents
Release / build (amd64, linux) (push) Failing after 11s
Release / build (amd64, darwin) (push) Failing after 12s
Release / build (arm64, darwin) (push) Failing after 11s
Release / build (arm64, linux) (push) Failing after 11s
Release / release (push) Has been skipped
Release / chart (push) Failing after 13s
Release / docker (push) Failing after 19s
The alerts row was both Alertmanager's record and the human work queue, and
the two have different owners. The webhook upsert rewrites that row on every
notification; acknowledgement, comments and archiving were columns on it that
the upsert happened not to touch. So an alert that resolved and re-fired days
later still read as acknowledged by whoever acked the first occurrence — the
ack outlived the thing it referred to. Nothing recorded transitions either:
rows are mutated in place, so there was no timeline and no way to compute how
long anything took.
Alerts are now read-only signal records with two states, and incidents are
the work item: triggered, acknowledged or resolved, with an assignee, a
snooze, notes and an append-only timeline. Many alerts map to one incident,
and a new occurrence opens a new incident, which is what makes a stale ack
impossible rather than merely unlikely.
Correlation uses Alertmanager's own groupKey. It already grouped the alerts
according to the group_by routing tree the operator configured and sends the
result on every webhook, where it was being discarded; adopting it means
changing group_by in alertmanager.yml changes correlation here, with no
second grouping scheme to configure and keep in sync.
An incident opens only when an alert transitions into firing — an unseen
fingerprint, a newer startsAt, or a resolved alert starting again. The
unchanged notifications Alertmanager re-sends every repeat_interval are none
of those. That rule is what lets manual resolution be terminal: without it,
closing an incident by hand would be undone by the next re-send of an alert
that never stopped firing, and the button would be a lie. Snooze covers the
"not now" case instead. Incidents otherwise resolve by cascade, once every
alert under them has stopped firing, whether by webhook or by expiry.
New incidents are assigned to whoever holds today's schedule entry. The
schedule table has existed since the first release with nothing reading it.
Also here, following from the split:
- Incident severity is a high-water mark over its alerts, never lowered.
An incident that hit critical was a critical incident, and downgrading a
live one would demote it in the queue while the work is still open.
- /api/stats/incidents reports MTTA and MTTR, null rather than zero until
there is something to average. Neither was computable before.
- Alert archiving becomes sweeper-only housekeeping; the archive people
interact with is the incident's.
Breaking: the alert acknowledge, archive and comment endpoints are gone, and
the alert object drops the acknowledgement fields and gains incident_id. The
README maps each removed endpoint to its replacement. Migration 008 backfills
an incident per existing alert, archived ones included so no comment is
orphaned, carrying acknowledgements across and turning comments into timeline
notes.
Both documented alert contracts are untouched: received_at still advances on
every accepted payload, re-sends included, and resolution_source still says
how much to trust ends_at. The upsert is byte-for-byte what it was, now
running inside the ingest transaction.
terdut-server-0.4.0
v0.4.0
|
||
|
|
a602ff3efc |
Document received_at and resolution_source as public contract
The API reference listed endpoints but never the alert object's fields, so
two of them were load-bearing for clients while being described nowhere.
received_at appeared only in passing, as a stats filter; resolution_source
only inside the stale-expiry prose.
Both carry meaning a client cannot derive on its own. starts_at comes from
Prometheus and never changes for an alert instance, so received_at is the
only signal that a firing alert is still being refreshed — it advances on
every accepted webhook, including the unchanged notifications Alertmanager
re-sends every repeat_interval. resolution_source then says how much to
trust ends_at: under 'alertmanager' it is an end time somebody reported,
but under 'expiry' nothing ever reported one, so it is either a stale
watermark or the sweep timestamp, and only an upper bound.
README gains an alert object field table plus a contract section for each,
including the nullability rules and the advice to tolerate unrecognised
resolution_source values. The field comments in models.Alert now say these
are public API rather than ingest details, and the upsert carries a note at
the received_at line, which is where a regression would be introduced.
Three tests lock the newly documented behaviour, none of which was covered
before — the whole suite passed with the received_at bump deleted from the
upsert, because the expiry tests only ever set that column via SQL:
- a re-send advances received_at and leaves starts_at alone
- a discarded out-of-order retry does not count as a heartbeat
- an expiry resolve preserves a reported ends_at watermark and stamps
sweep time only when none was known
|
||
|
|
79afd05ea5 |
chart: skip existing releases in chart-release workflow
chart-release.yml fires on any charts/** push to main and ran
chart-releaser against the committed Chart.yaml version, failing with
422 already_exists whenever that version was already published. A tagged
release also publishes the chart from release.yml, which seds the
version from the tag, so the two workflows raced and this one lost.
Brings it to parity with the chart job in release.yml, which has carried
skip_existing since
|
||
|
|
42e846f876 |
Expire stale firing alerts
Release / build (amd64, darwin) (push) Failing after 12s
Release / build (arm64, darwin) (push) Failing after 11s
Release / build (arm64, linux) (push) Failing after 11s
Release / release (push) Has been skipped
Release / docker (push) Failing after 19s
Release / build (amd64, linux) (push) Failing after 12s
Release / chart (push) Failing after 9s
A resolved webhook was the only path out of the firing state, so a
notification that was dropped, silenced, or lost to a restart pinned an
alert as firing forever — Prometheus showed it resolved while
terdut-server kept listing it. The archiver only ever touched resolved
alerts, and both the list and stats queries compared status with plain
equality, so a stale row was indistinguishable from a live one.
A sweeper pass now resolves firing alerts on either of two signals: the
ends_at watermark Alertmanager sets on outgoing firing notifications has
passed (plus a grace period for clock skew), or no webhook has refreshed
the alert within TERDUT_STALE_AFTER (default 6h, above Alertmanager's 4h
repeat_interval). Such alerts get resolution_source = 'expiry',
distinguishing them from a real 'alertmanager' resolve.
Two related webhook bugs fixed alongside:
- The upsert had no ordering guard, so a retried firing notification
arriving after the resolved one resurrected the alert. Payloads for
an older alert instance are now discarded: a stale retry carries the
same startsAt, a genuine re-fire a newer one.
- archived_at was never cleared on re-fire, leaving a re-fired alert
archived and invisible in the default list.
Stats now exclude archived alerts to match the default list view; this
lowers historical firing/resolved totals.
The chart exposes both sweeper durations via sweeper.staleAfter and
sweeper.archiveAfter.
v0.3.0
terdut-server-0.3.0
|
||
|
|
debc4bf78c |
Add alert archiving
Release / build (amd64, darwin) (push) Failing after 2m46s
Release / build (amd64, linux) (push) Failing after 2m26s
Release / build (arm64, darwin) (push) Failing after 1m40s
Release / build (arm64, linux) (push) Failing after 10s
Release / release (push) Has been skipped
Release / chart (push) Failing after 11s
Release / docker (push) Failing after 19s
Alerts can be manually archived (POST /api/alerts/{id}/archive) or
unarchived (DELETE /api/alerts/{id}/archive). A background goroutine
auto-archives resolved alerts older than TERDUT_ARCHIVE_AFTER (default 7d).
GET /api/alerts hides archived alerts by default; ?archived=true shows them.
v0.2.0
|
||
|
|
36468a68ed |
chart: add bootstrap job (v0.2.0)
Post-install/post-upgrade Job that calls /api/bootstrap on first deploy and stores the admin API key in a Secret (<release>-admin-key by default). Exits cleanly on subsequent upgrades when bootstrap is already complete. Adds ServiceAccount, Role (secrets:create), and RoleBinding as hook resources.terdut-server-0.2.0 |
||
|
|
885ba73d12 | chart: guard version sed behind tag pattern check | ||
|
|
1451682cdd | chart: add skip_existing and workflow_dispatch trigger | ||
|
|
a30c52f6dd |
Fix: move var declaration after imports
Release / build (amd64, darwin) (push) Failing after 1m8s
Release / build (arm64, darwin) (push) Failing after 1m5s
Release / build (amd64, linux) (push) Failing after 1m8s
Release / build (arm64, linux) (push) Failing after 21s
Release / release (push) Has been skipped
Release / chart (push) Failing after 1m27s
Release / docker (push) Failing after 3m4s
|
||
|
|
6baaf96638 |
Add release workflow, LICENSE, and version stamping
Tag-triggered workflow builds multi-platform binaries, pushes a multi-arch Docker image to GHCR, bumps and releases the Helm chart, and creates a GitHub release with all binary artifacts. Adds GPL-3.0 LICENSE and version variable stamped at build time via ldflags. |
||
|
|
591290e4ee |
Add Helm chart and chart-release workflow
charts/terdut-server/ — Helm chart for Kubernetes deployment: - Deployment (replicas=1, /healthz probes, TERDUT_DB_PATH=/data/terdut.db) - Service (ClusterIP :8080) - PVC (1Gi, synology-iscsi) mounted at /data - HTTPRoute via envoy-main gateway .github/workflows/chart-release.yml — packages and publishes the chart to gh-pages branch on any push to main that touches charts/; repo URL will be https://yeniklas.github.io/terdut-server once the repo is made public |
||
|
|
17f09558cb |
Stage 7: Dockerfile, integration tests, updated README
Dockerfile: - Multi-stage build (golang:1.25-alpine → scratch) - CGO_ENABLED=0, static binary, stripped with -ldflags="-w -s" (~11 MB) Tests (13 cases, internal/api/api_test.go): - Auth middleware: missing token, invalid token, valid token - Bootstrap idempotency (second call → 403) - Alert upsert: same fingerprint updates row; different fingerprints add rows - Acknowledge: set and clear, verified via GET - Comment ownership: only author can delete own comment (404 for others) - Schedule conflict: duplicate date → 409; multi-date rollback on partial conflict - Stats: totals, by-hour returns 24 slots, by-day returns 7 slots README: quick start, Docker, env vars, Alertmanager config, full API reference |
||
|
|
923fc8bf9c |
Stage 6: alert statistics endpoints
- GET /api/stats/alerts — total/firing/resolved counts - GET /api/stats/alerts/top — most frequent alert names (?limit, default 10) - GET /api/stats/alerts/by-hour — counts for all 24 hours (zeros filled in) - GET /api/stats/alerts/by-day — counts for all 7 days with names (zeros filled in) - All endpoints accept optional ?from/?to (YYYY-MM-DD) to filter by received_at |
||
|
|
f8f209dcba |
Stage 5: on-call schedule
- Migration 005: schedule_entries table (date TEXT UNIQUE, one person per day)
- POST /api/schedule — assign user to one or more dates in a single
transaction; any date conflict rejects the whole request (409)
- GET /api/schedule — list all entries ordered by date, optional ?from/?to
- GET /api/schedule/current — today's on-call user (UTC date), 404 if none
- DELETE /api/schedule/{id} — remove an entry (204)
|
||
|
|
c3348a410a |
Stage 4: alert acknowledgement and comments
- Migration 004: acknowledged_by/acknowledged_at columns on alerts,
alert_comments table (FK cascade on delete)
- POST /api/alerts/{id}/acknowledge — stamps authed user + timestamp,
returns updated alert with acknowledged_by username
- DELETE /api/alerts/{id}/acknowledge — clears ack (204)
- GET /api/alerts/{id}/comments — list in chronological order
- POST /api/alerts/{id}/comments — add comment (returns 201)
- DELETE /api/alerts/{id}/comments/{commentID} — own comments only (204)
- All alert queries now LEFT JOIN users for ack username
|