Files
Niklas Ye dc39e3a5d3
CI / chart (pull_request) Successful in 1s
CI / security (pull_request) Successful in 17s
CI / test (pull_request) Successful in 2m5s
Move the database to Postgres, before teams need the schema
First step of #1, and it goes first for one reason: #4 adds a team_id to
nearly every table, and doing that twice -- once for SQLite, once for
Postgres -- is work nobody gets paid for. The teams migrations now only
have to be written against one database.

The ten SQLite migrations are replaced by a single Postgres baseline
rather than ported one by one. They were incremental in a way that has
no value on a fresh install: 004 adds columns 008 drops again, and 008's
backfill rewrites data a Postgres database never had. The history stays
in git; the schema they add up to is now 001_baseline.sql.

Timestamps stay BIGINT unix seconds and are NOT converted to timestamptz.
Everything in Go already speaks epochs, so converting would have been a
second, larger change riding along inside this one. It is worth doing on
its own. The JSON columns did move to jsonb, because #4 will want to
filter and index on labels.

Most of the port is mechanical -- 170 placeholders from ? to $1 -- but
four things needed more than a search and replace:

  * Dynamically built WHERE clauses cannot keep their numbering straight
    by hand, so they hand out placeholders through sqlArgs instead. A
    filter can now be added or reordered without renumbering anything.

  * SUM(resolved_at IS NULL) was SQLite counting a boolean as 0 or 1.
    Postgres has no sum(boolean), and this was breaking every dead man's
    switch -- silently, since the sweeper only logs. Now COUNT(*) FILTER.

  * unixepoch() became FLOOR(EXTRACT(EPOCH FROM now()))::bigint. The
    FLOOR is load-bearing: a bare cast rounds half up, so a row written
    at .6 of a second claimed a timestamp a second in the future and
    disagreed with the time.Now().Unix() the Go side stamps.

  * The unique-violation check matched SQLite's error text. It matches
    SQLSTATE 23505 now, so a renamed constraint cannot turn a 409 back
    into a 500.

Tests need a real Postgres, because there is no in-memory Postgres the
way there was an in-memory SQLite. Each test gets its own schema on a
shared server -- cheaper than a database each, and still isolated.
TERDUT_TEST_DSN says where it is; `make test-db` starts one locally and
ci.yaml runs one as a service container. An unset DSN fails the suite
rather than skipping it: a run that quietly tests nothing is worse than
one that does not run.

TestMigration_BackfillCarriesAckAndComments is deleted along with the
migrations it replayed. What it protected -- an upgrade not losing
acknowledgements and comments -- now belongs to scripts/sqlite-to-postgres.go,
which is build-tagged so the SQLite driver stays out of the server
binary. Both are meant to be deleted once this install has migrated.

The chart loses the PVC, the data volume and the python backup sidecar,
and requires database.dsnSecret.name: it provisions no database and
cannot guess where the credentials live, so a render without it is meant
to fail. Backups move to where Postgres actually runs. The other half of
that -- the postgresql CR, the k8up pg_dump annotation and the network
policy -- is a change to the wrapper chart in Ryuvia/charts and is not in
here.

Verified rather than assumed: the gate is green with -race against
Postgres 17, govulncheck and gitleaks are clean, and the migration script
was run end to end against a SQLite database built at the old schema and
seeded in every table. Ids survive, so incidents keep their numbers and
every foreign key still points where it did; the identity sequences are
moved past the copied ids, and a webhook after the migration opened
incident 12 rather than colliding at 1.
2026-09-20 10:44:12 +02:00

152 lines
6.0 KiB
YAML

name: CI
# The release workflow gates a tag, which is late: a broken commit sits green until
# somebody decides to publish. This runs the same checks on the way in.
#
# push is scoped to main rather than all branches so that a branch pushed as part of a
# pull request is not checked twice.
#
# No actions/checkout, deliberately -- same as the letsvisit and charts workflows. The
# runner image is ubuntu:22.04 whose `nodejs` package is Node 12, and actions/checkout@v4
# is built with ES2022 static initialiser blocks, so it dies with
# `SyntaxError: Unexpected token '{'` before running. Cloning with git directly avoids JS
# actions entirely. This repo is public, so the clone needs no credential at all.
#
# `${{ }}` values are passed through `env:` and referenced as quoted shell variables: a
# ref name is attacker-influenced by anyone who can push a branch or open a PR, and
# expanding one straight into `run:` is a shell-injection vector.
on:
push:
branches: [main]
pull_request:
# A rapid series of pushes only needs the last one checked.
concurrency:
group: ci-${{ github.ref }}
cancel-in-progress: true
env:
REPO_URL: https://git.ryuvia.com/niklas/terdut-server.git
jobs:
test:
runs-on: ubuntu-latest
container:
# Runs inside the toolchain image rather than installing Go per job. Note this puts
# the job on the dind bridge, which cannot reach github.com or get.helm.sh --
# proxy.golang.org and git.ryuvia.com are reachable, which is all this job needs.
image: golang:1.26.6-bookworm
# act_runner destroys a job's own volumes when it finishes, so without these every
# run re-downloads the whole module graph. The names must appear in the runner's
# container.valid_volumes allowlist (charts/act-runner in the k8s repo); unlisted
# volumes are dropped silently, so a workflow that looks correct can still be
# running uncached.
volumes:
- go-mod-cache:/go/pkg/mod
- go-build-cache:/root/.cache/go-build
- gobin-cache:/go/bin
# The suite needs a real Postgres -- there is no in-memory Postgres the way there was
# an in-memory SQLite, so each test gets its own schema on a shared server instead.
# The job and the service share the dind bridge, so the service is reachable by its
# name rather than on localhost.
services:
postgres:
image: postgres:17-alpine
env:
POSTGRES_USER: terdut
POSTGRES_PASSWORD: terdut
POSTGRES_DB: terdut_test
options: >-
--health-cmd "pg_isready -U terdut -d terdut_test"
--health-interval 5s
--health-timeout 5s
--health-retries 12
env:
# `make test` fails without this rather than skipping, so a green job here means
# the tests actually ran against a database.
TERDUT_TEST_DSN: postgres://terdut:terdut@postgres:5432/terdut_test?sslmode=disable
steps:
- name: Checkout
env:
REF_NAME: ${{ github.ref_name }}
HEAD_SHA: ${{ github.event.pull_request.head.sha }}
run: |
if [ -n "$HEAD_SHA" ]; then
# A pull_request ref_name is "<n>/merge", which is not a fetchable branch.
git clone "$REPO_URL" .
git checkout -q "$HEAD_SHA"
else
git clone --depth=1 --branch "$REF_NAME" "$REPO_URL" .
fi
# The gate is the Makefile's rather than a second copy of it here, the way riksdata
# and rd-web already do it. `make fmt lint test` is exactly what a developer runs, so
# a green pipeline and a green working copy mean the same thing by construction
# instead of by remembering to update two files together.
#
# The reasoning that used to live here moved with the targets: why gofmt is checked
# at all (import order survives `go vet`, and both repos sat unformatted through a
# green run and a release -- 9046f6e), why both of gofmt's failure modes need
# handling, and why `test` adds -race when this job does not have to.
- name: Format, vet and test
run: make fmt lint test
# Runs on every push and pull request, unlike the image scan, which needs something
# published to scan and so lives in release.yaml. Both are needed: govulncheck reads the
# source and its module graph, trivy reads the built artifact, and neither sees what the
# other does.
security:
runs-on: ubuntu-latest
container:
image: golang:1.26.6-bookworm
volumes:
- go-mod-cache:/go/pkg/mod
- go-build-cache:/root/.cache/go-build
- gobin-cache:/go/bin
steps:
- name: Checkout
env:
REF_NAME: ${{ github.ref_name }}
HEAD_SHA: ${{ github.event.pull_request.head.sha }}
run: |
if [ -n "$HEAD_SHA" ]; then
git clone "$REPO_URL" .
git checkout -q "$HEAD_SHA"
else
git clone --depth=1 --branch "$REF_NAME" "$REPO_URL" .
fi
- name: Go vulnerability scan (govulncheck)
run: make security-go
- name: Secret scan (gitleaks)
run: make security-secrets
# Host mode, no `container:`: helm is baked into the runner image, and a container job
# could not install it -- get.helm.sh is unreachable from the dind bridge. Same reason
# release.yaml's chart job runs on the host.
#
# The chart had no lint step in any workflow until 2026-09-01: release.yaml packaged and
# pushed it without rendering it first, so a template that did not compile would have
# been found by Flux rather than here.
chart:
runs-on: ubuntu-latest
steps:
- name: Checkout
env:
REF_NAME: ${{ github.ref_name }}
HEAD_SHA: ${{ github.event.pull_request.head.sha }}
run: |
if [ -n "$HEAD_SHA" ]; then
git clone "$REPO_URL" .
git checkout -q "$HEAD_SHA"
else
git clone --depth=1 --branch "$REF_NAME" "$REPO_URL" .
fi
- name: Lint and render the chart
run: make helm-lint