Move the database to Postgres, before teams need the schema
First step of #1, and it goes first for one reason: #4 adds a team_id to nearly every table, and doing that twice -- once for SQLite, once for Postgres -- is work nobody gets paid for. The teams migrations now only have to be written against one database. The ten SQLite migrations are replaced by a single Postgres baseline rather than ported one by one. They were incremental in a way that has no value on a fresh install: 004 adds columns 008 drops again, and 008's backfill rewrites data a Postgres database never had. The history stays in git; the schema they add up to is now 001_baseline.sql. Timestamps stay BIGINT unix seconds and are NOT converted to timestamptz. Everything in Go already speaks epochs, so converting would have been a second, larger change riding along inside this one. It is worth doing on its own. The JSON columns did move to jsonb, because #4 will want to filter and index on labels. Most of the port is mechanical -- 170 placeholders from ? to $1 -- but four things needed more than a search and replace: * Dynamically built WHERE clauses cannot keep their numbering straight by hand, so they hand out placeholders through sqlArgs instead. A filter can now be added or reordered without renumbering anything. * SUM(resolved_at IS NULL) was SQLite counting a boolean as 0 or 1. Postgres has no sum(boolean), and this was breaking every dead man's switch -- silently, since the sweeper only logs. Now COUNT(*) FILTER. * unixepoch() became FLOOR(EXTRACT(EPOCH FROM now()))::bigint. The FLOOR is load-bearing: a bare cast rounds half up, so a row written at .6 of a second claimed a timestamp a second in the future and disagreed with the time.Now().Unix() the Go side stamps. * The unique-violation check matched SQLite's error text. It matches SQLSTATE 23505 now, so a renamed constraint cannot turn a 409 back into a 500. Tests need a real Postgres, because there is no in-memory Postgres the way there was an in-memory SQLite. Each test gets its own schema on a shared server -- cheaper than a database each, and still isolated. TERDUT_TEST_DSN says where it is; `make test-db` starts one locally and ci.yaml runs one as a service container. An unset DSN fails the suite rather than skipping it: a run that quietly tests nothing is worse than one that does not run. TestMigration_BackfillCarriesAckAndComments is deleted along with the migrations it replayed. What it protected -- an upgrade not losing acknowledgements and comments -- now belongs to scripts/sqlite-to-postgres.go, which is build-tagged so the SQLite driver stays out of the server binary. Both are meant to be deleted once this install has migrated. The chart loses the PVC, the data volume and the python backup sidecar, and requires database.dsnSecret.name: it provisions no database and cannot guess where the credentials live, so a render without it is meant to fail. Backups move to where Postgres actually runs. The other half of that -- the postgresql CR, the k8up pg_dump annotation and the network policy -- is a change to the wrapper chart in Ryuvia/charts and is not in here. Verified rather than assumed: the gate is green with -race against Postgres 17, govulncheck and gitleaks are clean, and the migration script was run end to end against a SQLite database built at the old schema and seeded in every table. Ids survive, so incidents keep their numbers and every foreign key still points where it did; the identity sequences are moved past the copied ids, and a webhook after the migration opened incident 12 rather than colliding at 1.
This commit is contained in:
@@ -9,7 +9,7 @@ Incident management server for teams using Prometheus Alertmanager.
|
||||
- Alert and incident statistics, including MTTA and MTTR
|
||||
- Web UI for phones and desktops, served by the same binary
|
||||
- REST API with per-user API key authentication
|
||||
- Single binary, SQLite storage — trivial to self-host
|
||||
- Single binary plus a Postgres — straightforward to self-host
|
||||
|
||||
---
|
||||
|
||||
@@ -89,11 +89,14 @@ the web UI (`/incidents/{id}`).
|
||||
|
||||
```bash
|
||||
docker build -t terdut-server .
|
||||
docker run -p 8080:8080 -v $(pwd)/data:/data \
|
||||
-e TERDUT_DB_PATH=/data/terdut.db \
|
||||
docker run -p 8080:8080 \
|
||||
-e TERDUT_DB_DSN='postgres://terdut:secret@host.docker.internal:5432/terdut?sslmode=disable' \
|
||||
terdut-server
|
||||
```
|
||||
|
||||
The server creates its own schema on startup and needs a reachable Postgres; it stores nothing on
|
||||
disk, so there is no volume to mount.
|
||||
|
||||
### Kubernetes
|
||||
|
||||
A Helm chart is published from this repository as an OCI artifact, versioned in lockstep
|
||||
@@ -116,24 +119,30 @@ than an `Ingress`. TLS is terminated at the gateway, so the server itself never
|
||||
| `networking.listener` | `""` | Gateway listener (`sectionName`) to bind to. Empty attaches to every matching listener, **including plaintext HTTP** — set it to the HTTPS listener's name to serve TLS only |
|
||||
| `networking.servicePort` | `8080` | Port the route forwards to; keep in sync with `service.port` |
|
||||
| `bootstrap.enabled` | `true` | Runs a post-install hook that creates the first user and stores its API key in the `<release>-admin-key` Secret. Already-bootstrapped servers are left alone |
|
||||
| `backupSidecar.enabled` | `true` | Adds an idle `python` sidecar and the [k8up](https://k8up.io/) annotations that dump the database through it |
|
||||
| `database.dsnSecret.name` | `""` | **Required.** Existing Secret holding the Postgres DSN. The chart provisions no database |
|
||||
| `database.dsnSecret.key` | `dsn` | Key within that Secret |
|
||||
|
||||
The API key travels in an `Authorization: Bearer` header, so set `networking.listener` whenever the
|
||||
hostname is reachable outside a trusted network.
|
||||
|
||||
#### The database
|
||||
|
||||
The chart provisions no database: it takes a DSN from a Secret and expects a Postgres that already
|
||||
exists. In this cluster the wrapper chart declares an `acid.zalan.do/v1 postgresql` CR and passes
|
||||
the Secret the operator writes; anywhere else, any reachable Postgres 14+ will do.
|
||||
|
||||
The server migrates its own schema on startup, so a new database only has to exist and be writable.
|
||||
|
||||
#### Backups
|
||||
|
||||
The server image is `FROM scratch` — the binary and nothing else — so there is no interpreter to
|
||||
run a database dump in, and the database runs in WAL mode, where a file-level copy of the volume is
|
||||
not crash-consistent. The chart therefore ships an idle `python:*-alpine` sidecar that shares the
|
||||
data volume, and points k8up's `backupcommand` at it with `k8up.io/backupcommand-container`. Without
|
||||
that annotation k8up execs into `.spec.containers[0]` and the dump fails.
|
||||
Postgres is backed up where it runs, not from here. The database pod carries a
|
||||
[k8up](https://k8up.io/) `k8up.io/backupcommand` annotation that streams a `pg_dump`, the same way
|
||||
gitea and immich do in this cluster.
|
||||
|
||||
The dump is buffered and sanity-checked before its first byte reaches stdout, because k8up streams
|
||||
stdout straight into Restic: a dump that dies partway is otherwise stored as a silently truncated
|
||||
snapshot that k8up still reports as successful.
|
||||
|
||||
Set `backupSidecar.enabled=false` if you back the volume up some other way.
|
||||
This used to be the app's problem: the SQLite database lived on a PVC beside the server, the image
|
||||
is `FROM scratch` with no interpreter to dump it, and WAL mode makes a file-level copy of the volume
|
||||
non-crash-consistent — so the chart shipped an idle `python:*-alpine` sidecar purely to give k8up
|
||||
somewhere to exec. The sidecar, the PVC and the `backupSidecar` values are all gone.
|
||||
|
||||
---
|
||||
|
||||
@@ -142,7 +151,7 @@ Set `backupSidecar.enabled=false` if you back the volume up some other way.
|
||||
| Variable | Default | Description |
|
||||
|---|---|---|
|
||||
| `TERDUT_ADDR` | `:8080` | TCP address to listen on |
|
||||
| `TERDUT_DB_PATH` | `terdut.db` | Path to the SQLite database file |
|
||||
| `TERDUT_DB_DSN` | — | **Required.** Postgres connection string, e.g. `postgres://terdut:secret@localhost:5432/terdut?sslmode=require` |
|
||||
| `TERDUT_ARCHIVE_AFTER` | `168h` (7d) | How long a resolved alert or incident stays in the default list before being auto-archived |
|
||||
| `TERDUT_STALE_AFTER` | `6h` | How long a firing alert may go without a refreshing webhook before it is treated as resolved — **must exceed your Alertmanager `repeat_interval`** |
|
||||
| `TERDUT_DEADMAN_MATCHERS` | `alertname=Watchdog` | Which alerts are [dead man's switches](#dead-mans-switch). `;` separates matchers, `,` the label conditions within one, `=` is exact equality. Every matcher must name an `alertname` |
|
||||
@@ -662,6 +671,33 @@ averages over incidents that have actually been acknowledged or resolved, and ar
|
||||
|
||||
---
|
||||
|
||||
## Upgrading from SQLite
|
||||
|
||||
Versions up to v0.10.2 stored everything in a SQLite file. From the Postgres release onwards
|
||||
the server needs `TERDUT_DB_DSN` and keeps nothing on disk.
|
||||
|
||||
The cutover is ordered — the server must not be running while the copy happens:
|
||||
|
||||
```bash
|
||||
# 1. Stop the old server, keeping its database file.
|
||||
# 2. Create an empty Postgres database, then let the new binary build the schema:
|
||||
TERDUT_DB_DSN='postgres://terdut:secret@localhost:5432/terdut?sslmode=disable' ./terdut &
|
||||
# ...watch for "listening on", then stop it again.
|
||||
# 3. Copy the data across:
|
||||
go run -tags migrate ./scripts/sqlite-to-postgres.go \
|
||||
-sqlite /data/terdut.db \
|
||||
-dsn 'postgres://terdut:secret@localhost:5432/terdut?sslmode=disable'
|
||||
# 4. Start the new server for good.
|
||||
```
|
||||
|
||||
The copy preserves every id, so incidents keep their numbers and the timeline, alert
|
||||
membership, outbox and ack tokens all still point where they did. It refuses a target that
|
||||
already has rows, so a second run cannot double-insert. On Kubernetes, step 3 runs as a Job
|
||||
with the same image against the PVC before it is removed.
|
||||
|
||||
The script is deliberately temporary: it is the only thing left that needs the SQLite driver,
|
||||
and both should be deleted once the installs that need them have migrated.
|
||||
|
||||
## Upgrading to incidents
|
||||
|
||||
The incidents release moves the workflow off alerts, which is a **breaking API
|
||||
@@ -712,15 +748,23 @@ There is no migration and no schema change. An existing open incident from a
|
||||
## Development
|
||||
|
||||
```bash
|
||||
go test ./... # run all tests
|
||||
make test-db # start a local Postgres for the tests (podman or docker)
|
||||
make test # run all tests
|
||||
go build ./... # compile all packages
|
||||
go run ./cmd/terdut # run locally
|
||||
go run ./cmd/terdut # run locally (needs TERDUT_DB_DSN)
|
||||
```
|
||||
|
||||
The tests need a real Postgres, because the server does — there is no in-memory Postgres the
|
||||
way there was an in-memory SQLite. `TERDUT_TEST_DSN` says where it is, `make test-db` starts
|
||||
one on port 5433 and prints the DSN, and `make test-db-stop` removes it. Each test gets its
|
||||
own schema on that server, so tests cannot see each other's rows. An unset `TERDUT_TEST_DSN`
|
||||
fails the suite rather than skipping it: a run that quietly tests nothing is worse than one
|
||||
that does not run.
|
||||
|
||||
`make fmt lint test helm-lint` is the gate. It mirrors `.gitea/workflows/ci.yaml` step for
|
||||
step, so a green run here means a green pipeline — with one deliberate exception: `make test`
|
||||
adds `-race`, which CI does not. The sweeper, the notifier goroutine and the dead man's switch
|
||||
sweep all touch the same single database connection, and a race between them would surface as
|
||||
sweep all run concurrently against the same database, and a race between them would surface as
|
||||
a flaky incident in production rather than as a red build.
|
||||
|
||||
The web UI lives in `internal/web/static/` as plain HTML, CSS and ES modules,
|
||||
@@ -776,7 +820,9 @@ Two things the release process needs to know about this repo:
|
||||
do not bump the wrapper chart to that version — it cannot unpublish anything. The image is
|
||||
`FROM scratch`, so trivy sees exactly one target, the Go binary and its module graph.
|
||||
- **The wrapper chart's `values.yaml` has two `tag:` lines** — the app image and the python
|
||||
backup sidecar — so `chart-bump` is given `--image` to say which one moves.
|
||||
backup sidecar — so `chart-bump` is given `--image` to say which one moves. The sidecar is
|
||||
on its way out with SQLite: once the wrapper chart drops it and declares a `postgresql` CR
|
||||
instead, there is one `tag:` line again, and `--image` becomes belt and braces.
|
||||
|
||||
The wrapper chart must have **its own `version:` bumped in the same commit**. Flux reconciles
|
||||
with `reconcileStrategy: ChartVersion`, so a chart whose version did not change produces no
|
||||
|
||||
Reference in New Issue
Block a user