Advisory-lock DB migrations against concurrent replica startup
Migrate's check-then-apply loop against schema_migrations had no locking: two replicas booting at once against a fresh or partially-migrated database could both pass the "not yet applied" check for the same file and race applying it, crashing whichever lost the duplicate-key insert (confirmed: reverting the lock fails the new test 10/10 on a duplicate-key violation, racing as early as the CREATE TABLE IF NOT EXISTS schema_migrations statement itself). Hold a Postgres advisory lock for Migrate's whole run, on a dedicated connection reserved via db.Conn so lock and unlock happen on the same session. Blocking (pg_advisory_lock), unlike the archiver/notifier's pg_try_advisory_lock: on boot there's no later tick to defer to, so a second replica should wait for the first to finish migrating rather than skip ahead. Adds internal/db's first test file, exercising two concurrent Migrate calls against a fresh schema. Still open: the new-incident-insert race on a webhook for a brand-new groupKey, noted in the chart's updated comment. Login rate limiting staying in-process, diluted across replicas, is an accepted tradeoff. Co-authored-by: Claude <noreply@anthropic.com>
This commit is contained in:
@@ -1,6 +1,7 @@
|
||||
package db
|
||||
|
||||
import (
|
||||
"context"
|
||||
"database/sql"
|
||||
"embed"
|
||||
"fmt"
|
||||
@@ -62,6 +63,16 @@ func Open(dsn string) (*sql.DB, error) {
|
||||
}
|
||||
}
|
||||
|
||||
// migrationLockKey is the Postgres advisory lock Migrate holds for its whole
|
||||
// run. Two replicas starting at once would otherwise race the check-then-apply
|
||||
// loop below against schema_migrations: the loser could crash on a
|
||||
// duplicate-key insert, or contend with the winner's uncommitted DDL. Blocking
|
||||
// (pg_advisory_lock, not pg_try_advisory_lock as the archiver and notifier
|
||||
// use): on boot there is no later tick to defer to, so the right behaviour is
|
||||
// to wait for the other replica to finish migrating, not to skip ahead and
|
||||
// start serving against an unmigrated schema.
|
||||
const migrationLockKey int64 = 7265_0003
|
||||
|
||||
// Migrate applies every embedded migration that has not been applied yet, in
|
||||
// filename order, recording each in schema_migrations.
|
||||
//
|
||||
@@ -69,6 +80,22 @@ func Open(dsn string) (*sql.DB, error) {
|
||||
// migration that failed half way used to leave the schema in whatever state it
|
||||
// had reached. Postgres has transactional DDL, so the rollback is real.
|
||||
func Migrate(db *sql.DB) error {
|
||||
ctx := context.Background()
|
||||
conn, err := db.Conn(ctx)
|
||||
if err != nil {
|
||||
return fmt.Errorf("migrate: acquire connection: %w", err)
|
||||
}
|
||||
defer conn.Close()
|
||||
|
||||
if _, err := conn.ExecContext(ctx, "SELECT pg_advisory_lock($1)", migrationLockKey); err != nil {
|
||||
return fmt.Errorf("migrate: acquire advisory lock: %w", err)
|
||||
}
|
||||
defer func() {
|
||||
if _, err := conn.ExecContext(ctx, "SELECT pg_advisory_unlock($1)", migrationLockKey); err != nil {
|
||||
log.Printf("migrate: release advisory lock: %v", err)
|
||||
}
|
||||
}()
|
||||
|
||||
if _, err := db.Exec(`CREATE TABLE IF NOT EXISTS schema_migrations (
|
||||
version TEXT PRIMARY KEY,
|
||||
applied_at BIGINT NOT NULL DEFAULT FLOOR(EXTRACT(EPOCH FROM now()))::bigint
|
||||
|
||||
Reference in New Issue
Block a user