Expire stale firing alerts
Release / build (amd64, darwin) (push) Failing after 12s
Release / build (arm64, darwin) (push) Failing after 11s
Release / build (arm64, linux) (push) Failing after 11s
Release / release (push) Has been skipped
Release / docker (push) Failing after 19s
Release / build (amd64, linux) (push) Failing after 12s
Release / chart (push) Failing after 9s

A resolved webhook was the only path out of the firing state, so a
notification that was dropped, silenced, or lost to a restart pinned an
alert as firing forever — Prometheus showed it resolved while
terdut-server kept listing it. The archiver only ever touched resolved
alerts, and both the list and stats queries compared status with plain
equality, so a stale row was indistinguishable from a live one.

A sweeper pass now resolves firing alerts on either of two signals: the
ends_at watermark Alertmanager sets on outgoing firing notifications has
passed (plus a grace period for clock skew), or no webhook has refreshed
the alert within TERDUT_STALE_AFTER (default 6h, above Alertmanager's 4h
repeat_interval). Such alerts get resolution_source = 'expiry',
distinguishing them from a real 'alertmanager' resolve.

Two related webhook bugs fixed alongside:

  - The upsert had no ordering guard, so a retried firing notification
    arriving after the resolved one resurrected the alert. Payloads for
    an older alert instance are now discarded: a stale retry carries the
    same startsAt, a genuine re-fire a newer one.
  - archived_at was never cleared on re-fire, leaving a re-fired alert
    archived and invisible in the default list.

Stats now exclude archived alerts to match the default list view; this
lowers historical firing/resolved totals.

The chart exposes both sweeper durations via sweeper.staleAfter and
sweeper.archiveAfter.
This commit is contained in:
Niklas Ye
2026-07-28 11:49:39 +02:00
parent debc4bf78c
commit 42e846f876
12 changed files with 391 additions and 42 deletions
+2 -4
View File
@@ -160,8 +160,9 @@ func handleStatsByDay(db *sql.DB) http.HandlerFunc {
}
// statsFilter builds a WHERE clause and args from optional ?from and ?to query params.
// Archived alerts are always excluded, matching the default GET /api/alerts view.
func statsFilter(q url.Values) (where string, args []any) {
clauses := []string{}
clauses := []string{"archived_at IS NULL"}
if from := q.Get("from"); from != "" {
if t, err := time.Parse("2006-01-02", from); err == nil {
clauses = append(clauses, "received_at >= ?")
@@ -174,8 +175,5 @@ func statsFilter(q url.Values) (where string, args []any) {
args = append(args, t.UTC().AddDate(0, 0, 1).Unix())
}
}
if len(clauses) == 0 {
return "1=1", args
}
return strings.Join(clauses, " AND "), args
}