Niklas Ye
42e846f876
Release / build (amd64, darwin) (push) Failing after 12s
Release / build (arm64, darwin) (push) Failing after 11s
Release / build (arm64, linux) (push) Failing after 11s
Release / release (push) Has been skipped
Release / docker (push) Failing after 19s
Release / build (amd64, linux) (push) Failing after 12s
Release / chart (push) Failing after 9s
A resolved webhook was the only path out of the firing state, so a
notification that was dropped, silenced, or lost to a restart pinned an
alert as firing forever — Prometheus showed it resolved while
terdut-server kept listing it. The archiver only ever touched resolved
alerts, and both the list and stats queries compared status with plain
equality, so a stale row was indistinguishable from a live one.
A sweeper pass now resolves firing alerts on either of two signals: the
ends_at watermark Alertmanager sets on outgoing firing notifications has
passed (plus a grace period for clock skew), or no webhook has refreshed
the alert within TERDUT_STALE_AFTER (default 6h, above Alertmanager's 4h
repeat_interval). Such alerts get resolution_source = 'expiry',
distinguishing them from a real 'alertmanager' resolve.
Two related webhook bugs fixed alongside:
- The upsert had no ordering guard, so a retried firing notification
arriving after the resolved one resurrected the alert. Payloads for
an older alert instance are now discarded: a stale retry carries the
same startsAt, a genuine re-fire a newer one.
- archived_at was never cleared on re-fire, leaving a re-fired alert
archived and invisible in the default list.
Stats now exclude archived alerts to match the default list view; this
lowers historical firing/resolved totals.
The chart exposes both sweeper durations via sweeper.staleAfter and
sweeper.archiveAfter.
29 lines
1.1 KiB
Go
29 lines
1.1 KiB
Go
package models
|
|
|
|
import "time"
|
|
|
|
type Alert struct {
|
|
ID int64 `json:"id"`
|
|
Fingerprint string `json:"fingerprint"`
|
|
Name string `json:"name"`
|
|
Status string `json:"status"` // "firing" or "resolved"
|
|
Labels map[string]string `json:"labels"`
|
|
Annotations map[string]string `json:"annotations"`
|
|
StartsAt time.Time `json:"starts_at"`
|
|
EndsAt *time.Time `json:"ends_at,omitempty"`
|
|
GeneratorURL string `json:"generator_url"`
|
|
ReceivedAt time.Time `json:"received_at"`
|
|
|
|
// Populated when the alert has been acknowledged.
|
|
AcknowledgedByID *int64 `json:"acknowledged_by_id,omitempty"`
|
|
AcknowledgedByUser *string `json:"acknowledged_by,omitempty"`
|
|
AcknowledgedAt *time.Time `json:"acknowledged_at,omitempty"`
|
|
|
|
// ResolutionSource records why a resolved alert left the firing state:
|
|
// "alertmanager" for a real resolved webhook, "expiry" when the sweeper
|
|
// inferred it after the alert stopped being refreshed.
|
|
ResolutionSource *string `json:"resolution_source,omitempty"`
|
|
|
|
ArchivedAt *time.Time `json:"archived_at,omitempty"`
|
|
}
|