Niklas Ye
279ef6cf8b
Release / build (amd64, linux) (push) Failing after 11s
Release / build (amd64, darwin) (push) Failing after 12s
Release / build (arm64, darwin) (push) Failing after 11s
Release / build (arm64, linux) (push) Failing after 11s
Release / release (push) Has been skipped
Release / chart (push) Failing after 13s
Release / docker (push) Failing after 19s
The alerts row was both Alertmanager's record and the human work queue, and
the two have different owners. The webhook upsert rewrites that row on every
notification; acknowledgement, comments and archiving were columns on it that
the upsert happened not to touch. So an alert that resolved and re-fired days
later still read as acknowledged by whoever acked the first occurrence — the
ack outlived the thing it referred to. Nothing recorded transitions either:
rows are mutated in place, so there was no timeline and no way to compute how
long anything took.
Alerts are now read-only signal records with two states, and incidents are
the work item: triggered, acknowledged or resolved, with an assignee, a
snooze, notes and an append-only timeline. Many alerts map to one incident,
and a new occurrence opens a new incident, which is what makes a stale ack
impossible rather than merely unlikely.
Correlation uses Alertmanager's own groupKey. It already grouped the alerts
according to the group_by routing tree the operator configured and sends the
result on every webhook, where it was being discarded; adopting it means
changing group_by in alertmanager.yml changes correlation here, with no
second grouping scheme to configure and keep in sync.
An incident opens only when an alert transitions into firing — an unseen
fingerprint, a newer startsAt, or a resolved alert starting again. The
unchanged notifications Alertmanager re-sends every repeat_interval are none
of those. That rule is what lets manual resolution be terminal: without it,
closing an incident by hand would be undone by the next re-send of an alert
that never stopped firing, and the button would be a lie. Snooze covers the
"not now" case instead. Incidents otherwise resolve by cascade, once every
alert under them has stopped firing, whether by webhook or by expiry.
New incidents are assigned to whoever holds today's schedule entry. The
schedule table has existed since the first release with nothing reading it.
Also here, following from the split:
- Incident severity is a high-water mark over its alerts, never lowered.
An incident that hit critical was a critical incident, and downgrading a
live one would demote it in the queue while the work is still open.
- /api/stats/incidents reports MTTA and MTTR, null rather than zero until
there is something to average. Neither was computable before.
- Alert archiving becomes sweeper-only housekeeping; the archive people
interact with is the incident's.
Breaking: the alert acknowledge, archive and comment endpoints are gone, and
the alert object drops the acknowledgement fields and gains incident_id. The
README maps each removed endpoint to its replacement. Migration 008 backfills
an incident per existing alert, archived ones included so no comment is
orphaned, carrying acknowledgements across and turning comments into timeline
notes.
Both documented alert contracts are untouched: received_at still advances on
every accepted payload, re-sends included, and resolution_source still says
how much to trust ends_at. The upsert is byte-for-byte what it was, now
running inside the ingest transaction.
69 lines
2.9 KiB
Go
69 lines
2.9 KiB
Go
package models
|
|
|
|
import "time"
|
|
|
|
// Incident is the human work item: the thing that gets acknowledged, assigned,
|
|
// snoozed, discussed and resolved. Alerts are the machine-owned signal records
|
|
// underneath it — many alerts map to one incident, correlated by the groupKey
|
|
// Alertmanager already computed from the operator's group_by configuration.
|
|
//
|
|
// Nothing here is ever written by the Alertmanager webhook except Status, which
|
|
// the webhook and the sweeper may flip to "resolved" once every member alert has
|
|
// stopped firing.
|
|
type Incident struct {
|
|
ID int64 `json:"id"`
|
|
GroupKey string `json:"group_key"`
|
|
Title string `json:"title"`
|
|
GroupLabels map[string]string `json:"group_labels"`
|
|
|
|
// Status is "triggered", "acknowledged" or "resolved".
|
|
Status string `json:"status"`
|
|
|
|
// Severity is the highest `severity` label across the alerts that were
|
|
// firing when it was last recomputed. It is deliberately not cleared when an
|
|
// incident resolves — a resolved incident should still say how bad it was.
|
|
Severity *string `json:"severity,omitempty"`
|
|
|
|
TriggeredAt time.Time `json:"triggered_at"`
|
|
|
|
AcknowledgedByID *int64 `json:"acknowledged_by_id,omitempty"`
|
|
AcknowledgedByUser *string `json:"acknowledged_by,omitempty"`
|
|
AcknowledgedAt *time.Time `json:"acknowledged_at,omitempty"`
|
|
|
|
AssignedToID *int64 `json:"assigned_to_id,omitempty"`
|
|
AssignedToUser *string `json:"assigned_to,omitempty"`
|
|
|
|
// SnoozedUntil hides the incident from the default queue without closing it.
|
|
// A timestamp in the past reads as "not snoozed"; nothing sweeps it.
|
|
SnoozedUntil *time.Time `json:"snoozed_until,omitempty"`
|
|
|
|
ResolvedAt *time.Time `json:"resolved_at,omitempty"`
|
|
|
|
// ResolutionSource is "alerts" when every member alert stopped firing, or
|
|
// "manual" when a human closed it. Manual resolution is terminal: a later
|
|
// occurrence opens a new incident rather than reopening this one.
|
|
ResolutionSource *string `json:"resolution_source,omitempty"`
|
|
|
|
ArchivedAt *time.Time `json:"archived_at,omitempty"`
|
|
|
|
// Alerts is populated by GET /api/incidents/{id} only.
|
|
Alerts []Alert `json:"alerts,omitempty"`
|
|
}
|
|
|
|
// IncidentEvent is one entry in an incident's timeline. The table is append-only
|
|
// and is the only history this server keeps — alert rows are mutated in place.
|
|
//
|
|
// Type is one of: triggered, alert_added, alert_resolved, acknowledged,
|
|
// unacknowledged, assigned, snoozed, unsnoozed, resolved, note. A nil UserID
|
|
// means the server acted rather than a person.
|
|
type IncidentEvent struct {
|
|
ID int64 `json:"id"`
|
|
IncidentID int64 `json:"incident_id"`
|
|
Type string `json:"type"`
|
|
UserID *int64 `json:"user_id,omitempty"`
|
|
Username *string `json:"username,omitempty"`
|
|
AlertID *int64 `json:"alert_id,omitempty"`
|
|
Detail *string `json:"detail,omitempty"`
|
|
CreatedAt time.Time `json:"created_at"`
|
|
}
|