Document received_at and resolution_source as public contract
The API reference listed endpoints but never the alert object's fields, so
two of them were load-bearing for clients while being described nowhere.
received_at appeared only in passing, as a stats filter; resolution_source
only inside the stale-expiry prose.
Both carry meaning a client cannot derive on its own. starts_at comes from
Prometheus and never changes for an alert instance, so received_at is the
only signal that a firing alert is still being refreshed — it advances on
every accepted webhook, including the unchanged notifications Alertmanager
re-sends every repeat_interval. resolution_source then says how much to
trust ends_at: under 'alertmanager' it is an end time somebody reported,
but under 'expiry' nothing ever reported one, so it is either a stale
watermark or the sweep timestamp, and only an upper bound.
README gains an alert object field table plus a contract section for each,
including the nullability rules and the advice to tolerate unrecognised
resolution_source values. The field comments in models.Alert now say these
are public API rather than ingest details, and the upsert carries a note at
the received_at line, which is where a regression would be introduced.
Three tests lock the newly documented behaviour, none of which was covered
before — the whole suite passed with the received_at bump deleted from the
upsert, because the expiry tests only ever set that column via SQL:
- a re-send advances received_at and leaves starts_at alone
- a discarded out-of-order retry does not count as a heartbeat
- an expiry resolve preserves a reported ends_at watermark and stamps
sweep time only when none was known
This commit is contained in:
@@ -147,6 +147,86 @@ passed. Resolved alerts carry `resolution_source`: `"alertmanager"` for a real
|
||||
resolved webhook, `"expiry"` when the sweeper inferred it (see
|
||||
[Stale alert expiry](#stale-alert-expiry)).
|
||||
|
||||
#### The alert object
|
||||
|
||||
Returned by `GET /api/alerts` (as an array) and `GET /api/alerts/{id}`.
|
||||
Timestamps are RFC 3339 in UTC. Fields marked *optional* are omitted entirely
|
||||
when unset, so clients must treat them as nullable.
|
||||
|
||||
| Field | Type | Notes |
|
||||
|---|---|---|
|
||||
| `id` | integer | Server-assigned; stable for the life of the row |
|
||||
| `fingerprint` | string | Alertmanager's fingerprint — the upsert key |
|
||||
| `name` | string | From the `alertname` label |
|
||||
| `status` | string | `"firing"` or `"resolved"` |
|
||||
| `labels` | object | String→string, as sent by Alertmanager |
|
||||
| `annotations` | object | String→string, as sent by Alertmanager |
|
||||
| `starts_at` | timestamp | When the alert instance began, **per Prometheus** |
|
||||
| `ends_at` | timestamp | *optional* — absent while no end is known |
|
||||
| `generator_url` | string | Link back to the originating Prometheus |
|
||||
| `received_at` | timestamp | When the server last accepted a webhook for this alert — see below |
|
||||
| `acknowledged_by_id` | integer | *optional* — user id |
|
||||
| `acknowledged_by` | string | *optional* — username |
|
||||
| `acknowledged_at` | timestamp | *optional* |
|
||||
| `resolution_source` | string | *optional* — `"alertmanager"` or `"expiry"` |
|
||||
| `archived_at` | timestamp | *optional* — set while archived |
|
||||
|
||||
##### `received_at` is a liveness heartbeat
|
||||
|
||||
`starts_at` comes from Prometheus and **never changes** for the lifetime of an
|
||||
alert instance. It says when the problem began, not whether it is still
|
||||
happening — an alert that started twelve days ago looks identical whether
|
||||
Alertmanager refreshed it a minute ago or went silent a week ago.
|
||||
|
||||
`received_at` is the field that answers "is this still live". It is set to the
|
||||
server's clock on **every accepted webhook** for that fingerprint, including the
|
||||
unchanged firing notifications Alertmanager re-sends every `repeat_interval`.
|
||||
Clients may rely on this:
|
||||
|
||||
- **A firing alert whose `received_at` is advancing is still being refreshed.**
|
||||
Stale-dating it against `repeat_interval` is a valid liveness check, and it is
|
||||
what the built-in sweeper does (see
|
||||
[Stale alert expiry](#stale-alert-expiry)).
|
||||
- **`received_at` tracks accepted payloads, not delivery attempts.** A retry
|
||||
that describes an older instance than the stored one is discarded, and a
|
||||
discarded payload does not move `received_at`.
|
||||
- **It stops advancing once the alert resolves,** because Alertmanager stops
|
||||
re-sending. On an alert resolved by the sweeper
|
||||
(`"resolution_source": "expiry"`) it therefore marks the last time
|
||||
Alertmanager was actually heard from, which is earlier than `ends_at`.
|
||||
|
||||
`GET /api/alerts` is ordered by `received_at` descending — most recently
|
||||
refreshed first — and the `?from=` / `?to=` filters on both the alert and stats
|
||||
endpoints select on `received_at`, not `starts_at`.
|
||||
|
||||
##### `resolution_source` says how much to trust `ends_at`
|
||||
|
||||
An alert can leave the firing state two ways, and `resolution_source` records
|
||||
which happened. Clients may rely on this:
|
||||
|
||||
- **Absent while firing.** It is set only on resolve, and a re-fire under the
|
||||
same fingerprint clears it again, so its presence always agrees with
|
||||
`"status": "resolved"`.
|
||||
- **`"alertmanager"` — a real resolved webhook arrived.** `ends_at` is the end
|
||||
time Alertmanager reported. It is an observed value and can be displayed as
|
||||
fact.
|
||||
- **`"expiry"` — the sweeper inferred the resolve** because Alertmanager stopped
|
||||
refreshing the alert (see [Stale alert expiry](#stale-alert-expiry)). Nothing
|
||||
ever reported an end, so **`ends_at` is approximate**: it is either the stale
|
||||
`endsAt` watermark from the last notification, or — when that notification
|
||||
carried none — the time the sweep ran, which lags the last real contact by up
|
||||
to `TERDUT_STALE_AFTER` plus a sweep interval. Treat it as "no later than",
|
||||
not as when the problem stopped.
|
||||
|
||||
On these alerts `received_at` is the more truthful signal: it marks the last
|
||||
time Alertmanager was actually heard from. Surfacing the distinction is
|
||||
worthwhile, since `"expiry"` can also mean the alert is still firing and the
|
||||
notification path broke.
|
||||
|
||||
Treat the value as an open set and tolerate ones you do not recognise — new
|
||||
sources may be added, and unknown values should degrade to "resolved, reason
|
||||
unknown" rather than being rejected.
|
||||
|
||||
### On-call schedule
|
||||
|
||||
| Method | Path | Description |
|
||||
|
||||
Reference in New Issue
Block a user