Compare commits

..

4 Commits

Author SHA1 Message Date
Niklas Ye 2b396d22d6 Set the chart's placeholder version to 0.26.0
CI / chart (push) Successful in 1s
CI / security (push) Successful in 14s
CI / test (push) Successful in 2m56s
Release / test (push) Successful in 4s
Release / chart (push) Successful in 2s
Release / binaries (push) Successful in 22s
Release / image (push) Successful in 57s
Release / scan-image (push) Successful in 4s
Cosmetic: make helm-package sets the published version and appVersion
from the tag, so these two fields decide nothing (see the comment
above them). Kept in step anyway, same as dc92f51 and e8d45f9, so a
tree heading for v0.26.0 doesn't say 0.25.0.
2026-09-26 08:28:24 +02:00
Niklas Ye 1f1faa437c Show the escalation ladder as a list, with who it would page and where it is
Team -> Escalation was the draft form on the page, which showed the
ladder only as inputs. It is now a table in the style of Switches and
Sources: a row per level with a status badge, who it pages, the wait
before the next level, and the open incidents currently waiting on it.
Below it, the repeat count, the fallback topic and when the ladder last
escalated (linking the incident). The editor moved into an "Edit ladder"
sheet, so a poll of the page underneath can no longer throw away half an
edit, and the page-level draft state went with it.

Targets are resolved to who they mean today, and the badge says what
would actually happen: Ready, Escalating (an unanswered incident has
climbed to level 2 or higher), or Pages nobody. The last is the one worth
seeing before an incident finds it: an empty rota, a person with no ntfy
topic or a disabled account each make a rung a silence with a number on
it, and the target says which. The rules are pageLevel's own, so the
page cannot promise a page the notifier would skip.

"Last escalated" comes from the escalated timeline events that already
exist, so there is no migration. Acknowledging or resolving takes an
incident off the ladder, so Escalating clears then while the history
stays.

API: GET /escalation gains status and waiting per level, username,
reachable and problem per target, and last_escalated_at and
last_escalated_incident_id. Output only and additive; PUT is unchanged
and terdut-tui needs nothing.
2026-09-26 08:28:24 +02:00
Niklas Ye dc92f51cf8 Set the chart's placeholder version to 0.25.0
CI / chart (push) Successful in 1s
CI / security (push) Successful in 31s
CI / test (push) Successful in 3m20s
Release / test (push) Successful in 4s
Release / chart (push) Successful in 2s
Release / binaries (push) Successful in 20s
Release / image (push) Successful in 1m2s
Release / scan-image (push) Successful in 2s
Cosmetic: make helm-package sets the published version and appVersion
from the tag, so these two fields decide nothing (see the comment
above them). Kept in step anyway, same as e8d45f9 and 3ee8583, so a
tree heading for v0.25.0 doesn't say 0.24.0.
2026-09-26 07:52:07 +02:00
Niklas Ye d675f8ec9b List alert sources with their status and last arrival on Team -> Sources
Like Team -> Switches, the page is now a table: a status badge (Active
if the key posted within a day, Quiet if it has but not lately, Never
used), when it last posted a webhook, when an alert last arrived on it,
how many distinct alerts it refreshed in the last 24 hours, and when it
was created. Adding a source moved into a "New source" sheet, and owners
can rename one from its row.

"Last alert" and the count needed alerts to remember which source they
came in on, which they never did, so migration 010 adds
alerts.integration_id and every accepted payload stamps it. Last sender
wins when two sources post the same fingerprint. It is not backfilled: a
NULL says "before this was recorded" rather than guessing, and it heals
by itself as Alertmanager re-sends each alert every repeat_interval.
Revoking a source keeps its alerts, unattributed.

Last webhook and last alert are separate on purpose: a payload with
nothing usable in it stamps the first and not the second. The Quiet
threshold is a fixed day, a colour and not an alarm, since silence that
should page is what dead man's switches are for.

The counts are indexed subqueries (alerts_integration_idx) rather than a
join, which would read every alert a source ever delivered.

API: the integrations list gains status, last_alert_at and alerts_24h,
and PATCH /api/teams/{id}/integrations/{id} renames. Both are additive;
terdut-tui needs nothing.
2026-09-26 07:52:07 +02:00
12 changed files with 895 additions and 141 deletions
+4 -3
View File
@@ -718,13 +718,14 @@ administrator who is not in the team gets the same `404` as anybody else.
| `GET` | `/api/teams/{teamID}/members` | member | Who is in the team |
| `POST` | `/api/teams/{teamID}/members` | **owner** | Add a member, or change their role `{"user_id","role"}` |
| `DELETE` | `/api/teams/{teamID}/members/{userID}` | **owner** | Remove a member. `409` for the last owner |
| `GET` | `/api/teams/{teamID}/integrations` | member | List integrations. Never returns keys |
| `GET` | `/api/teams/{teamID}/integrations` | member | List integrations. Never returns keys. Each carries `status` (`active` if its key posted within 24h, `quiet` if it has but not lately, `never`), `last_used_at` (last webhook, usable or not), `last_alert_at` (when an alert last arrived on it) and `alerts_24h` (distinct alerts it refreshed in the last day). Alerts delivered before the source was recorded (migration 010) have none, so the last two fill in as Alertmanager re-sends them |
| `PATCH` | `/api/teams/{teamID}/integrations/{integrationID}` | **owner** | Rename `{"name"}`. The key does not change |
| `POST` | `/api/teams/{teamID}/integrations` | **owner** | Mint an integration `{"name","kind"}` — key and URL shown once |
| `DELETE` | `/api/teams/{teamID}/integrations/{integrationID}` | **owner** | Revoke an integration |
| `DELETE` | `/api/teams/{teamID}/integrations/{integrationID}` | **owner** | Revoke an integration. Alerts it delivered stay, unattributed |
| `GET` | `/api/teams/{teamID}/invites` | **owner** | The team's invite links, with their uses and expiry. Never the tokens |
| `POST` | `/api/teams/{teamID}/invites` | **owner** | Mint one `{"role","max_uses"}` — the full URL is returned once |
| `DELETE` | `/api/teams/{teamID}/invites/{inviteID}` | **owner** | Revoke a link before it expires |
| `GET` | `/api/teams/{teamID}/escalation` | member | The team's [escalation ladder](#escalation) `{repeat_count, fallback_topic, levels[]}`. Empty levels means the team has none |
| `GET` | `/api/teams/{teamID}/escalation` | member | The team's [escalation ladder](#escalation) `{repeat_count, fallback_topic, levels[], last_escalated_at?, last_escalated_incident_id?}`. Empty levels means the team has none. Each level also carries `status` (`ready`, `escalating` when an unanswered incident has climbed to it, `unreachable` when nobody on it could be woken), `waiting` (ids of the open incidents on it) and, per target, `username` (who it means today — the person on call, for a rota target), `reachable` and `problem`. The extra fields are output only; `PUT` takes the plain shape |
| `PUT` | `/api/teams/{teamID}/escalation` | **owner** | Replace it wholesale. `400` for a level with no targets or no timeout — a rung that pages nobody is a silence with a number on it |
| `GET` | `/api/teams/{teamID}/deadman/switches` | member | The team's [dead man's switches](#dead-mans-switch), each `{id, name, matcher, timeout_seconds, severity, status, last_heartbeat_at, last_triggered_at, open_incident_id, sources[]}`. `status` is `healthy`, `dead` or `dormant`; `sources` has one entry per heartbeat fingerprint. Empty when the team watches nothing |
| `POST` | `/api/teams/{teamID}/deadman/switches` | **owner** | Add one: `{name?, matcher, timeout_seconds, severity?}`. `400` when the matcher names no `alertname` or holds several, or the timeout is not positive — a switch that silently watches nothing is the failure this feature exists to prevent |
+2 -2
View File
@@ -15,5 +15,5 @@ type: application
# appVersion and image.tag in values.yaml no longer agree, and that is not an oversight:
# image.tag stays "latest", which is what a local install actually pulls. appVersion is
# metadata and drives nothing.
version: 0.24.0
appVersion: "v0.24.0"
version: 0.26.0
appVersion: "v0.26.0"
+15 -10
View File
@@ -78,7 +78,7 @@ type ingested struct {
// post, and which team the alerts belong to.
func handleIntegrationWebhook(db *sql.DB, notify NotifyConfig) http.HandlerFunc {
return func(w http.ResponseWriter, r *http.Request) {
teamID, err := teamIDForKey(r.Context(), db, chi.URLParam(r, "key"))
src, err := sourceForKey(r.Context(), db, chi.URLParam(r, "key"))
if err != nil {
if errors.Is(err, errUnknownIntegration) {
// 401 and not 404: the path is real, the key is not, and a
@@ -90,11 +90,12 @@ func handleIntegrationWebhook(db *sql.DB, notify NotifyConfig) http.HandlerFunc
respond(w, http.StatusInternalServerError, errResp("internal error"))
return
}
receiveWebhook(w, r, db, notify, teamID)
receiveWebhook(w, r, db, notify, src)
}
}
func receiveWebhook(w http.ResponseWriter, r *http.Request, db *sql.DB, notify NotifyConfig, teamID int64) {
func receiveWebhook(w http.ResponseWriter, r *http.Request, db *sql.DB, notify NotifyConfig, src alertSource) {
teamID := src.teamID
var payload amPayload
if err := decodeJSON(r, &payload); err != nil {
respond(w, http.StatusBadRequest, errResp("invalid payload"))
@@ -104,7 +105,7 @@ func receiveWebhook(w http.ResponseWriter, r *http.Request, db *sql.DB, notify N
// Alertmanager retries anything that is not 2xx, and a retry of a payload
// we failed to store is more useful than an error it cannot act on — so
// failures are logged, not surfaced.
if err := ingest(r.Context(), db, notify, teamID, payload); err != nil {
if err := ingest(r.Context(), db, notify, src, payload); err != nil {
log.Printf("webhook ingest (team %d, group %q): %v", teamID, payload.GroupKey, err)
}
@@ -114,7 +115,8 @@ func receiveWebhook(w http.ResponseWriter, r *http.Request, db *sql.DB, notify N
// ingest stores a payload's alerts and reconciles the incident for its group.
// The whole payload is one transaction: an incident that opened but whose alerts
// failed to link would be a work item nobody could act on.
func ingest(ctx context.Context, db *sql.DB, notify NotifyConfig, teamID int64, payload amPayload) error {
func ingest(ctx context.Context, db *sql.DB, notify NotifyConfig, src alertSource, payload amPayload) error {
teamID := src.teamID
tx, err := db.BeginTx(ctx, nil)
if err != nil {
return err
@@ -129,7 +131,7 @@ func ingest(ctx context.Context, db *sql.DB, notify NotifyConfig, teamID int64,
return err
}
accepted, err := upsertAlerts(ctx, tx, deadman, teamID, payload.Alerts)
accepted, err := upsertAlerts(ctx, tx, deadman, src, payload.Alerts)
if err != nil {
return err
}
@@ -186,7 +188,8 @@ func ingest(ctx context.Context, db *sql.DB, notify NotifyConfig, teamID int64,
// upsertAlerts stores each alert of a payload and reports what changed. Payloads
// the ordering guard rejected are left out entirely.
func upsertAlerts(ctx context.Context, tx *sql.Tx, deadman deadmanSet, teamID int64, alerts []amAlert) ([]ingested, error) {
func upsertAlerts(ctx context.Context, tx *sql.Tx, deadman deadmanSet, src alertSource, alerts []amAlert) ([]ingested, error) {
teamID := src.teamID
now := time.Now().Unix()
accepted := make([]ingested, 0, len(alerts))
@@ -242,8 +245,8 @@ func upsertAlerts(ctx context.Context, tx *sql.Tx, deadman deadmanSet, teamID in
if _, err := tx.ExecContext(ctx, `
INSERT INTO alerts
(team_id, fingerprint, name, status, labels, annotations, starts_at, ends_at,
generator_url, received_at, resolution_source)
VALUES ($1, $2, $3, $4, $5::jsonb, $6::jsonb, $7, $8, $9, $10, $11)
generator_url, received_at, resolution_source, integration_id)
VALUES ($1, $2, $3, $4, $5::jsonb, $6::jsonb, $7, $8, $9, $10, $11, $12)
ON CONFLICT (team_id, fingerprint) DO UPDATE SET
status = excluded.status,
labels = excluded.labels,
@@ -257,6 +260,8 @@ func upsertAlerts(ctx context.Context, tx *sql.Tx, deadman deadmanSet, teamID in
-- is a breaking API change — see models.Alert.ReceivedAt.
received_at = excluded.received_at,
resolution_source = excluded.resolution_source,
-- Last sender wins; see migration 010.
integration_id = excluded.integration_id,
-- A re-fire makes the alert current again, so it leaves the archive.
archived_at = CASE WHEN excluded.status = 'firing'
THEN NULL ELSE alerts.archived_at END
@@ -267,7 +272,7 @@ func upsertAlerts(ctx context.Context, tx *sql.Tx, deadman deadmanSet, teamID in
teamID, a.Fingerprint, name, a.Status,
string(labelsJSON), string(annotationsJSON),
a.StartsAt.Unix(), endsAtUnix,
a.GeneratorURL, now, resolutionSource,
a.GeneratorURL, now, resolutionSource, src.integrationID,
); err != nil {
return nil, err
}
+185 -1
View File
@@ -346,10 +346,194 @@ func handleGetEscalation(db *sql.DB) http.HandlerFunc {
respond(w, http.StatusInternalServerError, errResp("internal error"))
return
}
respond(w, http.StatusOK, escalationResponse(policy, teamID))
view, err := escalationStatus(r.Context(), db, teamID, policy)
if err != nil {
respond(w, http.StatusInternalServerError, errResp("internal error"))
return
}
respond(w, http.StatusOK, view)
}
}
// Level statuses, as the Escalation page colours them.
const (
levelReady = "ready"
levelEscalating = "escalating"
levelUnreachable = "unreachable"
)
// escalationTargetView is a target with who it means today and whether that
// person can actually be woken. The extra fields are output only: the PUT body
// is the plain escalationTargetJSON, and anything else in it is ignored.
type escalationTargetView struct {
escalationTargetJSON
// Username is who the target resolves to right now: the named person, or
// whoever the rota says is on call today. Empty when nobody is.
Username string `json:"username,omitempty"`
// Reachable is whether a page to this target would go anywhere, and Problem
// says why not when it would not — the same conditions pageLevel skips on.
Reachable bool `json:"reachable"`
Problem string `json:"problem,omitempty"`
}
type escalationLevelView struct {
Position int64 `json:"position"`
TimeoutSeconds int64 `json:"timeout_seconds"`
Targets []escalationTargetView `json:"targets"`
// Status is unreachable when no target of the level could be woken — a rung
// that looks configured and pages nobody, which is worth seeing before an
// incident finds it — escalating when an unanswered incident has climbed to
// it, and ready otherwise.
Status string `json:"status"`
// Waiting lists the open, unacknowledged incidents currently on this level.
Waiting []int64 `json:"waiting"`
}
type escalationView struct {
TeamID int64 `json:"team_id"`
RepeatCount int64 `json:"repeat_count"`
FallbackTopic string `json:"fallback_topic"`
Levels []escalationLevelView `json:"levels"`
// LastEscalatedAt is when an incident of this team last moved up the ladder,
// or ran off the end of it, and LastEscalatedIncidentID which one. Absent
// when nothing ever has: a ladder nobody has needed yet.
LastEscalatedAt *time.Time `json:"last_escalated_at,omitempty"`
LastEscalatedIncidentID *int64 `json:"last_escalated_incident_id,omitempty"`
}
// escalationStatus is a team's ladder together with what it would do right now
// and what it has been doing. The resolution follows pageLevel's rules, so the
// page cannot promise a page that the notifier would skip.
func escalationStatus(ctx context.Context, db *sql.DB, teamID int64, policy *escalationPolicy) (escalationView, error) {
base := escalationResponse(policy, teamID)
out := escalationView{
TeamID: teamID, RepeatCount: base.RepeatCount, FallbackTopic: base.FallbackTopic,
Levels: []escalationLevelView{},
}
if !policy.configured() {
return out, nil
}
onCall, err := currentOnCall(ctx, db, teamID)
if err != nil {
return out, err
}
type account struct {
username string
topic bool
disabled bool
}
accounts := map[int64]account{}
lookup := func(id int64) (account, error) {
if a, ok := accounts[id]; ok {
return a, nil
}
var a account
var topic *string
var disabledAt *int64
if err := db.QueryRowContext(ctx,
"SELECT username, ntfy_topic, disabled_at FROM users WHERE id = $1", id).
Scan(&a.username, &topic, &disabledAt); err != nil {
return a, err
}
a.topic = topic != nil && *topic != ""
a.disabled = disabledAt != nil
accounts[id] = a
return a, nil
}
waiting := map[int64][]int64{}
rows, err := db.QueryContext(ctx, `
SELECT id, escalation_level FROM incidents
WHERE team_id = $1 AND resolved_at IS NULL AND archived_at IS NULL
AND status = 'triggered' AND escalation_level > 0
ORDER BY id`, teamID)
if err != nil {
return out, err
}
for rows.Next() {
var id, level int64
if err := rows.Scan(&id, &level); err != nil {
rows.Close()
return out, err
}
waiting[level] = append(waiting[level], id)
}
rows.Close()
if err := rows.Err(); err != nil {
return out, err
}
for _, l := range base.Levels {
level := escalationLevelView{
Position: l.Position, TimeoutSeconds: l.TimeoutSeconds,
Targets: []escalationTargetView{}, Waiting: []int64{},
}
if w := waiting[l.Position]; w != nil {
level.Waiting = w
}
anyReachable := false
for _, t := range l.Targets {
view := escalationTargetView{escalationTargetJSON: t}
userID := t.UserID
if t.Kind == "oncall" {
userID = onCall
}
switch {
case userID == nil:
view.Problem = "nobody is on call today"
default:
a, err := lookup(*userID)
switch {
case err != nil:
view.Problem = "account not found"
case a.disabled:
view.Username, view.Problem = a.username, "account is disabled"
case !a.topic:
view.Username, view.Problem = a.username, "has no ntfy topic"
default:
view.Username, view.Reachable = a.username, true
}
}
anyReachable = anyReachable || view.Reachable
level.Targets = append(level.Targets, view)
}
switch {
case !anyReachable:
level.Status = levelUnreachable
case l.Position >= 2 && len(level.Waiting) > 0:
level.Status = levelEscalating
default:
level.Status = levelReady
}
out.Levels = append(out.Levels, level)
}
var incidentID, at int64
switch err := db.QueryRowContext(ctx, `
SELECT e.incident_id, e.created_at
FROM incident_events e JOIN incidents i ON i.id = e.incident_id
WHERE i.team_id = $1 AND e.type = $2
ORDER BY e.created_at DESC, e.id DESC LIMIT 1`, teamID, evEscalated).
Scan(&incidentID, &at); {
case err == sql.ErrNoRows:
case err != nil:
return out, err
default:
t := time.Unix(at, 0).UTC()
out.LastEscalatedAt, out.LastEscalatedIncidentID = &t, &incidentID
}
return out, nil
}
type escalationLevelJSON struct {
Position int64 `json:"position"`
TimeoutSeconds int64 `json:"timeout_seconds"`
+119
View File
@@ -383,3 +383,122 @@ func TestEscalation_SkipsUnreachableTargets(t *testing.T) {
t.Errorf("a target with no topic should page nothing, paged %v", got)
}
}
// ---------------------------------------------------------------------------
// The ladder as the Escalation page reads it
// ---------------------------------------------------------------------------
type ladderLevel struct {
Status string `json:"status"`
Waiting []int64 `json:"waiting"`
Targets []struct {
Kind string `json:"kind"`
Username string `json:"username"`
Reachable bool `json:"reachable"`
Problem string `json:"problem"`
} `json:"targets"`
}
type ladderView struct {
Levels []ladderLevel `json:"levels"`
LastEscalatedAt *string `json:"last_escalated_at"`
LastEscalatedIncidentID *int64 `json:"last_escalated_incident_id"`
}
func readLadder(t *testing.T, s *ts) ladderView {
t.Helper()
var v ladderView
decode(t, s.req(t, http.MethodGet, "/api/teams/"+defaultTeam+"/escalation", nil), &v)
return v
}
// Targets say who they mean today, so "whoever is on call" is a name and not a
// promise.
func TestEscalation_StatusResolvesTargets(t *testing.T) {
s, _ := notifyTS(t, api.NotifyConfig{PublicURL: "https://terdut.example.com", RepeatEvery: 15 * time.Minute})
second := teamUser(t, s, "second", "terdut-second")
ladder(t, s, second, 0, "terdut-fallback")
v := readLadder(t, s)
if len(v.Levels) != 2 {
t.Fatalf("expected 2 levels, got %d", len(v.Levels))
}
if got := v.Levels[0].Targets[0]; got.Kind != "oncall" || got.Username != "admin" || !got.Reachable {
t.Errorf("the rota target should resolve to the person on call, got %+v", got)
}
if got := v.Levels[1].Targets[0]; got.Username != "second" || !got.Reachable {
t.Errorf("the named target should be reachable, got %+v", got)
}
if v.Levels[0].Status != "ready" || v.Levels[1].Status != "ready" || v.LastEscalatedAt != nil {
t.Errorf("an idle, healthy ladder is ready and has never escalated, got %+v", v)
}
}
// A rung that would page nobody is called out before an incident finds it.
func TestEscalation_StatusFlagsUnreachableLevels(t *testing.T) {
s, _ := notifyTS(t, api.NotifyConfig{PublicURL: "https://terdut.example.com", RepeatEvery: 15 * time.Minute})
silent := teamUser(t, s, "silent", "terdut-silent")
ladder(t, s, silent, 0, "terdut-fallback")
// Nobody on call today, and the named person loses their topic.
s.exec(t, "DELETE FROM schedule_entries")
s.exec(t, "UPDATE users SET ntfy_topic = NULL WHERE id = $1", silent)
v := readLadder(t, s)
if v.Levels[0].Status != "unreachable" || v.Levels[0].Targets[0].Problem != "nobody is on call today" {
t.Errorf("an empty rota should make level 1 unreachable, got %+v", v.Levels[0])
}
if v.Levels[1].Status != "unreachable" || v.Levels[1].Targets[0].Problem != "has no ntfy topic" {
t.Errorf("a person with no topic should make level 2 unreachable, got %+v", v.Levels[1])
}
s.exec(t, "UPDATE users SET disabled_at = 1 WHERE id = $1", silent)
if p := readLadder(t, s).Levels[1].Targets[0].Problem; p != "account is disabled" {
t.Errorf("a disabled account should say so, got %q", p)
}
}
// Where unanswered incidents are right now, and when the ladder last did its
// job.
func TestEscalation_StatusShowsWhoIsWaitingAndLastEscalation(t *testing.T) {
s, _ := notifyTS(t, api.NotifyConfig{PublicURL: "https://terdut.example.com", RepeatEvery: 15 * time.Minute})
second := teamUser(t, s, "second", "terdut-second")
ladder(t, s, second, 0, "terdut-fallback")
postWebhook(t, s, []map[string]any{
amAlert("fp-wait", "DiskFull", "firing", "2026-05-20T10:00:00Z", zeroTime, nil),
})
s.sweepNotify(t)
// On level 1 it is waiting, which is normal and not yet an escalation.
v := readLadder(t, s)
if len(v.Levels[0].Waiting) != 1 || v.Levels[0].Status != "ready" || v.LastEscalatedAt != nil {
t.Fatalf("a fresh incident waits on level 1 quietly, got %+v", v)
}
overdue(t, s, 1)
s.sweepNotify(t)
v = readLadder(t, s)
if v.Levels[1].Status != "escalating" || len(v.Levels[1].Waiting) != 1 || v.Levels[1].Waiting[0] != 1 {
t.Errorf("level 2 should be escalating with the incident on it, got %+v", v.Levels[1])
}
if v.LastEscalatedAt == nil || v.LastEscalatedIncidentID == nil || *v.LastEscalatedIncidentID != 1 {
t.Errorf("the escalation should be recorded, got %+v", v)
}
// Somebody answers: nothing is waiting, but the history stays.
s.req(t, http.MethodPost, "/api/incidents/1/acknowledge", nil).Body.Close()
v = readLadder(t, s)
if v.Levels[1].Status != "ready" || len(v.Levels[1].Waiting) != 0 || v.LastEscalatedAt == nil {
t.Errorf("an acknowledged incident stops waiting but stays in the history, got %+v", v)
}
}
// No ladder is a real answer, not an error.
func TestEscalation_StatusWithoutALadder(t *testing.T) {
s, _ := notifyTS(t, api.NotifyConfig{PublicURL: "https://terdut.example.com", RepeatEvery: 15 * time.Minute})
v := readLadder(t, s)
if len(v.Levels) != 0 || v.LastEscalatedAt != nil {
t.Errorf("a team with no ladder should read as empty, got %+v", v)
}
}
+1
View File
@@ -151,6 +151,7 @@ func NewRouter(db *sql.DB, notify NotifyConfig, cfg config.Config) http.Handler
// Integrations: where a team's alerts come in, and the key that says so.
r.Get("/api/teams/{teamID}/integrations", handleListIntegrations(db))
r.Post("/api/teams/{teamID}/integrations", handleCreateIntegration(db, notify.PublicURL))
r.Patch("/api/teams/{teamID}/integrations/{integrationID}", handleRenameIntegration(db))
r.Delete("/api/teams/{teamID}/integrations/{integrationID}", handleDeleteIntegration(db))
// The rota is per team. /api/schedule/current is the exception: it
+171
View File
@@ -0,0 +1,171 @@
package api_test
import (
"bytes"
"net/http"
"strings"
"testing"
"time"
)
// listSources reads a team's alert sources as the Sources page does.
func listSources(t *testing.T, tm teamFixture) []map[string]any {
t.Helper()
return list(t, tm.call(http.MethodGet, "/api/teams/"+id64(tm.id)+"/integrations", nil))
}
// addSource mints a second source in a team and returns its key.
func addSource(t *testing.T, tm teamFixture, name string) string {
t.Helper()
var out struct {
Key string `json:"key"`
}
decode(t, tm.call(http.MethodPost, "/api/teams/"+id64(tm.id)+"/integrations",
map[string]string{"name": name}), &out)
return out.Key
}
// A source that has never posted is "never", with nothing to say about alerts.
func TestSources_NeverUsedIsBlank(t *testing.T) {
s := newTS(t)
tm := newTeam(t, s, "red")
got := listSources(t, tm)
if len(got) != 1 {
t.Fatalf("expected 1 source, got %d", len(got))
}
src := got[0]
if src["status"] != "never" || src["last_used_at"] != nil || src["last_alert_at"] != nil {
t.Errorf("a source nobody has posted on should be blank, got %v", src)
}
if src["alerts_24h"].(float64) != 0 {
t.Errorf("alerts_24h = %v, want 0", src["alerts_24h"])
}
}
// Each source is credited with what arrived on its own key, and only that.
func TestSources_AlertsAreAttributedToTheirSource(t *testing.T) {
s := newTS(t)
tm := newTeam(t, s, "red")
second := addSource(t, tm, "staging")
postToIntegration(t, s, tm.key, "fp-1", "DiskFull")
postToIntegration(t, s, tm.key, "fp-2", "CPUHot")
got := listSources(t, tm)
first, other := got[0], got[1]
if first["status"] != "active" || first["last_used_at"] == nil || first["last_alert_at"] == nil {
t.Errorf("the source that posted should be active with timestamps, got %v", first)
}
if first["alerts_24h"].(float64) != 2 {
t.Errorf("alerts_24h = %v, want 2", first["alerts_24h"])
}
if other["status"] != "never" || other["alerts_24h"].(float64) != 0 {
t.Errorf("the other source should be untouched, got %v", other)
}
// Re-sending the same alert on the other key moves it: last sender wins.
postToIntegration(t, s, second, "fp-1", "DiskFull")
got = listSources(t, tm)
if got[0]["alerts_24h"].(float64) != 1 || got[1]["alerts_24h"].(float64) != 1 {
t.Errorf("fp-1 should have moved to the second source, got %v and %v",
got[0]["alerts_24h"], got[1]["alerts_24h"])
}
}
// A payload with no alerts in it is a webhook, not an alert: the source was
// heard from, and nothing arrived.
func TestSources_EmptyPayloadStampsUseButNotAlert(t *testing.T) {
s := newTS(t)
tm := newTeam(t, s, "red")
resp, err := http.Post(s.URL+"/api/integrations/"+tm.key+"/alertmanager",
"application/json", bytes.NewReader([]byte(`{"version":"4","status":"firing","alerts":[]}`)))
if err != nil {
t.Fatalf("post: %v", err)
}
resp.Body.Close()
src := listSources(t, tm)[0]
if src["status"] != "active" || src["last_alert_at"] != nil {
t.Errorf("want active with no alert yet, got %v", src)
}
}
// Quiet is "has posted, not lately"; the alert counter forgets after a day but
// the last alert's timestamp is kept.
func TestSources_QuietAfterADay(t *testing.T) {
s := newTS(t)
tm := newTeam(t, s, "red")
postToIntegration(t, s, tm.key, "fp-1", "DiskFull")
old := time.Now().Add(-48 * time.Hour).Unix()
s.exec(t, "UPDATE integrations SET last_used_at = $1", old)
s.exec(t, "UPDATE alerts SET received_at = $1 WHERE fingerprint = 'fp-1'", old)
src := listSources(t, tm)[0]
if src["status"] != "quiet" {
t.Errorf("status = %v, want quiet", src["status"])
}
if src["alerts_24h"].(float64) != 0 {
t.Errorf("alerts_24h = %v, want 0", src["alerts_24h"])
}
if src["last_alert_at"] == nil {
t.Error("last_alert_at should survive the day")
}
}
// Revoking a source does not take its alerts with it.
func TestSources_RevokeKeepsTheAlerts(t *testing.T) {
s := newTS(t)
tm := newTeam(t, s, "red")
postToIntegration(t, s, tm.key, "fp-1", "DiskFull")
id := int64(listSources(t, tm)[0]["id"].(float64))
resp := tm.call(http.MethodDelete, "/api/teams/"+id64(tm.id)+"/integrations/"+id64(id), nil)
resp.Body.Close()
if resp.StatusCode != http.StatusNoContent {
t.Fatalf("revoke: %d", resp.StatusCode)
}
if got := len(list(t, tm.call(http.MethodGet, "/api/alerts", nil))); got != 1 {
t.Errorf("the alert should outlive its source, got %d alerts", got)
}
}
// Renaming is an owner's, scoped to the team, and does not touch the key.
func TestSources_Rename(t *testing.T) {
s := newTS(t)
tm := newTeam(t, s, "red")
other := newTeam(t, s, "blue")
id := int64(listSources(t, tm)[0]["id"].(float64))
path := "/api/teams/" + id64(tm.id) + "/integrations/" + id64(id)
resp := tm.call(http.MethodPatch, path, map[string]string{"name": " prod "})
resp.Body.Close()
if resp.StatusCode != http.StatusNoContent {
t.Fatalf("rename: %d", resp.StatusCode)
}
if name := listSources(t, tm)[0]["name"]; name != "prod" {
t.Errorf("name = %q, want it trimmed to prod", name)
}
postToIntegration(t, s, tm.key, "fp-1", "DiskFull") // the old key still works
for name, body := range map[string]map[string]string{
"empty": {"name": " "},
"too long": {"name": strings.Repeat("x", 101)},
} {
resp := tm.call(http.MethodPatch, path, body)
resp.Body.Close()
if resp.StatusCode != http.StatusBadRequest {
t.Errorf("%s name: expected 400, got %d", name, resp.StatusCode)
}
}
// Another team's owner cannot reach it.
resp = other.call(http.MethodPatch, "/api/teams/"+id64(other.id)+"/integrations/"+id64(id),
map[string]string{"name": "mine now"})
resp.Body.Close()
if resp.StatusCode != http.StatusNotFound {
t.Errorf("renaming another team's source: expected 404, got %d", resp.StatusCode)
}
}
+126 -19
View File
@@ -355,8 +355,42 @@ func isLastTeamOwner(ctx context.Context, db *sql.DB, teamID, userID int64) (boo
// Integrations
// ---------------------------------------------------------------------------
// handleListIntegrations lists a team's integrations. Never the keys: those
// exist in plaintext only in the response that created them.
// sourceQuietAfter is how long a source may go without posting before the
// Sources page calls it quiet rather than active. A day is longer than any
// repeat_interval worth having, so an Alertmanager that is up and has anything
// firing never crosses it; a source with nothing firing may, and that is a
// reason to look, not proof of a fault — which is why this is a colour and not
// an alarm. Dead man's switches are where silence pages.
const sourceQuietAfter = 24 * time.Hour
const (
sourceActive = "active"
sourceQuiet = "quiet"
sourceNever = "never"
)
// integrationStatus is an integration as the Sources page shows it.
type integrationStatus struct {
models.Integration
// Status is active when the key posted within sourceQuietAfter, quiet when
// it has posted but not lately, never when it has not posted at all.
Status string `json:"status"`
// LastAlertAt is when an alert last arrived on this source, which is not the
// same as when it last posted: a payload with nothing usable in it stamps
// last_used_at and not this. Absent until an alert has arrived since
// migration 010 started recording it.
LastAlertAt *time.Time `json:"last_alert_at,omitempty"`
// Alerts24h counts the distinct alerts this source refreshed in the last
// day. An alert re-sent every few hours counts once, not once per re-send.
Alerts24h int64 `json:"alerts_24h"`
}
// handleListIntegrations lists a team's integrations with what each has been
// delivering. Never the keys: those exist in plaintext only in the response that
// created them.
func handleListIntegrations(db *sql.DB) http.HandlerFunc {
return func(w http.ResponseWriter, r *http.Request) {
teamID, ok := teamParam(w, r)
@@ -367,28 +401,45 @@ func handleListIntegrations(db *sql.DB) http.HandlerFunc {
return
}
now := time.Now()
rows, err := db.QueryContext(r.Context(), `
SELECT id, team_id, kind, name, created_at, last_used_at
FROM integrations
WHERE team_id = $1
ORDER BY id`, teamID)
SELECT i.id, i.team_id, i.kind, i.name, i.created_at, i.last_used_at,
-- Scalar subqueries, not a join and GROUP BY: each is a
-- single range over alerts_integration_idx, where the join
-- would read every alert a source ever delivered.
(SELECT MAX(received_at) FROM alerts WHERE integration_id = i.id),
(SELECT COUNT(*) FROM alerts
WHERE integration_id = i.id AND received_at >= $2)
FROM integrations i
WHERE i.team_id = $1
ORDER BY i.id`, teamID, now.Add(-sourceQuietAfter).Unix())
if err != nil {
respond(w, http.StatusInternalServerError, errResp("internal error"))
return
}
defer rows.Close()
integrations := []models.Integration{}
integrations := []integrationStatus{}
for rows.Next() {
var i models.Integration
var i integrationStatus
var created int64
var lastUsed *int64
if err := rows.Scan(&i.ID, &i.TeamID, &i.Kind, &i.Name, &created, &lastUsed); err != nil {
var lastUsed, lastAlert *int64
if err := rows.Scan(&i.ID, &i.TeamID, &i.Kind, &i.Name, &created, &lastUsed,
&lastAlert, &i.Alerts24h); err != nil {
respond(w, http.StatusInternalServerError, errResp("internal error"))
return
}
i.CreatedAt = time.Unix(created, 0).UTC()
i.LastUsedAt = unixPtr(lastUsed)
i.LastAlertAt = unixPtr(lastAlert)
switch {
case i.LastUsedAt == nil:
i.Status = sourceNever
case now.Sub(*i.LastUsedAt) > sourceQuietAfter:
i.Status = sourceQuiet
default:
i.Status = sourceActive
}
integrations = append(integrations, i)
}
if err := rows.Err(); err != nil {
@@ -457,6 +508,54 @@ func handleCreateIntegration(db *sql.DB, publicURL string) http.HandlerFunc {
}
}
// handleRenameIntegration renames a source. The key is untouched, so nothing
// posting with it notices.
func handleRenameIntegration(db *sql.DB) http.HandlerFunc {
return func(w http.ResponseWriter, r *http.Request) {
teamID, ok := teamParam(w, r)
if !ok {
return
}
if !requireTeamOwner(w, r, teamID) {
return
}
id, err := strconv.ParseInt(chi.URLParam(r, "integrationID"), 10, 64)
if err != nil {
respond(w, http.StatusBadRequest, errResp("invalid integration id"))
return
}
var req struct {
Name string `json:"name"`
}
if err := decodeJSON(r, &req); err != nil {
respond(w, http.StatusBadRequest, errResp("invalid request body"))
return
}
req.Name = strings.TrimSpace(req.Name)
if req.Name == "" {
respond(w, http.StatusBadRequest, errResp("name is required"))
return
}
if len(req.Name) > 100 {
respond(w, http.StatusBadRequest, errResp("name is too long"))
return
}
res, err := db.ExecContext(r.Context(),
"UPDATE integrations SET name = $1 WHERE id = $2 AND team_id = $3", req.Name, id, teamID)
if err != nil {
respond(w, http.StatusInternalServerError, errResp("internal error"))
return
}
if n, _ := res.RowsAffected(); n == 0 {
respond(w, http.StatusNotFound, errResp("not found"))
return
}
w.WriteHeader(http.StatusNoContent)
}
}
func handleDeleteIntegration(db *sql.DB) http.HandlerFunc {
return func(w http.ResponseWriter, r *http.Request) {
teamID, ok := teamParam(w, r)
@@ -493,24 +592,32 @@ func integrationPath(key, kind string) string {
return "/api/integrations/" + key + "/" + kind
}
// teamIDForKey resolves an integration key to its team, and stamps the key's
// alertSource is who an arriving webhook is from: the integration whose key it
// used, and the team that integration puts its alerts in.
type alertSource struct {
integrationID int64
teamID int64
}
// sourceForKey resolves an integration key to its source, and stamps the key's
// last use. An unknown key is not an error worth distinguishing: the caller is
// told nothing beyond "no".
func teamIDForKey(ctx context.Context, db *sql.DB, key string) (int64, error) {
var teamID int64
func sourceForKey(ctx context.Context, db *sql.DB, key string) (alertSource, error) {
var src alertSource
err := db.QueryRowContext(ctx,
"SELECT team_id FROM integrations WHERE key_hash = $1", hashToken(key)).Scan(&teamID)
"SELECT id, team_id FROM integrations WHERE key_hash = $1", hashToken(key)).
Scan(&src.integrationID, &src.teamID)
if errors.Is(err, sql.ErrNoRows) {
return 0, errUnknownIntegration
return alertSource{}, errUnknownIntegration
}
if err != nil {
return 0, err
return alertSource{}, err
}
// Best effort, like an API key's: a failed stamp must not reject an alert.
db.ExecContext(ctx, //nolint:errcheck
"UPDATE integrations SET last_used_at = $1 WHERE key_hash = $2",
time.Now().Unix(), hashToken(key))
return teamID, nil
"UPDATE integrations SET last_used_at = $1 WHERE id = $2",
time.Now().Unix(), src.integrationID)
return src, nil
}
var errUnknownIntegration = errors.New("unknown integration key")
@@ -0,0 +1,21 @@
-- Which alert source an alert last arrived on.
--
-- Team -> Sources shows when each source last posted, which integrations
-- already knew (last_used_at, stamped on every webhook). What it could not say
-- was what a source delivered: an alert never recorded the key it came in on, so
-- "prod alertmanager" and "staging alertmanager" were indistinguishable once
-- inside. This column is that link, and lets the page show each source's last
-- alert and how many alerts it has kept fresh over the past day.
--
-- Last sender wins: every accepted payload restamps it, the way it advances
-- received_at. Two sources posting the same fingerprint into one team is
-- already one alert, and it is attributed to whichever spoke last.
--
-- Nullable, and not backfilled. Alerts that arrived before this migration have
-- no source, and NULL says so honestly rather than guessing. It heals by itself:
-- Alertmanager re-sends every alert each repeat_interval, and each re-send is an
-- accepted payload. Deleting a source keeps its alerts, unattributed.
ALTER TABLE alerts ADD COLUMN integration_id BIGINT REFERENCES integrations(id) ON DELETE SET NULL;
CREATE INDEX alerts_integration_idx ON alerts (integration_id, received_at)
WHERE integration_id IS NOT NULL;
+15 -6
View File
@@ -363,7 +363,11 @@ input:focus, textarea:focus { outline: none; border-color: var(--accent); box-sh
.badge.st-dead { background: var(--crit-soft); color: var(--crit); }
/* Dormant is the plain badge on purpose: nothing has gone wrong and nothing has
gone right, which is what the muted default already says. */
.badge.st-dormant { background: var(--surface-2); color: var(--muted); }
.badge.st-dormant, .badge.st-never { background: var(--surface-2); color: var(--muted); }
.badge.st-active { background: var(--ok-soft); color: var(--ok); }
.badge.st-quiet, .badge.st-escalating { background: var(--warn-soft); color: var(--warn); }
.badge.st-ready { background: var(--ok-soft); color: var(--ok); }
.badge.st-unreachable { background: var(--crit-soft); color: var(--crit); }
.badge.sev-critical { background: var(--crit-soft); color: var(--crit); }
.badge.sev-warning { background: var(--warn-soft); color: var(--warn); }
.badge.sev-info { background: var(--info-soft); color: var(--info); }
@@ -695,16 +699,16 @@ kbd {
.disabled-row td { opacity: 0.55; }
.btn-sm.danger { color: var(--crit); border-color: var(--crit-soft); }
/* --- dead man's switches -------------------------------------------------
/* --- status lists: alert sources and dead man's switches -----------------
Six columns do not fit a phone, so the table scrolls inside its card rather
than the page. A heartbeat under a switch with several is indented, the way
the escalation ladder indents its levels. */
.card-head { display: flex; align-items: center; justify-content: space-between; gap: 12px; flex-wrap: wrap; }
.table-scroll { overflow-x: auto; margin-top: 12px; }
.switch-table th, .switch-table td { white-space: nowrap; }
.switch-table td:nth-child(2) { white-space: normal; min-width: 12em; }
.switch-table .source-row td { border-bottom-style: dashed; }
.switch-table .source-row td:first-child { padding-left: 16px; }
.status-table th, .status-table td { white-space: nowrap; }
.status-table td.wrap { white-space: normal; min-width: 12em; }
.status-table .source-row td { border-bottom-style: dashed; }
.status-table .source-row td:first-child { padding-left: 16px; }
.source-labels { display: flex; flex-wrap: wrap; gap: 4px; align-items: center; }
.inline-form { display: flex; gap: 8px; margin-top: 12px; }
@@ -825,6 +829,11 @@ button.rota-day:hover { background: var(--surface-2); }
.ladder-head { display: flex; align-items: center; gap: 10px; margin-bottom: 6px; }
.ladder-targets { display: flex; flex-direction: column; gap: 6px; margin-top: 8px; }
.target-row { display: flex; gap: 6px; align-items: center; flex-wrap: wrap; }
.ladder-editor { display: flex; flex-direction: column; gap: 10px; align-items: flex-start; margin-top: 12px; }
/* A target that would not wake anybody says why, in place: it is the reason a
level is red, and the thing to go and fix. */
.target-line { display: flex; gap: 8px; align-items: baseline; flex-wrap: wrap; }
.target-problem { color: var(--crit); font-size: 12px; font-weight: 600; }
/* An integration key is shown exactly once, so it should look like something
to act on rather than another row of text. */
+2
View File
@@ -133,6 +133,8 @@ export const removeTeamMember = (id, userID) => call('DELETE', `/teams/${id}/mem
export const integrations = (id) => call('GET', `/teams/${id}/integrations`);
export const createIntegration = (id, name) =>
call('POST', `/teams/${id}/integrations`, { body: { name } });
export const renameIntegration = (id, integrationID, name) =>
call('PATCH', `/teams/${id}/integrations/${integrationID}`, { body: { name } });
export const deleteIntegration = (id, integrationID) =>
call('DELETE', `/teams/${id}/integrations/${integrationID}`);
+234 -100
View File
@@ -52,12 +52,10 @@ let freshKey = null; // an integration key, shown once, until the view is left
export function show(route) {
const next = route?.tab ?? null;
// A different sub-section wants different data, so the old answer goes
// rather than being shown under the new heading until the fetch lands. The
// ladder draft goes with it: it is an edit of the page being left.
// rather than being shown under the new heading until the fetch lands.
if (next !== tab) {
tab = next;
data = null;
draft = null;
}
if (!data) clear(view(), subnav(), spinner());
refresh();
@@ -185,7 +183,6 @@ function teamPicker() {
teamID = Number(select.value);
data = null;
freshKey = null;
draft = null;
refresh();
});
return h('div', { class: 'card' }, h('h2', { text: 'Team' }), select);
@@ -457,57 +454,124 @@ function memberSelect(selected) {
// --- escalation ------------------------------------------------------------
// The ladder is edited as a whole and sent as a whole, because the API replaces
// it wholesale: the levels are an order, and patching one rung would leave the
// numbering of the others undecided.
let draft = null;
const LEVEL_STATUS = {
ready: { label: 'Ready', hint: 'Somebody here can be woken.' },
escalating: { label: 'Escalating', hint: 'An unanswered incident has climbed to this level.' },
unreachable: { label: 'Pages nobody', hint: 'Nobody on this level can be woken right now.' },
};
const levelBadge = (status) => statusBadge(LEVEL_STATUS, status, 'ready');
// One target as the list shows it: who it means today, and why it would not
// wake them if it would not.
function targetLine(t) {
const label = t.kind === 'oncall'
? `On call${t.username ? ` · ${t.username}` : ''}`
: (t.username || 'Unknown person');
return h('div', { class: 'target-line' },
h('span', { text: label }),
t.problem && h('span', { class: 'target-problem', text: t.problem }));
}
function escalationCard() {
const esc = data.escalation;
if (!draft) {
draft = {
repeat_count: esc.repeat_count || 0,
fallback_topic: esc.fallback_topic || '',
levels: (esc.levels || []).map((l) => ({
timeout_seconds: l.timeout_seconds,
targets: (l.targets || []).map((t) => ({ kind: t.kind, user_id: t.user_id })),
const esc = data.escalation || {};
const levels = esc.levels || [];
const rows = levels.map((l) => h('tr', {},
h('td', {}, h('strong', { text: `Level ${l.position}` })),
h('td', {}, levelBadge(l.status)),
h('td', { class: 'wrap' }, ...l.targets.map(targetLine)),
h('td', { class: 'muted small', text: duration(l.timeout_seconds * 1000) }),
h('td', { class: 'small' }, l.waiting?.length
? l.waiting.flatMap((id, i) => [i > 0 && ', ', h('a', { href: `/incidents/${id}`, text: `#${id}` })])
: h('span', { class: 'muted', text: '—' })),
));
const facts = [];
if (levels.length) {
const n = esc.repeat_count || 0;
if (n) facts.push(`Then the whole ladder repeats ${n} more ${n === 1 ? 'time' : 'times'}.`);
facts.push(esc.fallback_topic
? ['Finally the ntfy topic ', h('code', { text: esc.fallback_topic }), ' is paged once.']
: 'No fallback topic: after the last level the chain just ends.');
facts.push(esc.last_escalated_at
? ['Last escalated ',
h('span', { title: when(esc.last_escalated_at), text: ago(esc.last_escalated_at) }),
' on ', h('a', { href: `/incidents/${esc.last_escalated_incident_id}`, text: `#${esc.last_escalated_incident_id}` }), '.']
: 'Nothing has needed to escalate yet.');
}
return h('div', { class: 'card' },
h('div', { class: 'card-head' },
h('h2', { text: 'Escalation' }),
isOwner() && h('button', {
class: 'btn', type: 'button', onclick: openLadderEditor,
text: levels.length ? 'Edit ladder' : 'Set up ladder',
})),
};
}
h('p', { class: 'muted small' },
'When a level’s wait passes and nobody has acknowledged, the next level is ',
'paged. Acknowledging or resolving stops it; snoozing pauses it.'),
levels.length
? h('div', { class: 'table-scroll' },
h('table', { class: 'admin-table status-table' },
h('thead', {}, h('tr', {},
h('th', { text: 'Level' }), h('th', { text: 'Status' }), h('th', { text: 'Pages' }),
h('th', { text: 'Then after' }), h('th', { text: 'Waiting now' }))),
h('tbody', {}, rows)))
: h('p', { class: 'muted' },
'No ladder. An unacknowledged incident re-pages the same person every ',
'reminder interval and nobody else is woken.'),
...facts.map((f) => h('p', { class: 'muted small' }, f)),
);
}
const body = [];
if (!draft.levels.length) {
body.push(h('p', { class: 'muted' },
'No ladder. An unacknowledged incident re-pages the same person every ',
'reminder interval and nobody else is woken.'));
}
// The ladder is edited as a whole and sent as a whole, because the API replaces
// it wholesale: the levels are an order, and patching one rung would leave the
// numbering of the others undecided. The draft lives in the sheet, so a poll of
// the page underneath cannot throw away half an edit.
function openLadderEditor() {
const esc = data.escalation || {};
const draft = {
repeat_count: esc.repeat_count || 0,
fallback_topic: esc.fallback_topic || '',
levels: (esc.levels || []).map((l) => ({
timeout_seconds: l.timeout_seconds,
targets: (l.targets || []).map((t) => ({ kind: t.kind, user_id: t.user_id })),
})),
};
draft.levels.forEach((level, i) => {
body.push(h('div', { class: 'ladder-level' },
h('div', { class: 'ladder-head' },
h('strong', { text: `Level ${i + 1}` }),
isOwner() && h('button', {
class: 'btn-sm danger', type: 'button', text: 'Remove',
onclick: () => { draft.levels.splice(i, 1); render(); },
})),
h('label', {}, 'Wait ', minutesInput(level.timeout_seconds, (secs) => {
level.timeout_seconds = secs;
}), ' before the next level'),
h('div', { class: 'ladder-targets' },
...level.targets.map((t, ti) => targetRow(level, t, ti)),
isOwner() && h('button', {
class: 'btn-sm', type: 'button', text: '+ target',
onclick: () => { level.targets.push({ kind: 'oncall' }); render(); },
})),
));
});
const body = h('div', { class: 'ladder-editor' });
const problem = h('p', { class: 'load-error', hidden: true });
if (isOwner()) {
body.push(h('button', {
const paint = () => {
const parts = [];
if (!draft.levels.length) {
parts.push(h('p', { class: 'muted small' }, 'No levels yet. Add the first one.'));
}
draft.levels.forEach((level, i) => {
parts.push(h('div', { class: 'ladder-level' },
h('div', { class: 'ladder-head' },
h('strong', { text: `Level ${i + 1}` }),
h('button', {
class: 'btn-sm danger', type: 'button', text: 'Remove',
onclick: () => { draft.levels.splice(i, 1); paint(); },
})),
h('label', {}, 'Wait ', minutesInput(level.timeout_seconds, (secs) => {
level.timeout_seconds = secs;
}), ' before the next level'),
h('div', { class: 'ladder-targets' },
...level.targets.map((t, ti) => targetRow(level, t, ti, paint)),
h('button', {
class: 'btn-sm', type: 'button', text: '+ target',
onclick: () => { level.targets.push({ kind: 'oncall' }); paint(); },
})),
));
});
parts.push(h('button', {
class: 'btn-sm', type: 'button', text: '+ level',
onclick: () => {
draft.levels.push({ timeout_seconds: 300, targets: [{ kind: 'oncall' }] });
render();
paint();
},
}));
@@ -520,31 +584,43 @@ function escalationCard() {
type: 'text', value: draft.fallback_topic, placeholder: 'terdut-oncall-all',
oninput: (e) => { draft.fallback_topic = e.target.value; },
});
body.push(h('label', {}, 'Repeat the whole ladder ', repeat, ' more times'));
body.push(h('label', {}, 'Then page this ntfy topic once ', fallback));
body.push(h('button', {
class: 'btn', type: 'button', text: 'Save ladder',
onclick: () => act(() => api.setEscalation(teamID, draft), { resetDraft: true }),
}));
}
parts.push(h('label', {}, 'Repeat the whole ladder ', repeat, ' more times'));
parts.push(h('label', {}, 'Then page this ntfy topic once ', fallback));
clear(body, ...parts);
};
paint();
return h('div', { class: 'card' },
h('h2', { text: 'Escalation' }),
h('p', { class: 'muted small' },
'When a level’s wait passes and nobody has acknowledged, the next level is ',
'paged. Acknowledging or resolving stops it; snoozing pauses it.'),
...body,
);
const save = h('button', { class: 'btn btn-primary', type: 'button', text: 'Save ladder' });
save.addEventListener('click', async () => {
try {
await api.setEscalation(teamID, draft);
} catch (err) {
problem.textContent = err.message;
problem.hidden = false;
return;
}
closeSheet(true);
refresh();
});
openSheet(() => [
h('h2', { class: 'sheet-title', text: 'Edit ladder' }),
body,
problem,
h('div', { class: 'sheet-actions' },
h('button', { class: 'btn', type: 'button', text: 'Cancel', onclick: () => closeSheet(false) }),
save),
]);
}
function targetRow(level, target, index) {
function targetRow(level, target, index, repaint) {
const kind = h('select', {},
h('option', { value: 'oncall', text: 'Whoever is on call', selected: target.kind === 'oncall' }),
h('option', { value: 'user', text: 'A specific person', selected: target.kind === 'user' }));
kind.addEventListener('change', () => {
target.kind = kind.value;
target.user_id = kind.value === 'user' ? (data.members[0] || {}).user_id : undefined;
render();
repaint();
});
const who = target.kind === 'user'
@@ -555,10 +631,10 @@ function targetRow(level, target, index) {
}
return h('div', { class: 'target-row' }, kind, who,
isOwner() && h('button', {
h('button', {
class: 'btn-sm danger', type: 'button', text: '×',
title: 'Remove this target',
onclick: () => { level.targets.splice(index, 1); render(); },
onclick: () => { level.targets.splice(index, 1); repaint(); },
}));
}
@@ -576,33 +652,54 @@ function minutesInput(seconds, onChange) {
function integrationsCard() {
const rows = (data.integrations || []).map((i) =>
h('tr', {},
h('td', {}, h('strong', { text: i.name })),
h('td', { class: 'muted small', text: i.kind }),
h('td', { class: 'muted small', text: i.last_used_at ? 'in use' : 'never used' }),
h('td', {}, isOwner() && h('button', {
class: 'btn-sm danger', type: 'button', text: 'Revoke',
onclick: async () => {
if (!(await confirm({
title: `Revoke ${i.name}?`,
text: 'Anything posting with this key stops delivering immediately.',
confirmLabel: 'Revoke',
danger: true,
}))) return;
act(() => api.deleteIntegration(teamID, i.id));
},
})),
h('td', {}, sourceBadge(i.status)),
h('td', { class: 'wrap' },
h('strong', { text: i.name }),
h('div', { class: 'muted small', text: i.kind })),
// When the key last posted, and when an alert last arrived on it. They
// differ: a payload with nothing usable in it stamps only the first.
h('td', { class: 'muted small' }, timeCell(i.last_used_at)),
h('td', { class: 'muted small' }, timeCell(i.last_alert_at)),
h('td', { class: 'muted small num', title: 'Distinct alerts refreshed in the last 24 hours',
text: String(i.alerts_24h ?? 0) }),
h('td', { class: 'muted small' }, h('span', { title: when(i.created_at), text: ago(i.created_at) })),
h('td', {}, isOwner() && h('div', { class: 'row-actions' },
h('button', {
class: 'btn-sm', type: 'button', text: 'Rename', onclick: () => openRenameSource(i),
}),
h('button', {
class: 'btn-sm danger', type: 'button', text: 'Revoke',
onclick: async () => {
if (!(await confirm({
title: `Revoke ${i.name}?`,
text: 'Anything posting with this key stops delivering immediately. Alerts it already delivered stay.',
confirmLabel: 'Revoke',
danger: true,
}))) return;
act(() => api.deleteIntegration(teamID, i.id));
},
}))),
));
return h('div', { class: 'card' },
h('h2', { text: 'Alert sources' }),
h('div', { class: 'card-head' },
h('h2', { text: 'Alert sources' }),
isOwner() && h('button', {
class: 'btn', type: 'button', text: 'New source', onclick: openNewSource,
})),
h('p', { class: 'muted small' },
'Alerts arrive on an integration key, which says both that the sender may ',
'post and which team the alerts belong to.'),
rows.length
? h('table', { class: 'admin-table' }, h('tbody', {}, rows))
: h('p', { class: 'muted', text: 'No alert source yet, so nothing can reach this team.' }),
freshKey && newKeyPanel(),
isOwner() && !freshKey && newIntegrationForm(),
rows.length
? h('div', { class: 'table-scroll' },
h('table', { class: 'admin-table status-table' },
h('thead', {}, h('tr', {},
h('th', { text: 'Status' }), h('th', { text: 'Source' }),
h('th', { text: 'Last webhook' }), h('th', { text: 'Last alert' }),
h('th', { class: 'num', text: 'Alerts 24h' }), h('th', { text: 'Created' }), h('th'))),
h('tbody', {}, rows)))
: h('p', { class: 'muted', text: 'No alert source yet, so nothing can reach this team.' }),
);
}
@@ -633,38 +730,77 @@ function newKeyPanel() {
);
}
function newIntegrationForm() {
const name = h('input', { type: 'text', placeholder: 'prod alertmanager', required: true });
const form = h('form', { class: 'inline-form' }, name,
h('button', { class: 'btn', type: 'submit', text: 'Add' }));
// A sheet with one name field, for adding a source and for renaming one: the two
// differ only in what they call and what they put in the box.
function openNameSheet({ title, submit, value, run }) {
const name = h('input', {
type: 'text', placeholder: 'prod alertmanager', required: true, value, autofocus: true,
});
const problem = h('p', { class: 'load-error', hidden: true });
const form = h('form', { class: 'stacked-form' },
h('label', {}, 'Name ', name),
problem,
h('div', { class: 'sheet-actions' },
h('button', { class: 'btn', type: 'button', text: 'Cancel', onclick: () => closeSheet(false) }),
h('button', { class: 'btn btn-primary', type: 'submit', text: submit })));
form.addEventListener('submit', async (e) => {
e.preventDefault();
try {
freshKey = await api.createIntegration(teamID, name.value.trim());
await refresh();
await run(name.value.trim());
} catch (err) {
error = err.message;
render();
problem.textContent = err.message;
problem.hidden = false;
return;
}
closeSheet(true);
refresh();
});
openSheet(() => [h('h2', { class: 'sheet-title', text: title }), form]);
}
function openNewSource() {
openNameSheet({
title: 'New source', submit: 'Add source', value: '',
// The key comes back once, and the card shows it until dismissed.
run: async (name) => { freshKey = await api.createIntegration(teamID, name); },
});
}
function openRenameSource(i) {
openNameSheet({
title: `Rename ${i.name}`, submit: 'Rename', value: i.name,
run: (name) => api.renameIntegration(teamID, i.id, name),
});
return form;
}
// --- dead man's switches ---------------------------------------------------
// Status badges, shared by the Sources and Switches lists: a table of label and
// hint per status, and one function to draw it. Module-level, so the two cards
// can be defined in either order.
const SWITCH_STATUS = {
healthy: { label: 'Healthy', hint: 'Heard from within its timeout.' },
dead: { label: 'Dead', hint: 'Silent for longer than its timeout.' },
dormant: { label: 'Dormant', hint: 'Nothing has matched yet, so there is nothing to lose.' },
};
function switchBadge(status) {
const s = SWITCH_STATUS[status] || SWITCH_STATUS.dormant;
const SOURCE_STATUS = {
active: { label: 'Active', hint: 'Posted within the last day.' },
quiet: { label: 'Quiet', hint: 'Has posted, but not in the last day. Nothing firing is a fine reason.' },
never: { label: 'Never used', hint: 'Nothing has been posted with this key yet.' },
};
function statusBadge(table, status, fallback) {
const s = table[status] || table[fallback];
const el = badge(s.label, `st-${status}`);
el.title = s.hint;
return el;
}
const switchBadge = (status) => statusBadge(SWITCH_STATUS, status, 'dormant');
const sourceBadge = (status) => statusBadge(SOURCE_STATUS, status, 'never');
const timeCell = (iso) => iso
? h('span', { title: when(iso), text: ago(iso) })
: h('span', { class: 'muted', text: 'never' });
@@ -681,7 +817,7 @@ const triggeredCell = (iso, incidentID) => {
function switchRows(sw) {
const main = h('tr', {},
h('td', {}, switchBadge(sw.status)),
h('td', {},
h('td', { class: 'wrap' },
h('strong', { text: sw.name }),
sw.name !== sw.matcher && h('div', { class: 'muted small' }, h('code', { text: sw.matcher }))),
h('td', { class: 'muted small' }, timeCell(sw.last_heartbeat_at)),
@@ -733,7 +869,7 @@ function deadmanCard() {
'quiet for longer than the switch’s timeout opens an incident.'),
switches.length
? h('div', { class: 'table-scroll' },
h('table', { class: 'admin-table switch-table' },
h('table', { class: 'admin-table status-table' },
h('thead', {}, h('tr', {},
h('th', { text: 'Status' }), h('th', { text: 'Switch' }),
h('th', { text: 'Last heartbeat' }), h('th', { text: 'Last triggered' }),
@@ -836,14 +972,12 @@ function membersCard() {
// act runs a write and reloads. Errors are shown rather than thrown away: a
// 409 from the last-owner guard or the schedule's conflict rule is the server
// explaining itself, and the reader needs to see it.
async function act(fn, { resetDraft = false } = {}) {
async function act(fn) {
try {
await fn();
error = null;
if (resetDraft) draft = null;
} catch (err) {
error = err.message;
}
if (!resetDraft) draft = null;
await refresh();
}